mirror of
https://github.com/danbulant/robotparser-rs
synced 2026-06-15 04:31:19 +00:00
40 lines
1.1 KiB
Markdown
40 lines
1.1 KiB
Markdown
# robotparser-rs
|
|
|
|
[](https://travis-ci.org/messense/robotparser-rs)
|
|
[](https://ci.appveyor.com/project/messense/robotparser-rs/branch/master)
|
|
[](https://coveralls.io/r/messense/robotparser-rs)
|
|
[](https://crates.io/crates/robotparser)
|
|
|
|
robots.txt parser for Rust.
|
|
|
|
|
|
## Installation
|
|
|
|
Add it to your ``Cargo.toml``:
|
|
|
|
```toml
|
|
[dependencies]
|
|
robotparser = "0.4"
|
|
```
|
|
|
|
Add ``extern crate robotparser`` to your crate root and your're good to go!
|
|
|
|
|
|
## Examples
|
|
|
|
```rust
|
|
extern crate robotparser;
|
|
|
|
use robotparser::RobotFileParser;
|
|
|
|
fn main() {
|
|
let parser = RobotFileParser::new("http://www.python.org/robots.txt");
|
|
parser.read();
|
|
assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
|
|
}
|
|
```
|
|
|
|
|
|
## License
|
|
|
|
This work is released under the MIT license. A copy of the license is provided in the [LICENSE](./LICENSE) file.
|