robotparser-rs/README.md

robotparser-rs

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.10"

Add extern crate robotparser to your crate root and your're good to go!

Examples

extern crate robotparser;

use robotparser::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.