dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-05-19 04:18:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
robotparser-rs/README.md
Maxim Zhukov abcb18dc97
Add support coverage via codecov + tarpaulin (#31) 
* workflows: add support coverage

Signed-off-by: Maxim Zhukov <mussitantesmortem@gmail.com>

* workflows: use HOME instead of '~' for fix macos cache build

Signed-off-by: Maxim Zhukov <mussitantesmortem@gmail.com>
2020-11-03 19:04:47 +08:00

1.4 KiB
Raw Blame History

robotparser-rs

GitHub Actions Coverage Status codecov Crates.io Dependency status

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.11"

Examples

use robotparser::http::RobotsTxtClient;
use robotparser::service::RobotsTxtService;
use reqwest::Client;
use url::Url;

fn main() {
    let client = Client::new();
    let robots_txt_url = Url::parse("https://www.python.org/robots.txt").unwrap();
    let robots_txt = client.fetch_robots_txt(robots_txt_url.origin()).unwrap().get_result();
    let fetch_url = Url::parse("https://www.python.org/robots.txt").unwrap();
    assert!(robots_txt.can_fetch("*", &fetch_url));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.