mirror of
https://github.com/danbulant/robotparser-rs
synced 2026-06-14 04:01:22 +00:00
No description
6ba403aab9
* Inital github-actions most taken from starship project ref #21 * rustfmt config remove unknown configuration options * Run rustfmt * clippy: use any instead of find..is_some * clippy: Remove the `clone` call: `self.crawl_delay` * Clippy fixes * Rustfmt fixes * clippy: fix dont need to add `&` to all patterns * clippy: fix needless `fn main` in doctest * clippy: fix if-then-else expression returns a bool literal * clippy: fix very complex type BoxFuture response * clippy: fix variable `line_no` is used as a loop counter * clippy: dereference the expression on tests * clippy: fix assert(true) will be optimized out by the compiler * github: name workflow |
||
|---|---|---|
| .github/workflows | ||
| src | ||
| tests | ||
| .gitignore | ||
| .travis.yml | ||
| appveyor.yml | ||
| Cargo.toml | ||
| LICENSE | ||
| README.md | ||
| rustfmt.toml | ||
robotparser-rs
robots.txt parser for Rust.
Installation
Add it to your Cargo.toml:
[dependencies]
robotparser = "0.11"
Add extern crate robotparser to your crate root and your're good to go!
Examples
use robotparser::http::RobotsTxtClient;
use robotparser::service::RobotsTxtService;
use reqwest::Client;
use url::Url;
fn main() {
let client = Client::new();
let robots_txt_url = Url::parse("https://www.python.org/robots.txt").unwrap();
let robots_txt = client.fetch_robots_txt(robots_txt_url.origin()).unwrap().get_result();
let fetch_url = Url::parse("https://www.python.org/robots.txt").unwrap();
assert!(robots_txt.can_fetch("*", &fetch_url));
}
License
This work is released under the MIT license. A copy of the license is provided in the LICENSE file.