mirror of
https://github.com/danbulant/robotparser-rs
synced 2026-05-19 04:18:42 +00:00
abcb18dc97
* workflows: add support coverage Signed-off-by: Maxim Zhukov <mussitantesmortem@gmail.com> * workflows: use HOME instead of '~' for fix macos cache build Signed-off-by: Maxim Zhukov <mussitantesmortem@gmail.com>
42 lines
1.4 KiB
Markdown
42 lines
1.4 KiB
Markdown
# robotparser-rs
|
|
|
|
[](https://github.com/messense/robotparser-rs/actions?query=workflow%3ACI)
|
|
[](https://coveralls.io/r/messense/robotparser-rs)
|
|
[](https://codecov.io/gh/messense/robotparser-rs)
|
|
[](https://crates.io/crates/robotparser)
|
|
[](https://deps.rs/repo/github/messense/robotparser-rs)
|
|
|
|
robots.txt parser for Rust.
|
|
|
|
|
|
## Installation
|
|
|
|
Add it to your ``Cargo.toml``:
|
|
|
|
```toml
|
|
[dependencies]
|
|
robotparser = "0.11"
|
|
```
|
|
|
|
|
|
## Examples
|
|
|
|
```rust
|
|
use robotparser::http::RobotsTxtClient;
|
|
use robotparser::service::RobotsTxtService;
|
|
use reqwest::Client;
|
|
use url::Url;
|
|
|
|
fn main() {
|
|
let client = Client::new();
|
|
let robots_txt_url = Url::parse("https://www.python.org/robots.txt").unwrap();
|
|
let robots_txt = client.fetch_robots_txt(robots_txt_url.origin()).unwrap().get_result();
|
|
let fetch_url = Url::parse("https://www.python.org/robots.txt").unwrap();
|
|
assert!(robots_txt.can_fetch("*", &fetch_url));
|
|
}
|
|
```
|
|
|
|
|
|
## License
|
|
|
|
This work is released under the MIT license. A copy of the license is provided in the [LICENSE](./LICENSE) file.
|