mirror of
https://github.com/danbulant/robotparser-rs
synced 2026-06-19 06:31:06 +00:00
2d19755779
* Migrated sites into robotsparser file. * Robots.txt refactoring. * Migrated to new version of url and reqwest.
4 lines
No EOL
33 B
Text
4 lines
No EOL
33 B
Text
target
|
|
Cargo.lock
|
|
.vscode/
|
|
.idea/ |