dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-06-16 13:11:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
92 commits 1 branch 0 tags 185 KiB
Commit graph

3 commits

Author SHA1 Message Date
Laurent Arnoud
df49f6bcf0
Error handling (#24) 
* Add test when url is invalid and panic

* Initial error handling

ref https://github.com/messense/robotparser-rs/issues/22

* Rename ErrorKind::HttpClient => ErrorKind::Http

* Implement std::error::Error and rename to Error
 2020-03-08 20:33:34 +08:00
Laurent Arnoud
1474a8cce9
Switch to https for urls (#23) 2020-03-02 15:20:44 +08:00
svmk
2d19755779
Refactoring of robotparser-rs (#20) 
* Migrated sites into robotsparser file.

* Robots.txt refactoring.

* Migrated to new version of url and reqwest.
 2020-01-31 17:00:58 +08:00