dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-06-16 21:21:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
92 commits 1 branch 0 tags 185 KiB
Commit graph

11 commits

Author SHA1 Message Date
Laurent Arnoud
1474a8cce9
Switch to https for urls (#23) 2020-03-02 15:20:44 +08:00
svmk
2d19755779
Refactoring of robotparser-rs (#20) 
* Migrated sites into robotsparser file.

* Robots.txt refactoring.

* Migrated to new version of url and reqwest.
 2020-01-31 17:00:58 +08:00
Laurent Arnoud
141109aa15 Add deps status badge (#15) 2018-04-02 10:31:34 +08:00
messense
d6127d25bb
Bump version to 0.10.0 2017-12-02 23:52:48 +08:00
messense
832eb63e36
Update doc refs version 2017-08-09 13:21:28 +08:00
messense
63016cd97a
Update version to 0.8 in docs 2017-04-02 00:08:43 +08:00
messense
a94a5b34cc
Replace 0.6 with 0.7 in README and docstring 2016-10-21 14:39:08 +08:00
messense
7e3a47b7f2
Update version in README and documentation 2016-08-25 20:38:05 +08:00
messense
7f9d08b307 Better documentation 2016-03-06 16:31:44 +08:00
messense
179d6e5f87 Update README.md 2015-07-04 23:43:10 +08:00
messense
7c7f09012e Create README.md 2015-06-28 01:30:41 +08:00