dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-06-17 05:31:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
90 commits 1 branch 0 tags 185 KiB
Commit graph

12 commits

Author SHA1 Message Date
svmk
2d19755779
Refactoring of robotparser-rs (#20) 
* Migrated sites into robotsparser file.

* Robots.txt refactoring.

* Migrated to new version of url and reqwest.
 2020-01-31 17:00:58 +08:00
Laurent Arnoud
19bd3b4cf6
Use host_str instead domain for HostInternal 2018-03-11 22:05:14 +01:00
messense
8ede3b4218
Feature gate hyper 2016-10-02 13:17:15 +08:00
messense
17a36159a7 Add Request-rate support. Closes #4 2016-08-21 11:35:24 +08:00
Mikhail Svetov
626fa947e4 Added sitemap support. 2016-02-08 17:23:46 +07:00
Mikhail Svetov
02eedd6246 Improvements. 2016-01-13 21:18:07 +07:00
Mikhail Svetov
49a2b9c93d Added crawl delay feature. 2016-01-13 20:51:54 +07:00
messense
ac7d7aa8db Refactor some API to be more generic over Rust strings 2015-10-21 15:40:06 +08:00
messense
cbe515e761 Add RobotFileParser.read test case 2015-07-01 10:23:07 +08:00
messense
8e2944d8a4 Bug fixes and add more test cases 2015-07-01 00:01:15 +08:00
messense
e8928ec198 Bug fix and add more tests 2015-06-28 09:59:52 +08:00
messense
ece1e1f3c4 Initial commit. 2015-06-28 01:29:05 +08:00