9 commits

Author	SHA1	Message	Date
messense	17a36159a7	Add Request-rate support. Closes #4	2016-08-21 11:35:24 +08:00
Mikhail Svetov	626fa947e4	Added sitemap support.	2016-02-08 17:23:46 +07:00
Mikhail Svetov	02eedd6246	Improvements.	2016-01-13 21:18:07 +07:00
Mikhail Svetov	49a2b9c93d	Added crawl delay feature.	2016-01-13 20:51:54 +07:00
messense	ac7d7aa8db	Refactor some API to be more generic over Rust strings	2015-10-21 15:40:06 +08:00
messense	cbe515e761	Add RobotFileParser.read test case	2015-07-01 10:23:07 +08:00
messense	8e2944d8a4	Bug fixes and add more test cases	2015-07-01 00:01:15 +08:00
messense	e8928ec198	Bug fix and add more tests	2015-06-28 09:59:52 +08:00
messense	ece1e1f3c4	Initial commit.	2015-06-28 01:29:05 +08:00