dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-06-17 21:51:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
95 commits 1 branch 0 tags 185 KiB
Commit graph

40 commits

Author SHA1 Message Date
Laurent Arnoud
6ba403aab9
Inital github-actions (#25) 
* Inital github-actions

most taken from starship project
ref #21

* rustfmt config remove unknown configuration options

* Run rustfmt

* clippy: use any instead of find..is_some

* clippy: Remove the `clone` call: `self.crawl_delay`

* Clippy fixes

* Rustfmt fixes

* clippy: fix dont need to add `&` to all patterns

* clippy: fix needless `fn main` in doctest

* clippy: fix if-then-else expression returns a bool literal

* clippy: fix very complex type BoxFuture response

* clippy: fix variable `line_no` is used as a loop counter

* clippy: dereference the expression on tests

* clippy: fix assert(true) will be optimized out by the compiler

* github: name workflow
 2020-03-30 09:32:32 +08:00
Laurent Arnoud
df49f6bcf0
Error handling (#24) 
* Add test when url is invalid and panic

* Initial error handling

ref https://github.com/messense/robotparser-rs/issues/22

* Rename ErrorKind::HttpClient => ErrorKind::Http

* Implement std::error::Error and rename to Error
 2020-03-08 20:33:34 +08:00
Laurent Arnoud
1474a8cce9
Switch to https for urls (#23) 2020-03-02 15:20:44 +08:00
svmk
2d19755779
Refactoring of robotparser-rs (#20) 
* Migrated sites into robotsparser file.

* Robots.txt refactoring.

* Migrated to new version of url and reqwest.
 2020-01-31 17:00:58 +08:00
messense
2205ef3b5d Remove clippy for now 2018-09-22 11:33:28 +08:00
Laurent Arnoud
13e60c226a Update reqwest to 0.9 (#16) 2018-09-22 11:30:41 +08:00
Laurent Arnoud
19bd3b4cf6
Use host_str instead domain for HostInternal 2018-03-11 22:05:14 +01:00
messense
d6127d25bb
Bump version to 0.10.0 2017-12-02 23:52:48 +08:00
messense
5ad2785e07
Fix some clippy warnings 2017-12-02 23:51:52 +08:00
Laurent Arnoud
79471efcc2
Update reqwest to 0.8 2017-12-02 16:39:25 +01:00
messense
832eb63e36
Update doc refs version 2017-08-09 13:21:28 +08:00
messense
44eb658238
Upgrade reqwest to 0.7 2017-08-09 13:16:27 +08:00
messense
63016cd97a
Update version to 0.8 in docs 2017-04-02 00:08:43 +08:00
messense
b4b0752e96
Upgrade reqwest to 0.5 2017-04-02 00:05:16 +08:00
Laurent Arnoud
9f5b4c27d4
Use request higher level Client crate for openssl updates 
see https://github.com/hyperium/hyper/issues/907#issuecomment-255509020
 2016-11-12 18:22:55 +01:00
messense
a94a5b34cc
Replace 0.6 with 0.7 in README and docstring 2016-10-21 14:39:08 +08:00
messense
8ede3b4218
Feature gate hyper 2016-10-02 13:17:15 +08:00
messense
7e3a47b7f2
Update version in README and documentation 2016-08-25 20:38:05 +08:00
messense
553b47f7e7
Change from_response method signature 2016-08-25 18:01:17 +08:00
Ernestas Poskus
36c90ec169 Add method read_from_response 2016-08-25 09:59:35 +03:00
messense
17a36159a7 Add Request-rate support. Closes #4 2016-08-21 11:35:24 +08:00
messense
a5254dbc52 style: rustfmt source codes 2016-08-21 10:58:18 +08:00
Laurent Arnoud
e3663eb257
Use std::time instead of deprecated crate time 2016-08-20 20:24:40 +02:00
Laurent Arnoud
8b50c90f94
Set UserAgent as robotparser-rs 2016-05-04 23:43:32 +02:00
messense
caca158589 Use Cow instead of owned String 2016-04-27 11:26:28 +08:00
messense
a552d99210
Do not deny all clippy issues 
Signed-off-by: messense <messense@icloud.com>
 2016-04-25 13:23:01 +08:00
messense
77b604d2af
Bump version to 0.4.3 
Signed-off-by: messense <messense@icloud.com>
 2016-04-25 12:33:15 +08:00
messense
74fddfd5cb Add implementation of Default for Entry 2016-03-17 13:29:58 +08:00
messense
7f9d08b307 Better documentation 2016-03-06 16:31:44 +08:00
messense
2d9c028dfa Refactor codes like clippy suggested 2016-03-06 16:19:58 +08:00
Mikhail Svetov
fcfb6d9df6 Improvements. 2016-02-08 17:25:27 +07:00
Mikhail Svetov
626fa947e4 Added sitemap support. 2016-02-08 17:23:46 +07:00
Mikhail Svetov
02eedd6246 Improvements. 2016-01-13 21:18:07 +07:00
Mikhail Svetov
49a2b9c93d Added crawl delay feature. 2016-01-13 20:51:54 +07:00
messense
ac7d7aa8db Refactor some API to be more generic over Rust strings 2015-10-21 15:40:06 +08:00
messense
160ed940ae Use join instead of connect 2015-08-12 17:53:28 +08:00
messense
8e2944d8a4 Bug fixes and add more test cases 2015-07-01 00:01:15 +08:00
messense
e8928ec198 Bug fix and add more tests 2015-06-28 09:59:52 +08:00
messense
6153773f24 Correct the usage of RefCell.borrow 2015-06-28 01:39:03 +08:00
messense
ece1e1f3c4 Initial commit. 2015-06-28 01:29:05 +08:00