dan/robotparser-rs
1
0
Fork 0
mirror of https://github.com/danbulant/robotparser-rs synced 2026-05-19 04:18:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
94 commits 1 branch 0 tags 185 KiB
Commit graph

94 commits

This branch
This branch
All branches
Author SHA1 Message Date
messense
85c108ef43 Remove Travis & Appveyor 2020-03-30 09:36:57 +08:00
Laurent Arnoud
6ba403aab9
Inital github-actions (#25) 
* Inital github-actions

most taken from starship project
ref #21

* rustfmt config remove unknown configuration options

* Run rustfmt

* clippy: use any instead of find..is_some

* clippy: Remove the `clone` call: `self.crawl_delay`

* Clippy fixes

* Rustfmt fixes

* clippy: fix dont need to add `&` to all patterns

* clippy: fix needless `fn main` in doctest

* clippy: fix if-then-else expression returns a bool literal

* clippy: fix very complex type BoxFuture response

* clippy: fix variable `line_no` is used as a loop counter

* clippy: dereference the expression on tests

* clippy: fix assert(true) will be optimized out by the compiler

* github: name workflow
 2020-03-30 09:32:32 +08:00
Laurent Arnoud
df49f6bcf0
Error handling (#24) 
* Add test when url is invalid and panic

* Initial error handling

ref https://github.com/messense/robotparser-rs/issues/22

* Rename ErrorKind::HttpClient => ErrorKind::Http

* Implement std::error::Error and rename to Error
 2020-03-08 20:33:34 +08:00
Laurent Arnoud
1474a8cce9
Switch to https for urls (#23) 2020-03-02 15:20:44 +08:00
svmk
2d19755779
Refactoring of robotparser-rs (#20) 
* Migrated sites into robotsparser file.

* Robots.txt refactoring.

* Migrated to new version of url and reqwest.
 2020-01-31 17:00:58 +08:00
messense
cb7df85b83 Bump version to 0.10.2 2018-09-22 11:47:15 +08:00
messense
86a483be75 Use openssl 1.0.2p on Appveyor 2018-09-22 11:44:43 +08:00
messense
2205ef3b5d Remove clippy for now 2018-09-22 11:33:28 +08:00
Laurent Arnoud
13e60c226a Update reqwest to 0.9 (#16) 2018-09-22 11:30:41 +08:00
messense
ed0f892875
Use openssl 1.0.2o open AppVeyor 2018-04-02 13:17:33 +08:00
Laurent Arnoud
141109aa15 Add deps status badge (#15) 2018-04-02 10:31:34 +08:00
messense
9d46f5ac99
Bump version to 0.10.1 2018-03-12 09:29:22 +08:00
messense
4688c72346
Merge pull request #14 from spk/fix-127.0.0.1-host-panic 
Use host_str instead domain for HostInternal
 2018-03-12 09:23:09 +08:00
Laurent Arnoud
19bd3b4cf6
Use host_str instead domain for HostInternal 2018-03-11 22:05:14 +01:00
messense
953d1f0c0b
Again fix AppVeyor 2018-01-11 13:43:24 +08:00
messense
6854290e18
Use openssl 1.0.2n on Appveyor 2018-01-11 12:31:40 +08:00
messense
6adf740663
Use url crate 1.x version 2018-01-11 11:28:52 +08:00
messense
d6127d25bb
Bump version to 0.10.0 2017-12-02 23:52:48 +08:00
messense
5ad2785e07
Fix some clippy warnings 2017-12-02 23:51:52 +08:00
messense
3b24290e5c
Appveyor use openssl 1.0.2m 2017-12-02 23:49:41 +08:00
messense
450cafadfd
Merge pull request #13 from spk/update-request 
Update reqwest to 0.8
 2017-12-02 23:47:59 +08:00
Laurent Arnoud
79471efcc2
Update reqwest to 0.8 2017-12-02 16:39:25 +01:00
messense
832eb63e36
Update doc refs version 2017-08-09 13:21:28 +08:00
messense
8fdbd92042
Bump version to 0.9.0 2017-08-09 13:17:06 +08:00
messense
44eb658238
Upgrade reqwest to 0.7 2017-08-09 13:16:27 +08:00
messense
63016cd97a
Update version to 0.8 in docs 2017-04-02 00:08:43 +08:00
messense
9f57b38e02
Bump version to 0.8.2 2017-04-02 00:05:43 +08:00
messense
b4b0752e96
Upgrade reqwest to 0.5 2017-04-02 00:05:16 +08:00
messense
cd4d32bf28
Bump version to 0.8.1 2017-03-13 23:10:57 +08:00
messense
b27c5544eb Merge pull request #12 from spk/less-strict-reqwest-vers 
Less strict version on reqwest
 2017-03-12 22:11:27 +08:00
messense
426365bc9d
Set RUST_BACKTRACE=1 on Travis CI 2017-03-12 22:10:33 +08:00
Laurent Arnoud
be7f58adec
Less strict version on reqwest 
Using MINOR from semver
 2017-03-12 15:03:38 +01:00
messense
101f926adc
Bump version to 0.8.0 2016-11-14 16:14:40 +08:00
messense
e3b9ca6c6b Merge pull request #11 from spk/use-request-http-client 
Use request higher level Client crate for openssl updates
 2016-11-14 16:10:18 +08:00
Laurent Arnoud
9f5b4c27d4
Use request higher level Client crate for openssl updates 
see https://github.com/hyperium/hyper/issues/907#issuecomment-255509020
 2016-11-12 18:22:55 +01:00
messense
a94a5b34cc
Replace 0.6 with 0.7 in README and docstring 2016-10-21 14:39:08 +08:00
messense
29c009114c
Bump version to 0.7.0 2016-10-05 13:30:10 +08:00
messense
760a4e723b Merge pull request #10 from messense/feature/feature-gate-hyper 
Feature gate hyper
 2016-10-05 13:28:45 +08:00
messense
66661fda1e
Upgrade Win32 OpenSSL on Appveyor CI 2016-10-05 10:50:22 +08:00
messense
8ede3b4218
Feature gate hyper 2016-10-02 13:17:15 +08:00
messense
7e3a47b7f2
Update version in README and documentation 2016-08-25 20:38:05 +08:00
messense
6bebf15aca
Bump version to 0.6.0 2016-08-25 20:30:12 +08:00
messense
c32933db75 Merge pull request #8 from messense/feature/from-response-mutability 
Change from_response method signature
 2016-08-25 18:22:03 +08:00
messense
553b47f7e7
Change from_response method signature 2016-08-25 18:01:17 +08:00
messense
96a6593723 Merge pull request #7 from ernestas-poskus/features/add_method_read_from_response 
Add method from_response
 2016-08-25 17:52:49 +08:00
Ernestas Poskus
36c90ec169 Add method read_from_response 2016-08-25 09:59:35 +03:00
messense
d0351ecd2a Bump version to 0.5.0 2016-08-21 21:52:53 +08:00
messense
a732ad5085 Merge pull request #6 from messense/request-rate 
Add Request-rate support
 2016-08-21 21:42:27 +08:00
messense
17a36159a7 Add Request-rate support. Closes #4 2016-08-21 11:35:24 +08:00
messense
384ae57caa Add .vscode/ to .gitignore 2016-08-21 11:00:21 +08:00