2 Repositories
Rust robotstxt Libraries
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
rbdt 🚨 🚨 🚨 🚨 rbdt is a work in progress, currently being extracted out of another (private) project for the purpose of open sourcing and better so
0 Nov 9, 2021
A native Rust port of Google's robots.txt parser and matcher C++ library.
robotstxt A native Rust port of Google's robots.txt parser and matcher C++ library. Native Rust port, no third-part crate dependency Zero unsafe code
72 Dec 11, 2022