2 Repositories
Rust crawling Libraries
🗄️ A simple CLI for converting WARC to Parquet.
warc-parquet 🗄️ A utility for converting WARC to Parquet. 📦 Install The binary may be installed via cargo: $ cargo install warc-parquet To use the c
89 Jun 5, 2023
Rust crate for configurable parallel web crawling, designed to crawl for content
url-crawler A configurable parallel web crawler, designed to crawl a website for content. Changelog Docs.rs Example extern crate url_crawler; use std:
56 Aug 22, 2021