Rust Crawler Resources

Atra - The smaller way to crawl !!This read me will we reworked in a few days. Currently I am working on a better version and a wiki for the config fi

spider-rs The spider project ported to Node.js Getting Started npm i @spider-rs/spider-rs --save import { Website, pageTitle } from "@spider-rs/spider

🕷️ crawly A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules. 🚀 Features Concurren

🌊 seaward Installation cargo install seaward On NetBSD a pre-compiled binary is available from the official repositories. To install it, simply run:

crypto-msg-parser The parser library to parse messages from crypto-crawler. Architecture crypto-msg-parser is the parser library to parse messages fro

netrunner netrunner is a tool to help build, validate, & create archives for Spyglass lenses. Lenses are a simple set of rules that tell a crawler whi

carbonbot A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc. Run To quickly get started, cop

Rust library to detect bots using a user-agent string

Sponsored by: rebuilderd-debian-buildinfo-crawler This program parses the Packages.xz debian package index, attempts to discover the right buildinfo f

Broad web crawling is an activity of going through practically boundless web by starting from a set of locations(urls) and following outgoing links. Usually it doesn't matter where you start from as long as it has outgoing links to external domains.

url-crawler A configurable parallel web crawler, designed to crawl a website for content. Changelog Docs.rs Example extern crate url_crawler; use std: