Rust Url-crawler Resources

Atra - The smaller way to crawl !!This read me will we reworked in a few days. Currently I am working on a better version and a wiki for the config fi

spider-rs The spider project ported to Node.js Getting Started npm i @spider-rs/spider-rs --save import { Website, pageTitle } from "@spider-rs/spider

Tyupy Get URL(s) title in any format. Features Get URL(s) title in any format*. Markdown Org-mode Fancy error message and colorful output. Cross-platf

when3meet 🦀 The Rust when2meet CLI Install | Usage | Contributing & Issues | Docs Built with ❤️ and 🦀 by Garrett Ladley Install cargo install when3m

bref A modest URL shortener for personal usage. Shortens URLs to a 6 characters-length key that is generated based on current time. Why? Why not? Usag

🕷️ crawly A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules. 🚀 Features Concurren

🌊 seaward Installation cargo install seaward On NetBSD a pre-compiled binary is available from the official repositories. To install it, simply run:

A gateway that proxies signed requests to other APIs. Check the docs for more info. If you are looking for the managed version checkout this link http

pathbuster A path-normalization pentesting tool Todos • Installation • Usage • Examples • Contributing • License • Join Discord Todos Implement --filt

enum-dir 一款快速枚举目录的小工具，使用 Rust 编写，扫起来真的很快。 用于发现目标站点中可能存在的路径信息，同时支持字典模式和暴力枚举模式。 本工具仅用于学习 Rust 以及 Rust 协程 Tokio 使用，严禁用于非法用途。 如果使用本工具从事违法犯罪活动，造成的任何后果，本人不承

crypto-msg-parser The parser library to parse messages from crypto-crawler. Architecture crypto-msg-parser is the parser library to parse messages fro

netrunner netrunner is a tool to help build, validate, & create archives for Spyglass lenses. Lenses are a simple set of rules that tell a crawler whi

urlchecker A simple url checker for finding fraud url(s) or nearest url while being fast (threading) Eg:- use std::collections::HashMap; use urlchecke

Redirector a redirector written in rust intended for permanent human-readable redirects. The idea was semi-inspired by the book Software Engineering a

Shurly Shurly, this is a URL shortener with API management Features Management of destinations through a REST'ish API Permanent/temporary redirects; p

quad-url This is the crate to work with URL and open links in miniquad/macroquad environment. Web demo. Usage Add this to your Cargo.toml dependencies

emojied Shorten your URLs with emojis! Features Well, shorten your URLs! Customize what emoji to use. e.g Want to use an eggplant emoji? Sure, as long

Pup A command line tool aiming to upload the local image used in your markdown file to the GitHub repo and replace the local file path with the return

imgs - CLI Webscrapper image url web-scrapper written in rust This is a cli application completely written in rust, it aim is to scrap image-url from

carbonbot A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc. Run To quickly get started, cop

Rust library to detect bots using a user-agent string

Sponsored by: rebuilderd-debian-buildinfo-crawler This program parses the Packages.xz debian package index, attempts to discover the right buildinfo f

pb Build $ cargo build --release Environment Variables PB_DATA: /some/path (Default: ./pb_data) PB_SITE: Url of your site. (Default: http://localhost:

Broad web crawling is an activity of going through practically boundless web by starting from a set of locations(urls) and following outgoing links. Usually it doesn't matter where you start from as long as it has outgoing links to external domains.

Obsidian Plugin: Convert a URL into markdown Transforms a URL to markdown view if the website allows it. Installation Available in the community plugi

Clink Have you ever pasted a link in messenger and deleted all those fbclid, utm_source, and so on, GET params by hands? Clink does that for you. It s

urlshortener-rs A very simple urlshortener for Rust. This library aims to implement as much URL shortener services as possible and to provide an inter

inquerest Inquerest can parse complex url query into a SQL abstract syntax tree. Example this url: /person?age=lt.42&(student=eq.true|gender=eq.'M')&

url-scraper Rust crate for scraping URLs from HTML pages. Example extern crate url_scraper; use url_scraper::UrlScraper; fn main() { let director

url-crawler A configurable parallel web crawler, designed to crawl a website for content. Changelog Docs.rs Example extern crate url_crawler; use std:

A simple, fast, recursive content discovery tool written in Rust 🦀 Releases ✨ Example Usage ✨ Contributing ✨ Documentation 🦀 😕 What the heck is a f