A simple command line tool which quickly audits the Disallow entries of a site's robots.txt.

Related tags

Security tools penetration-testing pentesting penetration-testing-tools
Overview

Domo Arigato

A simple command line tool which quickly audits the Disallow entries of a site's robots.txt. Disallow entries can be used to stop search engines from indexing specific directories and files. Thus, sometimes it can be an easy way to find pages that it's worthwhile for the user to check out.

Domo will run through every Disallow entry, visit the page and return the HTTP Status Code. If any return 200, it means you'll be able to visit it.

You can also choose to search bing.com to see if the page has been indexed, and web.archive.org to see if it's been archived.

Usage

USAGE:
    domo [OPTIONS] --url <URL>

OPTIONS:
    -a, --archive      Search the URLs on web.archive.org and return the results
    -b, --bing         Search the URLs on Bing and return the results
    -h, --help         Print help information
    -o, --only200      Only show results with HTTP status 200
    -u, --url <URL>    URL to check the robots.txt
    -V, --version      Print version information

Example

root@kali:~/tools/domo# domo -abo -u targetsite.com

    ________                             _____        .__              __          
    \______ \   ____   _____   ____     /  _  \_______|__| _________ _/  |_  ____  
     |    |  \ /  _ \ /     \ /  _ \   /  /_\  \_  __ \  |/ ___\__  \\   __\/  _ \ 
     |    `   (  <_> )  Y Y  (  <_> ) /    |    \  | \/  / /_/  > __ \|  | (  <_> )
    /_______  /\____/|__|_|  /\____/  \____|__  /__|  |__\___  (____  /__|  \____/ 
            \/             \/                 \/        /_____/     \/             
            
Running Domo v1
Running @ targetsite.com, starting: 2023-04-04 17:34:58
       	 
http://targetsite.com/files/ 200 "OK"
http://targetsite.com/stats/ 200 "OK"
http://targetsite.com/icons/ 200 "OK"
http://targetsite.com/staff/ 200 "OK"
http://targetsite.com/primer/ 200 "OK"
http://targetsite.com/Unixhelp/ 200 "OK"
http://targetsite.com/HTML_Dictionary/ 200 "OK"
http://targetsite.com/webnews/ 200 "OK"

 -- 16 links have been analyzed and 8 of them are available.


Searching the Disallow entries on bing.com...

https://www.bing.com/search?q=site:targetsite.com/icons/ found

 -- 16 links have been analyzed and 1 of them are available.


Searching the Disallow entries on web.archive.org...

https://web.archive.org/web/*/targetsite.com/%7Echris/omc/ found
https://web.archive.org/web/*/targetsite.com/counter.html found
https://web.archive.org/web/*/targetsite.com/~chris/directions.html found
https://web.archive.org/web/*/targetsite.com/~chris/omc/ found
https://web.archive.org/web/*/targetsite.com/~chris/weird/ found
https://web.archive.org/web/*/targetsite.com/stats/ found
https://web.archive.org/web/*/targetsite.com/webnews/ found
https://web.archive.org/web/*/targetsite.com/%7Echris/weird/ found
https://web.archive.org/web/*/targetsite.com/files/ found
https://web.archive.org/web/*/targetsite.com/staff/ found
https://web.archive.org/web/*/targetsite.com/%7Echris/directions.html found
https://web.archive.org/web/*/targetsite.com/primer/ found
https://web.archive.org/web/*/targetsite.com/HTML_Dictionary/ found
https://web.archive.org/web/*/targetsite.com/cgi-bin/ found
https://web.archive.org/web/*/targetsite.com/Unixhelp/ found
https://web.archive.org/web/*/targetsite.com/icons/ found

 -- 16 links have been analyzed and 16 of them are available.
Finished in 1.31s

Disclaimer

This is intended to be used for your own servers or in contexts like pentesting and CTFs where you are authorised to be engaging with the server in this manner. If you choose to do anything else with it, that's your responsibility.

You might also like...
A simple scanner that loops through ips and checks if a minecraft server is running on port 25565

scanolotl Scanolotl is a simple scanner that loops through ips and checks if a minecraft server is running on port 25565. Scanolotl can also preform a

JustFr33z 3 Jul 28, 2022
simple multi-threaded port scanner written in rust
simple multi-threaded port scanner written in rust

knockson simple multi-threaded port scanner written in rust Install Using AUR https://aur.archlinux.org/packages/knockson-bin/ yay -Syu knockson-bin M

Josh Münte 4 Oct 5, 2022
A simple port scanner built using rust-lang

A simple port scanner built using rust-lang

Krisna Pranav 1 Nov 6, 2021
Simple verification of Rust programs via functional purification in Lean 2(!)
Simple verification of Rust programs via functional purification in Lean 2(!)

electrolysis About A tool for formally verifying Rust programs by transpiling them into definitions in the Lean theorem prover. Masters thesis: Simple

Sebastian Ullrich 300 Dec 11, 2022
Simple prepender virus written in Rust
Simple prepender virus written in Rust

Linux.Fe2O3 This is a POC ELF prepender written in Rust. I like writting prependers on languages that I'm learning and find interesting. As for the na

Guilherme Thomazi Bonicontro 91 Dec 9, 2022
A simple rust library for working with ZIP archives

rust-zip A simple rust library to read and write Zip archives, which is also my pet project for learning Rust. At the moment you can list the files in

Jorge Gorbe Moya 11 Aug 6, 2022
A simple allocator written in Rust that manages memory in fixed-size chunks.

Simple Chunk Allocator A simple no_std allocator written in Rust that manages memory in fixed-size chunks/blocks. Useful for basic no_std binaries whe

Philipp Schuster 7 Aug 8, 2022
A simple port sniffer(scanner) implementation with 🦀

A simple port sniffer(scanner) implementation with 🦀 Install from crates.io crago install ports-sniffer From aur: yay -S ports-sniffer Arguments Argu

Anas Elgarhy 8 Oct 10, 2022
Texting Robots: A Rust native `robots.txt` parser with thorough unit testing

Texting Robots Crate texting_robots is a library for parsing robots.txt files. A key design goal of this crate is to have a thorough test suite tested

Stephen Merity 20 Aug 17, 2022
A native Rust port of Google's robots.txt parser and matcher C++ library.

robotstxt A native Rust port of Google's robots.txt parser and matcher C++ library. Native Rust port, no third-part crate dependency Zero unsafe code

Folyd 72 Dec 11, 2022
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

rbdt 🚨 🚨 🚨 🚨 rbdt is a work in progress, currently being extracted out of another (private) project for the purpose of open sourcing and better so

Knuckleheads' Club 0 Nov 9, 2021
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.

🕷️ crawly A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules. 🚀 Features Concurren

CrystalSoft 5 Aug 29, 2023
interactcli-rs is a command-line program framework used to solve the problem of the integration of command-line and interactive modes, including functions such as unification of command-line interactive modes and sub-command prompts. The framework integrates clap and shellwords.

interactcli-rs 简体中文 interactcli-rs is a command-line program framework used to solve the problem of the integration of command-line and interactive mo

JiaShiwen 12 Nov 5, 2022
rsv is a command line tool to deal with small and big CSV, TXT, EXCEL files (especially 10G)

csv, excel toolkit written in Rust rsv is a command line tool to deal with small and big CSV, TXT, EXCEL files (especially 10G). rsv has following fe

Zhuang Dai 39 Jan 30, 2023
Quickly find all blackhole directories with a huge amount of filesystem entries in a flat structure
Quickly find all blackhole directories with a huge amount of filesystem entries in a flat structure

findlargedir About Findlargedir is a tool specifically written to help quickly identify "black hole" directories on an any filesystem having more than

Dinko Korunic 24 Jan 1, 2023
A small program which makes a rofi game launcher menu possible by creating .desktop entries for games
A small program which makes a rofi game launcher menu possible by creating .desktop entries for games

rofi-games A small program which makes a `rofi` game launcher menu possible by creating `.desktop` entries for games Installation Manual Clone repo: g

Rolv Apneseth 20 May 4, 2023
A command-line downloader for sites archived on the Wayback Machine

This is a small command-line utility I wrote to help with browsing archived websites from the Wayback Machine, which can sometimes be pretty slow.

Jonas Schievink 7 Oct 18, 2022
locdev is a handy CLI tool that simplifies the process of adding, removing, and listing entries in the hosts file.

locdev 🛠️ locdev is a handy CLI tool that simplifies the process of adding, removing, and listing entries in the hosts file. You no longer need to de

Nick Rempel 20 Jun 5, 2023
fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find
fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find

fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find. While it does not aim to support all of find's powerful functionality, it provides sensible (opinionated) defaults for a majority of use cases.

David Peter 25.9k Jan 9, 2023
Owner
Ember Hext
I like programming, hacking, tinkering, cybersec, physec | I forgot to make a career out of any of it, someone hire me
Ember Hext
GitHub
Checks your files for existence of Unicode BIDI characters which can be misused for supply chain attacks. See CVE-2021-42574

BIDI Character Detector This tool checks your files for existence of Unicode BIDI characters which can be misused for supply chain attacks to mitigate

null 5 Aug 26, 2022
A fast, simple, recursive content discovery tool written in Rust.

A simple, fast, recursive content discovery tool written in Rust ?? Releases ✨ Example Usage ✨ Contributing ✨ Documentation ?? ?? What the heck is a f

epi 3.6k Dec 30, 2022
subscout is a simple, nimble subdomain enumeration tool written in Rust language

subscout is a simple, nimble subdomain enumeration tool written in Rust language. It is designed to help bug bounty hunters, security professionals and penetration testers discover subdomains of a given target domain.

Dom Sec 5 Apr 5, 2023
link is a command and control framework written in rust

link link is a command and control framework written in rust. Currently in alpha. Table of Contents Introduction Features Feedback Build Process Ackno

null 427 Dec 24, 2022
A Comprehensive Web Fuzzer and Content Discovery Tool

rustbuster A Comprehensive Web Fuzzer and Content Discovery Tool Introduction Check the blog post: Introducing Rustbuster — A Comprehensive Web Fuzzer

Francesco Soncina 467 Dec 26, 2022
ctfsak is a tool to speed up common operations needed during CTFs

ctfsak (CTF Swiss Army Knife) This is a tool to help saving time during CTFs, where it's common to have to do a lot of encoding/decoding, encrypting/d

null 1 Dec 1, 2022
Binary coverage tool without binary modification for Windows

Summary Mesos is a tool to gather binary code coverage on all user-land Windows targets without need for source or recompilation. It also provides an

null 381 Dec 22, 2022
LLVM-CBE is a C-backend for LLVM, i.e. a tool that turns LLVM bitcode 1 into C

LLVM-CBE is a C-backend for LLVM, i.e. a tool that turns LLVM bitcode 1 into C. It requires to be built near LLVM, which was found to be so heavy dependency that shipping it as Cargo crate would be absurd.

Dmitrii - Demenev 2 May 26, 2022
A simple password manager written in Rust

ripasso A simple password manager written in Rust. The root crate ripasso is a library for accessing and decrypting passwords stored in pass format (G

Joakim Lundborg 548 Dec 26, 2022
A simple menu to keep all your most used one-liners and scripts in one place

Dama Desktop Agnostic Menu Aggregate This program aims to be a hackable, easy to use menu that can be paired to lightweight window managers in order t

null 47 Jul 23, 2022