A nginx log explorer

Overview

Latest Version MIT Chat on Miaou

Rhit reads your nginx log files in their standard location(even gzipped), does some analysis and tells you about it in pretty tables in your console, storing and polluting nothing.

It lets you filter hits by dates, status, referers or paths, and does trend analysis.

And it's fast enough (about one second per million lines) so you can iteratively try queries to build your insight.

Here's looking at dates and trends on January hits with status 2xx and 3xx:

intro

Summary

Installation

Rhit is only tested on linux but is expected to work on Mac.

From source

You need the Rust toolchain. Do

cargo install rhit

From binaries

You may download linux binaries from https://dystroy.org/rhit/download.

Basic Usage

If rhit is on the server, and the logs are at their usual location:

rhit

(you may have to prefix with sudo to read the files in /var/log)

Tell rhit what files to open:

rhit ~/trav/nginx-logs

Filtering

Filter on paths

Filtering can be quite simple:

rhit -p download

But the syntax allows for much more interesting queries.

You may use a regular expression.

For example when I want to see all downloads of broot:

rhit -p '^/download/.*broot(.exe)?$'

You may negate expressions with a !.

For example, I have many paths which are just a number (eg /12345) and If I want to filter them, I can do

rhit -p '!^/\d+$'

(remember to use simple quotes and not double quotes to not have your shell interpret the expression)

Separating filters with a comma is an easy way to do a "AND".

If I want to get paths which are neither broot nor just a number, I'll do

rhit -p '!^/\d+$,!broot'

If I want to get all paths containing a digit, but not just a number, and not broot, I do

rhit -p '!^/\d+$,!broot,\d'

For a more complex logic, switch to binary expressions with parentheses and logic operators &, | and !.

For example to get all paths containing dystroy or blog but not broot:

rhit -p '( dystroy | blog ) & !broot'

(add spaces inside parenthesis to avoid them being understood as part of a regular expression)

To get all paths containing dystroy but neither blog, nor space nor any 4 digits numbers:

rhit -p 'dystroy & !( \d{4} | space | blog )'

Filter on referer

rhit -r reddit

As for the path, you may use a complex expression.

Only show a specific day

rhit -d 12/25

This shows only Christmas hits, assuming all the hits are from the same year. If the log contains several years, you need to precise it, eg rhit -d 2020/12/25. Symmetrically, you may omit the month if it's not ambiguous: rhit -d 25.

Only show a period:

rhit -d 2020/12/25-2021/01/03
rhit -d 2020/12
rhit -d 2020
rhit -d '>2020/12/25'
rhit -d '!2020/12/25'
rhit -d '<12/25'

Filter by status

The syntax is quite versatile:

rhit -s 404
rhit -s 5xx
rhit -s 3xx,410-421
rhit -s 301-305
rhit -s '!404'
rhit -s '4xx,!404'

Filter by remote IP address

rhit -i 123.123.123.123
rhit -i !123.123.123.123

Combine filters

You can use several arguments.

For example, to get all paths resulting in a 404 but not the robots.txt (which are legit queries) or the /crashy path:

mixed-filtering

Choose what to show

Fields

The displayed fields can be chosen with the -f argument.

Default fields: date,status,ref,path

Available fields: date,method,status,ip,ref,path

For example to only show remote IP adresses, statuses, and referers:

rhit -f ip,status,ref

status and addresses

Detail Level

Table lengths is decided with the -l argument.

Use rhit -l 0 to have just a few lines in the various tables, and rhit -l 5 for huge tables. Default value is 1.

Path filtering

By default the paths tables don't show "resources" like images, CSS, JS. If you don't want this filtering, use the --all argument:

rhit -a

Key

The measure used for sorting, histograms, and trends is either hits (default) or bytes (bytes in the response).

It's highlighted in pink in the report.

You set it with the --key argument:

hits or bytes

Changes

Use the --changes (short: -c) argument so that Rhit shows you the paths, referers or remote addresses which are notably more popular or less popular.

Settings related to displayed fields and filtered values still apply.

Comments
  • cannot open any files, just get -- Error: no log file found in

    cannot open any files, just get -- Error: no log file found in "..."

    Used cargo install on MacOS and I can't seem to get this working. I tried a couple of files that seem like they should work and it just reports the error mentioned in the title. I even copied a test log message from one of the source files and pasted it into a file, but that doesn't work either.

    opened by tstack 8
  • Homebrew support and request github release

    Homebrew support and request github release

    πŸ‘‹ we have a pending PR of including this utility into the homebrew-core, it would be great that we can do some github release managements, so that the downstream packaging would be beneficial from it. Thanks!

    relates to:

    • https://github.com/Homebrew/homebrew-core/pull/81076
    opened by chenrui333 7
  • Can't specify multiple files

    Can't specify multiple files

    My log directory has files from different sub domains

    • default.access.log
    • user.access.log
    • etc

    I don't want all of them combined. I want to see one sub domain at a time.

    This works fine if there's only one file for a particular subdomain.

    rhit default.access.log
    

    But (after log rollover) there's multiple files for the same sub domain

    • default.access.log
    • default.access.log.1
    • etc.

    I can do one at a time, no problem, but I want a combined view of a single sub domain. I tried:

    rhit default.access*
    

    but I get Unrecognized argument: default.access.log.1

    So it seems rhit is finding the other file but is not treating it as a file to read, but rather like some other argument.

    Any ideas? Thanks

    opened by victor-david 4
  • Distribute binary releases on PyPI

    Distribute binary releases on PyPI

    I've recently added distribution of binary release on PyPI for roapi, I was wondering if you'd like to distribute rhit on PyPI. I can provide PR if it's OK.

    It will be easy to install rhit binary with pip which exists on most well known operating systems: pip install rhit.

    opened by messense 4
  • Do you want rhit on Windows ?

    Do you want rhit on Windows ?

    Rhit can be modified to compile on Windows (10+, with a modern terminal).

    But I'll only do it if there are enough potential users.

    So if you'd need to run rhit on Windows, tell me in this thread.

    question 
    opened by Canop 3
  • Ignore logrotate lines

    Ignore logrotate lines

    Some logrotates add lines when they rotate the logs. Using OpenBSD's newsyslog, for example, logs look like this:

    2021-03-21T00:00:01.881Z host newsyslog[56965]: logfile turned over
    [regular nginx log here]
    
    opened by copy 2
  • Option `--color`

    Option `--color`

    It would be nice to have an option --color like --color=yes|no|auto. auto would be if it's a terminal output color is used, if it's not color is not used. So program would default to auto.

    Why ? Cause sometime you want to log text to a file (or whatever) and you don't want escape character used to put color on terminal:

    For example:

    38^[[38;5;239mβ”‚^[[0m^[[38;5;204mβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–^[[0m        ^[[38;5;239mβ”‚^[[0m
    ^[[38;5;239mβ”‚^[[0m2021/01/27^[[38;5;239mβ”‚^[[0m  16K^[[38;5;239mβ”‚^[[0m  51^[[38;5;239mβ”‚^[[0m^[[38;5;204mβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž^[[0m    ^[[38;5;239mβ”‚^[[0m
    ^[[38;5;239mβ”‚^[[0m2021/01/28^[[38;5;239mβ”‚^[[0m 9088^[[38;5;239mβ”‚^[[0m  
    

    is unreadable.

    opened by Stargateur 2
  • Add instructions for installing on Arch Linux

    Add instructions for installing on Arch Linux

    Hello!

    I've been maintaining rhit in the Arch Linux community repository for a while now and I thought the instructions should be present in the documentation.

    ref: https://archlinux.org/packages/community/x86_64/rhit/

    opened by orhun 1
  • Support more output format

    Support more output format

    Hello brother, your tool is very good, which can improve the efficiency of operation and maintenance. Is there a plan to support the output of the analysis results in JSON or CSV, or the interface of the Prometheus reported.

    opened by fadewalk 3
  • Bot traffic

    Bot traffic

    I understand this is difficult / impossible to solve with perfect fidelity, but any option to try to filter out bot traffic, even just obvious stuff like "has a GoogleBot user agent," would be super useful. Right now I am using a grep pipeline to filter these out, anything built in to the tool would be better than that.

    opened by kevinburke 1
  • Multiple domain support

    Multiple domain support

    Thanks for building this tool.

    I run several different websites on the same VPS, and generally I'm interested in looking at traffic to one of them... is there a way to add a filter option to display summary stats for only log lines that match a specific Host field?

    opened by kevinburke 1
  • not the default of NGINX log, how to format it

    not the default of NGINX log, how to format it

    my nginx log like 2014/12/01 15:05:00|222.222.222.222|333.333.333.333|image|200|………………more then 60 field hao to use this project format these log can this function be realized?

    opened by kof8855 3
  • parse nginx log_format

    parse nginx log_format

    Right now rhit assumes the log format is the default one, which is quite common.

    Among the reasons I had not to interpret the configured log format:

    • it's painful to write an efficient enough variable parser
    • changing the configuration means the past isn't readable anymore, so it can't be perfect
    • the log_format variable lets you define unparsable and ambiguous log lines
    • you may define formats missing essential fields

    So, as long as it looks like I can have something working for 99% of users, using a few tricks and dynamic recognitions, I won't implement this issue.

    Nevertheless, I'm not sure not parsing the log_format will always be OK, hence this issue.

    opened by Canop 0
Releases(v1.7.1)
Owner
Canop
Denys SΓ©guret. Rust Remote Developer
Canop
A loki logger for the log facade

Loki Logger A loki logger for the log facade. Examples extern crate log; extern crate loki_logger; use log::LevelFilter; #[tokio::main] async fn main

Thomas Nicollet 11 Dec 24, 2022
rt-history: An RT-safe history log with error checking

rt-history: An RT-safe history log with error checking This is a bounded wait-free thread synchronization primitive which allows you to record the tim

Hadrien G. 3 Oct 11, 2022
Log for concurrent workloads, with support for atomic batches and in-order recovery

sharded-log A batch-oriented multi-threaded sharded log for workloads that occasionally flush logs into some other system. All batches have a 32-bit C

Komora 16 Nov 20, 2022
A cool log library built using rust-lang

RustLog A cool log library built using rust-lang Installation: Cargo.toml rustlog = { git = "https://github.com/krishpranav/rustlog" } log = "0.4.17"

Krisna Pranav 2 Jul 21, 2022
Log agent to replay time-stamped log stream

replayman Log agent to replay time-stamped log stream. Getting started Installation cargo install replayman Prepare for your data TSV with a header li

SpringQL 3 Dec 23, 2022
nginx bindings for Rust

nginx-rs This crate provides nginx bindings for Rust. Currently, only Linux is supported. How to Use Add nginx crate to Cargo.toml [dependencies] ngin

ArvanCloud 104 Jan 2, 2023
Generate nginx vhosts for mlcdf/cc-reverve-proxy

nvhosts Generate nginx vhosts given a configuration file. Made to work with mlcdf/cc-reverse-proxy. Usage Usage: nvhosts [-c <config>] [--example] [-V

Maxime Le Conte des Floris 0 Feb 28, 2022
Rust binding for NGINX

Rust for NGINX Rust bindings and wrappers for NGINX. Can be used for building dynamic modules and hacking NGINX using rust. Production Status This ver

NGINX, Inc. 291 Jan 2, 2023
Prometheus instrumentation service for the NGINX RTMP module.

nginx-rtmp-exporter Prometheus instrumentation service for the NGINX RTMP module. Usage nginx-rtmp-exporter [OPTIONS] --scrape-url <SCRAPE_URL> O

kaylen ✨ 2 Jul 3, 2022
A TCP proxy using HTTP - Reach SSH behind a Nginx reverse proxy

?? TCP over HTTP ?? The Questions ?? What does it do? You can proxy TCP traffic over HTTP. A basic setup would be: [Your TCP target] <--TCP-- [Exit No

Julian 185 Dec 15, 2022
Modern Drop-in Replacement for Nginx AutoIndex / FancyIndex!

MeowIndex A cute, feature-rich file listing module to replace nginx's autoindex / fancyindex. Features List files Show file icons Clickable, length-sa

Hykilpikonna 4 Feb 25, 2023
Simple and minimalist forward auth service intended for use with reverse proxies (Traefik, Caddy, nginx, etc)

nforwardauth nforwardauth is an extremely lightweight, blazing fast forward auth service that lets you use a single authentication middleware for all

Tony Duco 22 Apr 7, 2023
Fast fail2ban-like tools for parsing nginx logs

Fast2ban This is simple fail2ban-like replacement written in Rust. Usage: ./fast2ban # reads default config.toml from current directory ./fast2ban <co

null 36 May 10, 2023
A hackable, minimal, fast TUI file explorer, stealing ideas from nnn and fzf.

xplr A hackable, minimal, fast TUI file explorer, stealing ideas from nnn and fzf. [Quickstart] [Features] [Plugins] [Documentation] [Upgrade Guide] [

Arijit Basu 2.6k Jan 1, 2023
πŸ”΅πŸŸ  Portal Explorer β€” web visualization of mind-blowing portals using ray-tracing.

In Portal Explorer you can view how interesting portals are constructed, and visually explore their properties by moving and rotating them. This program doesn't work well on mobile, better opened from PC.

ilya sheprut 99 Dec 7, 2022
πŸ–₯ A feature rich terminal UI file transfer and explorer with support for SCP/SFTP/FTP/S3

?? A feature rich terminal UI file transfer and explorer with support for SCP/SFTP/FTP/S3

Christian Visintin 574 Jan 5, 2023
Opening explorer for lichess.org

lila-openingexplorer3 Personal opening explorer under development. Usage EXPLORER_LOG=lila_openingexplorer3=debug cargo run --release -- --lila https:

Niklas Fiekas 5 Dec 29, 2021
A gui api explorer written in Rust.

Zzz - GUI Api platform Pronounced "Zees"; as in "catching some Z's". A pun on RESTful APIs. example URL: https://jsonplaceholder.typicode.com/todos/ T

Ryan Blecher 0 Nov 11, 2021
A small block explorer for geth PoAs written in rust

Tesseracts A minimalistic block explorer initially created to learn rust. This block explorer has been created as a rust self-learning project to give

adria0.eth 15 Jun 25, 2022
Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.

Spacedrive A file explorer from the future. spacedrive.com Β» Download for macOS Β· Windows Β· Linux Β· iOS Β· watchOS Β· Android ~ Links will be added once

Spacedrive 16.2k Jan 7, 2023