Fast fail2ban-like tools for parsing nginx logs

Overview

Fast2ban

This is simple fail2ban-like replacement written in Rust.

Usage:

./fast2ban # reads default config.toml from current directory
./fast2ban <config.toml location>

Emits list of suspicious IPs to stdout, one per line, some information to stderr:

Using config file config.toml
Config: Config {
    log_file: "nginx.log",
    log_regex: "^(?P<ip>\\d+\\.\\d+\\.\\d+\\.\\d+) - [^ ]+ \\[(?P<DT>[^\\]]+)\\]",
    requests: 30,
    period: 30,
    date_format: "%d/%B/%Y:%H:%M:%S %z",
}
elapsed 398 ms, 100000 lines parsed, 0 datetime errors, 251256.28140703516 lines/s, banned = 565/3498

Configuration

Example config.toml:

# log file to parse. Pass '-' to use stdin
log_file = 'nginx.log'

# regexp for parsing log file. Must contain groups <ip> and <DT>, other groups are ignored for now
log_regex = '^(?P<ip>\d+\.\d+\.\d+\.\d+) - [^ ]+ \[(?P<DT>[^\]]+)\] "(\w+) (?P<addr>[^ ]*) HTTP/[\d.]+" (?P<code>\d+) \d+ "[^"]+" "[^"]+" "(?P<UA>[^"]+)'

# maximum number of requests for specific IP for specific time period
requests = 30

# time period in seconds
period = 30

# date format (see https://docs.rs/chrono/0.4.19/chrono/format/strftime/index.html for syntax).
# Note that timezone is required (because of using https://docs.rs/chrono/0.4.19/chrono/struct.DateTime.html#method.parse_from_str method)
date_format = '%d/%B/%Y:%H:%M:%S %z'

Using IPset to efficiently ban IPs

IPset is a fast and efficient way to ban IPs (compared to banning them one by one via separate iptables rules).

  1. Create IPset file:
ipset create banner hash:ip
  1. Create iptables rule for banning IPs:
iptables -I INPUT -p tcp -m multiport --dports 80,443 -m set --match-set banner src -j DROP
  1. Run something like this periodically:
# get last queries
tail -n 500000 /var/log/nginx/access.log | grep '/ HTTP/' > nginx.log
# create suspicious IPs list
./fast2ban > ips.txt
# create restore file for IPset
cat ips.txt xargs -n1 echo add banner > ipset-restore.txt
# add IPs to IPset 
ipset restore -exist < ipset-restore.txt

# or in single line:
tail -n 500000 /var/log/nginx/access.log | grep '/ HTTP/' > nginx.log && ./fast2ban | xargs -n1 echo add banner | ipset restore -exist

# or if using log_file = '-' to read from stdin:
tail -n 500000 /var/log/nginx/access.log | grep '/ HTTP/' | ./fast2ban | xargs -n1 echo add banner | ipset restore -exist
You might also like...
Tools to feature more lenient Polonius-based borrow-checker patterns in stable Rust
Tools to feature more lenient Polonius-based borrow-checker patterns in stable Rust

Though this be madness, yet there is method in 't. More context Hamlet: For yourself, sir, shall grow old as I am – if, like a crab, you could go back

A Matrix bot which can generate
A Matrix bot which can generate "This Week in X" like blog posts

hebbot A Matrix bot which can help to generate periodic / recurrent summary blog posts (also known as "This Week in X"). The bot was inspired by twim-

A conky-like system monitor made for the sole purpose of teaching myself rust-lang.

Pomky A conky-like system monitor made for the sole purpose of teaching myself rust-lang. It is not as configurable, modular, or feature packed as con

Like wc, but unicode-aware, and with per-line mode
Like wc, but unicode-aware, and with per-line mode

Like wc, but unicode-aware, and with per-line mode

A cargo plugin for showing a tree-like overview of a crate's modules.

cargo-modules Synopsis A cargo plugin for showing an overview of a crate's modules. Motivation With time, as your Rust projects grow bigger and bigger

Rust bindings to the dos-like framework
Rust bindings to the dos-like framework

dos-like for Rust   This project provides access to Mattias Gustavsson's dos-like framework, so as to write DOS-like applications in Rust. How to use

Embeddable tree-walk interpreter for a "mostly lazy" Lisp-like scripting language.

ceceio Embeddable tree-walk interpreter for a "mostly lazy" Lisp-like scripting language. Just a work-in-progress testbed for now. Sample usage us

A Rust-like Hardware Description Language transpiled to Verilog

Introduction This projects attempts to create a Rust-like hardware description language. Note that this has nothing to do with Rust itself, it just ha

An OS like a lump of mud.

slimeOS An OS like a lump of mud. Run Clone this repo, and just do make run, and then you will see: [rustsbi] RustSBI version 0.3.0-alpha.4, adapting

Owner
null
Fast and simple datetime, date, time and duration parsing for rust.

speedate Fast and simple datetime, date, time and duration parsing for rust. speedate is a lax† RFC 3339 date and time parser, in other words, it pars

Samuel Colvin 43 Nov 25, 2022
PE Parsing, but blazing fast

PE Parser A blazing fast ?? PE Parser written in Rust Motivation The main goals of pe-parser is to write something suitable for a PE Loader. Is editin

Isaac Marovitz 8 Apr 21, 2023
A fast little combinational parsing library.

neure A fast little combinational parsing library Performance rel is mean release, fat is mean release with lto=fat See examples Example use neure::*;

loren 9 Aug 16, 2023
S-expression parsing and writing in Rust

rsexp S-expression parsing and writing in Rust using nom parser combinators. This implemantion aims at being compatible with OCaml's sexplib. The main

Laurent Mazare 12 Oct 18, 2022
Tutorial for parsing with nom 5.

Nom Tutorial Nom is a wonderful parser combinators library written in Rust. It can handle binary and text files. Consider it where you would otherwise

Benjamin Kay 265 Dec 11, 2022
Lightweight parsing for Rust proc macros

Lightweight parsing for Rust proc macros Venial is a WIP parser for Rust proc macros. When writing proc macros that need to parse Rust code (such as a

Olivier FAURE 148 Dec 30, 2022
Static-checked parsing of regexes into structs

Statically-checked regex parsing into structs. This avoids common regex pitfalls like Off by one capture indexes Trying to get nonexistent captures De

Andrew Baxter 4 Dec 18, 2022
Tools for managing GitHub block lists

GitHub block list management Octocrabby is a small set of command-line tools and Octocrab extensions that are focused on managing block lists on GitHu

Travis Brown 97 Nov 3, 2022
Rust 核心库和标准库的源码级中文翻译,可作为 IDE 工具的智能提示 (Rust core library and standard library translation. can be used as IntelliSense for IDE tools)

Rust 标准库中文版 这是翻译 Rust 库 的地方, 相关源代码来自于 https://github.com/rust-lang/rust。 如果您不会说英语,那么拥有使用中文的文档至关重要,即使您会说英语,使用母语也仍然能让您感到愉快。Rust 标准库是高质量的,不管是新手还是老手,都可以从中

wtklbm 493 Jan 4, 2023
Conversion Tools API Rust client

ConversionTools Rust This Conversion Tools API Rust client allows you to use the site API and convert files faster and more conveniently. Site Convers

WinsomeQuill 2 Jan 23, 2022