A rust library for creating and managing logs of arbitrary binary data

Overview

binlog

Test crates.io API docs

A rust library for creating and managing logs of arbitrary binary data. Presently it's used to collect sensor data. But it should generally be helpful in cases where you need to store timeseries data, in a nearly (but not strictly) append-only fashion.

The underlying storage of logs are pluggable via a few traits. Binlog includes built-in implementations via sqlite, redis, and in-memory-only. Additionally, python bindings allow you to use (a subset of) binlog from python.

Usage

From Rust

A small example:

use binlog::{Entry, Error, Range, RangeableStore, SqliteStore, Store};
use std::borrow::Cow;
use string_cache::Atom;

/// Demonstrates the sqlite store, with results in `example.db`. You may want to delete that before
/// running this to see the results of this on an empty database.
fn main() -> Result<(), Error> {
    // Create a new datastore with sqlite backing. The result will be stored in example.db, with
    // default compression options. In-memory is also possible via
    // `binlog::MemoryStore::default()`.
    let store = SqliteStore::new("example.db", None)?;

    // Add 10 entries.
    for i in 1..11u8 {
        let entry = Entry::new_with_timestamp(i as i64, Atom::from("sqlite_example"), vec![i]);
        store.push(Cow::Owned(entry))?;
    }

    // Queries are done via `range`. Here we grab entries with any timestamp and any name.
    let range = store.range(.., None)?;
    // Count the number of entries.
    println!("initial count: {}", range.count()?);
    // We can also iterate on the entries.
    for entry in range.iter()? {
        println!("entry: {:?}", entry?);
    }

    // Remove the entries with 4 <= ts <= 6 and with the name `sqlite_example`.
    store.range(4..=6, Some(Atom::from("sqlite_example")))?.remove()?;

    // Now get the range of entries with 5 <= ts and with the name `sqlite_example`.
    let range = store.range(5.., Some(Atom::from("sqlite_example")))?;
    println!("count after range deletion: {}", range.count()?);
    for entry in range.iter()? {
        println!("entry: {:?}", entry?);
    }

    Ok(())
}

From Python

A small example:

from binlog import binlog
store = binlog.SqliteStore("example.db")
store.push(binlog.Entry(1, "pytest_push", [1, 2, 3]))

Stores

Stores implement the Store trait, and zero or more optional extensions depending on their supported functionality. A few stores implementations are built-in to binlog:

In-memory-only

The in-memory-only store has no means of persistence, but offers the full log functionality. This is also used internally for fuzzing other implementations against.

Redis

The redis implementation is enableable via the redis-store feature. Under the hood, it uses redis streams. It supports subscriptions, but not ranges.

Sqlite

The sqlite implementation is enableable via the sqlite-store feature. It supports ranges, but not subscriptions.

Testing

Unit tests

Tests can be run via make test. This will also be run in CI.

Benchmarks

Benchmarks can be run via make bench.

Fuzzing

A fuzzer is available, ensuring the the sqlite and in-memory datastores operate identically. Run it via make fuzz.

Checks

Lint and formatting checks can be run via make check. Equivalent checks will also run in CI.

You might also like...
Logging implementation for Rust

log A Rust library providing a lightweight logging facade. log documentation A logging facade provides a single logging API that abstracts over the ac

Structured, contextual, extensible, composable logging for Rust
Structured, contextual, extensible, composable logging for Rust

Getting started Introduction FAQ Crate list slog-rs - The Logging for Rust Introduction (please read) slog is an ecosystem of reusable components for

A highly configurable logging framework for Rust

log4rs log4rs is a highly configurable logging framework modeled after Java's Logback and log4j libraries. Warning If you are using the file rotation

Application level tracing for Rust.

Website | Chat | Documentation (master branch) Overview tracing is a framework for instrumenting Rust programs to collect structured, event-based diag

A Rust logger with various features.

Moe Logger (>ω<) Another logger based on pretty-env-logger and env_logger. Allow writing log to file with features like formatting, file rotation. Usa

godot-logger is an easy-to-use logger for godot-rust projects.

godot-logger is an easy-to-use logger for godot-rust projects. It prints logs to Godot's output console and supports module-specific log levels.

Another Key Logger Yet. Rust.

Another Key Logger Yet. Rust. For my very first experience of working with Rust, I decided to manage the keyboard, this time by logging and writing th

Task-based logging for rust
Task-based logging for rust

task_log task_log is a task-based logger. Installing Just add task_log = 0.1.4 to your Cargo.toml's dependency section. Example Let's get right to the

💬 A couple of functions to make logging in Rust easier.
💬 A couple of functions to make logging in Rust easier.

Rust logging ⛔ 💬 A couple of functions to make logging in Rust easier. Installation 📦 Just add the code of code.rs to your project. You can copy/pas

Comments
  • Just use i64 for timestamps

    Just use i64 for timestamps

    Rather than durations. i64s are all that can be realistically handled anyways (because of rusqlite/sqlite constraints), so rather than convert back and forth, we can just deal with raw i64s. It's safer, faster, and gives the user flexibility to determine how they want to model time.

    opened by ysimonson 0
  • Enable WAL

    Enable WAL

    Seems to perform better vs baseline, though push_parallel is still failing:

    test sqlite::benches::iter          ... bench:     176,909 ns/iter (+/- 3,573)
    test sqlite::benches::push          ... bench:   7,678,272 ns/iter (+/- 3,636,034)
    test sqlite::benches::push_parallel ... FAILED
    
    opened by ysimonson 0
  • Benchmarks

    Benchmarks

    Initial results:

    test memory::benches::iter          ... bench:       8,904 ns/iter (+/- 372)
    test memory::benches::push          ... bench:          53 ns/iter (+/- 5)
    test memory::benches::push_parallel ... bench:   6,643,523 ns/iter (+/- 490,757)
    test sqlite::benches::iter          ... bench:     182,264 ns/iter (+/- 17,654)
    test sqlite::benches::push          ... bench:   8,659,845 ns/iter (+/- 3,747,654)
    test sqlite::benches::push_parallel ... FAILED
    

    Sqlite is expected to run much slower since it's syncing to disk. push_parallel is currently failing because of heavy write contention causing DatabaseBusy errors. This should be addressable with wal2 mode + some other tweaks.

    opened by ysimonson 0
  • Windows support

    Windows support

    For now windows support is disabled, but it should be fairly straight-forward to add:

    1. Link sqlite

    2. Add windows releases to the maturin github workflow:

      windows:
        runs-on: windows-latest
        steps:
        - uses: actions/checkout@v2
        - uses: messense/maturin-action@v1
          with:
            command: build
            args: --release --no-sdist -o dist --cargo-extra-args="--all-features"
        - name: Upload wheels
          uses: actions/upload-artifact@v2
          with:
            name: wheels
            path: dist
    
    opened by ysimonson 0
Owner
Yusuf Simonson
Yusuf Simonson
Firecracker takes your HTTP logs and uses them to map your API flows and to detect anomalies in them.

Who is BLST and what do we do? BLST (Business Logic Security Testing) is a startup company that's developing an automatic penetration tester, replacin

BLST 692 Jan 2, 2023
Quickwit - the next-gen search & analytics engine built for logs

Quickwit is the next-gen search & analytics engine built for logs. It is a highly reliable & cost-efficient alternative to Elasticsearch.

Quickwit OSS 2.9k Dec 30, 2022
A cool log library built using rust-lang

RustLog A cool log library built using rust-lang Installation: Cargo.toml rustlog = { git = "https://github.com/krishpranav/rustlog" } log = "0.4.17"

Krisna Pranav 2 Jul 21, 2022
A logging library for eBPF programs.

aya-log - a logging library for eBPF programs Overview aya-log is a logging library for eBPF programs written using aya. Think of it as the log crate

null 18 Oct 13, 2022
A pretty, sensible logger for Rust - ideal for running examples and tests on a crate of choice

sensible-env-logger A pretty, sensible logger for Rust - ideal for running examples and tests on a crate of choice. This is a thin wrapper around pret

Ritvik Nag 3 Aug 9, 2022
Compute and derive metrics by watching a log file

Export Prometheus metrics from your web server's access log Web servers only offer limited metrics. For example, nginx only offers stub_status unless

Remi Rampin 1 Nov 4, 2022
Rall is an incredibly simple and intuitive logger

Really...? Another Logging Library? Yes! :P rall is an incredibly simple and intuitive logger, consider this crate a failure if you can't get setup wi

Hamothy 4 Sep 5, 2022
Log for concurrent workloads, with support for atomic batches and in-order recovery

sharded-log A batch-oriented multi-threaded sharded log for workloads that occasionally flush logs into some other system. All batches have a 32-bit C

Komora 16 Nov 20, 2022
A dead simple logger that works globally and across threads.

Woody A logger for Rust that's *actually* easy to use. Features Easy to use: Just import the macros and you're good to go. No need to configure anythi

Travis A. Wagner 11 Apr 13, 2023
A pretty, easy-to-use logger for Rust.

pretty-env-logger A simple logger built on top of env_logger. It is configured via an environment variable and writes to standard error with nice colo

Sean McArthur 390 Dec 29, 2022