Databento Binary Encoding (DBZ) - Fast message encoding and storage format for market data

Related tags

Command-line rust encoding serialization compression storage messaging zero-copy market-data low-latency zstd tick-data databento
Overview

dbz

build license

A library (dbz-lib) and CLI tool (dbz-cli) for working with Databento Binary Encoding (DBZ) files. Python bindings for dbz-lib are provided in the dbz-python package.

The Databento Binary Encoding + Zstandard compression (DBZ) is an efficient highly compressible binary encoding suitable for bulk financial time series data, which includes a metadata header.

Features

  • Performant binary encoding and decoding
  • Highly compressible with Zstandard
  • Extendable fixed-width schemas
  • Metadata header in a leading zstd skippable frame

The DBZ format relies on a compliant Zstandard decompressor to read the data. The basic metadata can be read without the need for zstd, as it is not compressed, however the symbology portion is.

Usage

See the respective READMEs for usage details:

Building

dbz is written in Rust, so you'll need to have Rust installed first.

To build, run the following commands:

git clone https://github.com/databento/dbz
cd dbz
cargo build --release
./target/release/dbz --help

For the Python bindings, you'll need Maturin installed as well. To install the bindings for your active Python environment, run the following:

maturin develop --release
python -c 'import dbz_python'""

Testing

Tests are run through cargo test.

License

Distributed under the Apache 2.0 License.

You might also like...
zman is a CLI year (time) progress that small, fast, and just one single binary.
zman is a CLI year (time) progress that small, fast, and just one single binary.

zman zman is a CLI year (time) progress that small, fast, and just one single binary. Features Show year progress Show month, and week progress Show r

azzamsa 17 Dec 21, 2022
A simple command line program to upload file or directory to web3.storage with optional encryption and compression
A simple command line program to upload file or directory to web3.storage with optional encryption and compression

w3s-cli A simple command line program to upload file or directory to web3.storage with optional encryption and compression. Features Uploads single fi

qdwang 5 Oct 22, 2022
Mirroring remote repositories to s3 storage, with atomic updates and periodic garbage collection.

rsync-sjtug WIP: This project is still under development, and is not ready for production use. rsync-sjtug is an open-source project designed to provi

SJTUG 57 Feb 22, 2023
Tool and framework for securely reading untrusted USB mass storage devices.

usbsas is a free and open source (GPLv3) tool and framework for securely reading untrusted USB mass storage devices. Description Following the concept

CEA IT Security 250 Aug 16, 2023
Concurrent and multi-stage data ingestion and data processing with Rust+Tokio

TokioSky Build concurrent and multi-stage data ingestion and data processing pipelines with Rust+Tokio. TokioSky allows developers to consume data eff

DanyalMh 29 Dec 11, 2022
An event replay tool for the Trento storage backend.

photofinish - a little, handy tool to replay events This tiny CLI tool aims to fulfill the need to replay some events and get fixtures. Photofinish re

null 5 Nov 10, 2022
ISG lets you use YouTube as cloud storage for ANY files, not just video
ISG lets you use YouTube as cloud storage for ANY files, not just video

I was working on this instead of my finals, hope you appreciate it. I'll add all relevant executables when I can Infinite-Storage-Glitch AKA ISG (writ

HistidineDwarf 3.6k Feb 23, 2023
A reliable key-value storage for modern software

Quick-KV A reliable key-value storage for modern software Features Binary Based Data-Store Serde Supported Data Types Thread Safe Links Documentation

null 3 Oct 11, 2023
HTTP client/libcurl TUI front end in Rust, with request + key storage
HTTP client/libcurl TUI front end in Rust, with request + key storage

Rust TUI HTTP Client with API Key Management This project is still in active development and although it is useable, there may still be bugs and signi

Preston Thorpe 23 Nov 9, 2023
Releases(v0.2.1)
Owner
Databento, Inc.
Pay as you go for market data
Databento, Inc.
GitHub
An apocalypse-resistant data storage format for the truly paranoid.

Carbonado An apocalypse-resistant data storage format for the truly paranoid. Designed to keep encrypted, durable, compressed, provably replicated con

diba-io 30 Dec 29, 2022
Single File Assets is a file storage format for images

SFA (Rust) Single File Assets is a file storage format for images. The packed images are not guaranteed to be of same format because the format while

null 1 Jan 23, 2022
Given a set of kmers (fasta format) and a set of sequences (fasta format), this tool will extract the sequences containing the kmers.

Kmer2sequences Description Given a set of kmers (fasta / fastq [.gz] format) and a set of sequences (fasta / fastq [.gz] format), this tool will extra

Pierre Peterlongo 22 Sep 16, 2023
Tight Model format is a lossy 3D model format focused on reducing file size as much as posible without decreasing visual quality of the viewed model or read speeds.

What is Tight Model Format The main goal of the tmf project is to provide a way to save 3D game assets compressed in such a way, that there are no not

null 59 Mar 6, 2023
Galileo OSNMA (Open Service Navigation Message Authentication)

galileo-osnma galileo-osnma is a Rust implementation of the Galileo OSNMA (Open Service Navigation Message Authentication) protocol. This protocol is

Daniel Estévez 26 Nov 25, 2022
A Rust program/library to write a Hello World message to the standard output.

hello-world Description hello-world is a Rust program and library that prints the line Hello, world! to the console. Why was this created? This progra

null 0 May 11, 2022
An interprocess message bus system built in Rust.

An interprocess message bus system built in Rust, which can be used to pass messages between multiple processes, even including kernel objects (HANDLE

Bytedance Inc. 5 Jul 16, 2023
UniSBOM is a tool to build a software bill of materials on any platform with a unified data format.

UniSBOM is a tool to build a software bill of materials on any platform with a unified data format. Work in progress Support MacOS Uses system_profile

Simone Margaritelli 32 Nov 2, 2022
PyO3's PyAny as a serde data format

serde-pyobject PyO3's PyAny as a serde data format Usage Serialize T: Serialize into &'py PyAny: use serde::Serialize; use pyo3::{Python, types::{PyAn

Jij 3 Nov 24, 2023
a simple, non-self-describing data-interchange format.

rust-fr 'rust-fr' (aka rust for real) is a simple, non-self-describing data-interchange format. installation You can use either of these methods. Add

Ayush 4 Feb 28, 2024