Efficiently store Rust idiomatic bytes related types in Avro encoding.

Overview

Serde Avro Bytes

Avro is a binary encoding format which provides a "bytes" type optimized to store &[u8] data like.

Unfortunately the apache_avro encodes Vec<u8> as an array of integers thus the encoded data are twice bigger than using the bytes.

#[derive(Serialize)]
struct Record {
    data: Vec<u8>
}

fn playground() {
    let record = Record {
        data: vec![1,2]
    };
    // data field
    // => encoded as in int array : [4,1,1,1,2,0]
    // => encoded as bytes : [4,1,2]
}   

This crate provided a set of module to handle idiomatic Rust types and encode its component as "bytes".

#[derive(Serialize, Deserialize)]
struct Record {
    #[serde(with = "serde_avro_bytes::bytes")]
    key: Vec<u8>,
    #[serde(with = "serde_avro_bytes::bytes::option")]
    key2: Option<Vec<u8>>,
    #[serde(with = "serde_avro_bytes::hashmap")]
    key3: HashMap<Vec<u8>, Vec<u8>>,
    #[serde(with = "serde_avro_bytes::btreemap::option")]
    key4: Option<BTreeMap<Vec<u8>, Vec<u8>>>,
    #[serde(with = "serde_avro_bytes::list")]
    key5: Vec<Vec<u8>>,
    #[serde(with = "serde_avro_bytes::list::option")]
    key6: Option<Vec<Vec<u8>>>,
}

Features

  • bstr: adds support for working with BStrings which are convenient wrappers for partially valid UTF-8 bytes sequences provided by the bst crate. See examples/bstr.rs.
You might also like...
Zero-copy, no-std proquint encoding and decoding

proqnt A pronounceable quintuplet, or proquint, is a pronounceable 5-letter string encoding a unique 16-bit integer. Proquints may be used to encode b

Super-simple, fully Rust powered
Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.

memex Super simple "memory" for LLM projects, semantic search, etc. Running the service Note that if you're running on Apple silicon (M1/M2/etc.), it'

Key-value store for embedded systems, for raw NOR flash, using an LSM-Tree.

ekv Key-value store for embedded systems, for raw NOR flash, using an LSM-Tree. Features None yet TODO Everything Minimum supported Rust version (MSRV

Simple, safe way to store and distribute tensors

safetensors Safetensors This repository implements a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-

Seamless Higher-Kinded Types in Rust

Seamless Higher-Kinded Types in Rust This is actual working code: pub trait FunctorA : HKT1 { fn mapB, F: FnMut(A) - B(self, f: F) - Self::W

A typemap for a set of known types optionally without heap allocation, and supporting iterating by traits

fixed_typemap docs.rs GitHub Sponsors Implements typemaps that support a lot of extra funcctionality using procedural macros. docs.rs has a lot more t

An implementation of a predicative polymorphic language with bidirectional type inference and algebraic data types

Vinilla Lang Vanilla is a pure functional programming language based on System F, a classic but powerful type system. Merits Simple as it is, Vanilla

Custom deserialization for fields that can be specified as multiple types.

serde-this-or-that Custom deserialization for fields that can be specified as multiple types. This crate works with Cargo with a Cargo.toml like: [dep

A library for transcoding between bytes in Astro Notation Format and Native Rust data types.

Rust Astro Notation A library for transcoding between hexadecimal strings in Astro Notation Format and Native Rust data types. Usage In your Cargo.tom

Stelar Software 1 Feb 4, 2022
A tool to deserialize data from an input encoding, transform it and serialize it back into an output encoding.

dts A simple tool to deserialize data from an input encoding, transform it and serialize it back into an output encoding. Requires rust >= 1.56.0. Ins

null 11 Dec 14, 2022
A peer-reviewed collection of articles/talks/repos which teach concise, idiomatic Rust.

This repository collects resources for writing clean, idiomatic Rust code. Please bring your own. ?? Idiomatic coding means following the conventions

Matthias 4.2k Dec 30, 2022
Rust-idiomatic, compliant, flexible and performant BIP21 crate

Rust implementation of BIP21 Rust-idiomatic, compliant, flexible and performant BIP21 crate. About Important: while lot of work went into polishing th

Martin Habovštiak 6 Dec 15, 2022
alto provides idiomatic Rust bindings for OpenAL 1.1 and extensions (including EFX).

alto alto provides idiomatic Rust bindings for OpenAL 1.1 and extensions (including EFX). WARNING Because Alto interacts with global C state via dynam

null 80 Aug 7, 2022
Safe, idiomatic bindings to cFE and OSAL APIs for Rust

n2o4 The n2o4 crate provides safe, idiomatic Rust bindings to the APIs of cFE and OSAL, the libraries of the Core Flight System (cFS). IMPORTANT NOTE

null 3 Aug 29, 2022
Encode/Decode bytes as emoji base2048

mojibake Encode and decode arbitrary bytes as a sequence of emoji optimized to produce the smallest number of graphemes. Description This is not a spa

null 15 Jul 23, 2023
A collection of comparison-benchmarks for Nova & related Proving systems

Nova benchmarks Here's a list of some of the benchmarks we've been taking to better understand how Nova performs vs other proof systems. Live version:

Privacy & Scaling Explorations (formerly known as appliedzkp) 18 Apr 17, 2023
Toolkit for simple calculations related to Data Comunication and Networks (only available in Spanish temporary)

CDR Toolkit Un toolkit creado para la asignatura Comunicación de Datos y Redes, cursada en la UIB. Es una potente y rápida CLI que ayuda a realizar lo

Jesús Castillo 4 Nov 17, 2023
Parse and encoding of data using the SCTE-35 standard.

SCTE-35 lib and parser for Rust Work in progress! This library provide access to parse and encoding of data using the SCTE-35 standard. This standard

Rafael Carício 4 May 6, 2022