rkyv (archive) is a zero-copy deserialization framework for Rust

Last update: Jun 16, 2022

rkyv

rkyv (archive) is a zero-copy deserialization framework for Rust

Discord docs.rs crates.io MIT license rustc 1.52+

Resources

Learning Materials

  • The rkyv book covers the motivation, architecture, and major features of rkyv
  • The rkyv discord is a great place to get help with specific issues and meet other people using rkyv

Documentation

Benchmarks

  • The rust serialization benchmark is a shootout style benchmark comparing many rust serialization solutions. It includes special benchmarks for zero-copy serialization solutions like rkyv.

Sister Crates

  • bytecheck, which rkyv uses for validation
  • ptr_meta, which rkyv uses for pointer manipulation
  • rend, which rkyv uses for endian-agnostic features

Example

::default(); serializer.serialize_value(&value).unwrap(); let bytes = serializer.into_serializer().into_inner(); let archived = unsafe { archived_root:: (&bytes[..]) }; assert_eq!(archived.int, value.int); assert_eq!(archived.string, value.string); assert_eq!(archived.option, value.option); let deserialized: Test = archived.deserialize(&mut Infallible).unwrap() assert_eq!(deserialized, value); ">
use rkyv::{
    archived_root,
    ser::{serializers::AllocSerializer, Serializer},
    Archive, Deserialize, Infallible, Serialize,
};

#[derive(Archive, Deserialize, Serialize, Debug, PartialEq)]
struct Test {
    int: u8,
    string: String,
    option: Option<Vec<i32>>,
}

let value = Test {
    int: 42,
    string: "hello world".to_string(),
    option: Some(vec![1, 2, 3, 4]),
};

let mut serializer = AllocSerializer::<256>::default();
serializer.serialize_value(&value).unwrap();
let bytes = serializer.into_serializer().into_inner();

let archived = unsafe { archived_root::
    (
    &bytes[..]) };

    assert_eq!(archived.int, value.int);

    assert_eq!(archived.string, value.string);

    assert_eq!(archived.option, value.option);


    let deserialized: Test 
    = archived.
    deserialize(
    &
    mut Infallible).
    unwrap()

    assert_eq!(deserialized, value);
   

Thanks

Thanks to all the sponsors that keep development sustainable. Special thanks to the following sponsors for going above and beyond supporting rkyv:

Platinum Sponsors

Dusk Network

Dusk Network is the first privacy blockchain for financial applications. Our mission is to enable any size enterprise to collaborate at scale, meet compliance requirements, and ensure that transaction data remains confidential.

Bronze Sponsors

Traverse Research

GitHub

https://github.com/rkyv/rkyv
Comments
  • 1. Bus Error while deserializing an f64

    Hi, I am having a Bus Error while deserializing one of my structs and I can't wrap my head around why.

    A bit of context first: I am using rkyv to serialize some data and throw it into and UDP socket. It might be important to note that the sending device has a x86_64 CPU and the other one an ARMv7 CPU (so 32bits) but I'm not using any unsized types or usize/isize so it shouldn't matter. I am using the strict feature of the crate.

    I have these nested structs that I want to send:

    #[derive(Debug, Clone, Archive, Serialize, Deserialize)]
    pub enum NoveltyBeatsModePacket {
        Data(NoveltyBeatsModeData),
        Abort,
        Goodbye(GoodbyeData),
    }
    #[derive(Debug, Clone, Archive, Serialize, Deserialize)]
    pub struct NoveltyBeatsModeData {
        pub novelty: NoveltyModeData,
        pub beat: bool,
    }
    #[derive(Debug, Clone, Archive, Serialize, Deserialize)]
    pub struct NoveltyModeData {
        pub value: f64,
        pub peak: f64,
    }
    

    And I have this code for deserialisation:

    let packet = unsafe {
        archived_value::<NoveltyBeatsModePacket>(&self.deserialize_scratch[..len], 0)
    };
    let packet: NoveltyBeatsModePacket = packet.deserialize(&mut AllocDeserializer).unwrap(); // <-- this produces a Bus Error
    

    After a bit of debugging I managed to get this stacktrace

    #0  0x0058896c in rkyv::core_impl::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211) at /home/lucas/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.4.1/src/core_impl/mod.rs:120
    #1  0x00581f78 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:37
    #2  0x00582204 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:50
    #3  0x00582390 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff209, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:43
    #4  0x0057ea74 in rswave_server::net::NetHandler::recv (self=0x7efff1d8) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_server/src/net.rs:147
    

    So it seams that the Bus Error is thrown by the deserialization of one of the f64 done by the impl_primitve macro. I am quite new to this library and don't know how do diagnose this issue any further so I am asking for your help.

    I should add that for the first packet the deserialization works but not for the next one, and the data received is the same as the data sent.

    Reviewed by icanwalkonwater at 2021-03-19 11:46
  • 2. Validation tests failing on Apple M1

    test validation::test_alloc::tests::basic_functionality ... FAILED
    
    failures:
    
    ---- validation::test_alloc::tests::basic_functionality stdout ----
    thread 'validation::test_alloc::tests::basic_functionality' panicked at 'called `Result::unwrap()`
    on an `Err` value: ContextError(ArchiveError(Unaligned { ptr: 0x16b97e316, align: 4 }))’,
    rkyv_test/src/validation/test_alloc.rs:128:16
    

    I didn’t investigate further but do let me know if I can help.

    Reviewed by b8591340 at 2021-09-01 03:15
  • 3. Add optional support for granular endianness

    I tried to #[derive(Archive)] a u64_be newtype and noticed it was missing, so this PR adds support for optional (feature "endian") granular endianness with non-multibyte impl_primitive and impl_atomic.

    I would have asked first, but it was a trivial change. I went by the existing impls assuming they matched rend’s API, so try and give a look and see if something needs to be added or adjusted.

    rkyv is such a nice project by the way, keep up the great work!

    Reviewed by b8591340 at 2021-09-01 01:06
  • 4. Cannot have `validate` and `no_std` with `-Z build-std`

    I'm using a custom rustc (esp-rs/rust to be specific) and can't afford to use the standard library. Evidently, bytecheck and ptr_meta have std feature enabled by default (here and here) which leads to this error:

       ...
       Compiling ptr_meta v0.1.4
    error[E0658]: use of unstable library feature 'restricted_std'
      |
      = help: add `#![feature(restricted_std)]` to the crate attributes to enable
    
    For more information about this error, try `rustc --explain E0658`.
    error: could not compile `ptr_meta` due to previous error
    

    ... which indicates that a crate is not doing #![no_std]. I can extern crate alloc though.

    Enabled features:

    [dependencies.rkyv]
    version = "0.7.22"
    default_features = false
    features = ["strict", "size_32", "validation", "alloc"]
    

    This library already amazes me. Thanks!

    Reviewed by mamins1376 at 2021-11-23 23:18
  • 5. Unaligned Buffer, only in Debug mode

    We have generated and stored a large data structure using rkyv.

    let mut serializer = AllocSerializer::<4096>::default();
    serializer.serialize_value(&data).unwrap();
    let buf = serializer.into_serializer().into_inner();
    file.write_all(&buf).await?;
    

    Then when deserializing it with

    unsafe { archived_root::<$proto>(&include_bytes!(location)[..]) }
    

    If we do this in debug mode we get the following panic:

    assertion failed: `(left == right)`
      left: `2`,
     right: `0`: unaligned buffer, expected alignment 8 but found alignment 4
    

    This assert is a debug_assert, so if we compile it in release mode, we get no panic BUT as far as we can tell the data is being deserialized correctly and is usable without errors or data consistency issues.

    The assertion seems to be checking for pointer alignment of some kind, as it all seems fine in release the alignment issue being reported by the debug assert seems strange to me.

    How best can we investigate this further?

    Reviewed by LLBlumire at 2021-10-27 08:58
  • 6. Is it possible to implement Serialize/Deserialize for serde_json Value type?

    Hello, Is it possible to implement Serialize/Deserialize for serde_json Value type?

    For example, it doesn't work for bincode because of required hints for deserelization.

    Reviewed by madmaxio at 2021-03-20 11:52
  • 7. Upgrade from 0.6 to 0.7 and archive(copy)

    Hi,

    I am in the process of upgrading from 0.6 to 0.7 and I am having an issue with archive(copy) . It seems that the copy feature is now behind a feature flag and requires nightly? Not sure if that is the case or it is a different feature?

    My same code is below:

    #[derive(Debug, Copy, Clone, PartialEq, rkyv::Archive, rkyv::Serialize)]
    #[archive(copy)]
    pub struct Value {
        id: u32,
        value: f64,
    }
    
    #[derive(Debug,  rkyv::Archive,  rkyv::Serialize)]
    struct ValueIndex {
        items: Vec<Value>
    }
    
    
    Reviewed by d00z3l at 2022-05-06 01:53
  • 8. Flattening RelPtr fields in a Rkyv Archive data struct

    If I have a struct using #[derive(rkyv::Archive)], a new struct with "Archive" in the name is created. This struct has the same fields as the normal struct, except that it wraps them in RelPtr and related types.

    This is fine if your code is the one and only user of these structs. However, what if you need to pass the structs to third-party code, like as a function argument?

    I can call .deserialize() to get back the normal struct without the RelPtrs, but that defeats the purpose of Rkyv.

    Since Rkyv supports trait objects, I could create a trait object to wrap my struct, and serialize the trait object. However, then I am permanently committing to dynamic dispatch.

    What I was thinking was whether there's a way to flatten the RelPtrs without cloning any data. We could walk the whole data structure, flattening the RelPtrs to normal pointers, and return a data struct with all borrowed data and a corresponding lifetime parameter.

    For example, ArchivedString would become &str, ArchivedVec<T> would become &Vec<T>, and so on. Bonus points if this could be Cow<str> and Cow<Vec<T>>.

    This wouldn't work for all possible data structs. In particular, when we hit a complex type like ArchivedVec<T>, it is necessary that T does not contain any more fields that need to be flattened, because we want to point to the Vec<T> that lives in Rkyv's memory block. I think in general, we would only support nested tuples and structs; vectors, hash maps, and so on would need to be leaves. So it would work for fixed-size types like i32, but may not work with String. (It would be really nice to figure out a way to have at least String or &str work in vectors.)

    I could write such a conversion function manually for all my Rkyv types, but that is cumbersome, and it feels like it belongs in the core library, at least as a feature.

    Thoughts?

    CC @Manishearth @zbraniecki

    Reviewed by sffc at 2021-04-06 05:40
  • 9. Provide `#[archive(as = "...")]` for users to provide their own archive types

    With the changes around ArchiveCopy (#119) the idea of a type being archiveable as itself needs to be changed too.

    Adding #[archive(self)] to a type should:

    • Add T: Archive<Archived = Self> bounds for each of the fields
    • Error on any uses of #[archive_attr] or other attributes except #[archive(copy_safe)]
    • Ensure that enums have an appropriate repr
    • Not generate an archived version of the type
    Reviewed by djkoloski at 2021-05-06 16:15
  • 10. Provide implementations for Uuid

    There are a couple popular crates that could use some outreach to add rkyv support. Uuid is a good one to start with and it will give a feel for what the process will be like.

    Reviewed by djkoloski at 2021-03-22 03:10
  • 11. Improve performance of check_archived_root

    I made two little example files:

    #![no_main]
    
    use iai::black_box;
    use rkyv::{check_archived_root, Aligned};
    
    const RKYV_BUF: Aligned<[u8; 88]> = Aligned([
        0, 1, 2, 0, 4, 5, 6, 0, 8, 9, 10, 0, 12, 13, 14, 0, 16, 17, 18, 0, 20, 21, 22, 0, 24, 25, 26,
        0, 28, 29, 30, 0, 32, 33, 34, 0, 36, 37, 38, 0, 40, 41, 42, 0, 44, 45, 46, 0, 48, 49, 50, 0,
        52, 53, 54, 0, 56, 57, 58, 0, 60, 61, 62, 0, 64, 65, 66, 0, 68, 69, 70, 0, 72, 73, 74, 0, 76,
        77, 78, 0, 176, 255, 255, 255, 20, 0, 0, 0,
    ]);
    
    #[no_mangle]
    fn main(_argc: isize, _argv: *const *const u8) -> isize {
        let archived = check_archived_root::<Vec<u32>>(black_box(&RKYV_BUF.0)).unwrap();
        assert_eq!(52629240u32, archived.iter().copied().sum());
        0
    }
    

    The second is the same, except that I use archived_root instead of check_archived_root.

    The unchecked version compiles to a nice and compact 475 bytes (measured using the wasm32-unknown-unknown target) and requires only 96 instructions, according to iai. However, the checked version compiles to 10078 bytes and requires 4858 instructions.

    Much of the code size is dlmalloc. If the checking code requires dlmalloc, then it's not #[no_std] compatible. I don't understand why validation code needs to allocate memory; it should just do some basic checks on the data and return "valid" or "not valid".

    Suggestions for improvement:

    1. Make check_archived_root be #[no_std]
    2. Follows from (1): verify that code size and performance hit are minimal when using check_archived_root instead of unsafe { archived_root }.
    Reviewed by sffc at 2021-04-11 10:08
  • 12. feature request: make derived archived type `pub` with original type not `pub`.

    This is because I'm creating an intermediate type. So my user creates type Original, from which I create an Intermediate, which derives Archive and produces an archived type that I want to expose publicly. Is this possible?

    Reviewed by droundy at 2022-06-01 00:00
  • 13. Deriving `Archive` leaks private types.

    I have a type that looks like:

    #[derive(Archive)]
    pub struct IdentifierId {
        inner: IdentifierIdInner
    }
    

    which is giving me an error because IdentifierIdInner is a private type (which is the whole point). Is there a solution to this? Presumably I could just manually implement Archive, but it would be much nicer to derive it.

    Here is the error message I get:

    error[E0446]: private type `IdentifierIdInner` in public interface
       --> workspace/nemo_core/src/hashtypes.rs:95:72
        |
    95  | #[derive(Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize, Archive)]
        |                                                                        ^^^^^^^ can't leak private type
    ...
    118 | enum IdentifierIdInner {
        | ---------------------- `IdentifierIdInner` declared as private
        |
        = note: this error originates in the derive macro `Archive` (in Nightly builds, run with -Z macro-backtrace for more info)
    
    
    Reviewed by droundy at 2022-05-31 18:10
  • 14. Serialization issues between x86_64 and i386

    My colleague and I have discovered an issue with serialization and deserialization between x86_64 and i386 platforms. As far as I can tell this issue only appears on x86 processors, as I tested on AArch64 and ThumbV8M without issue.

    The issue appears with the following structs:

    #[derive(Archive, Serialize, Deserialize, Clone, Debug)]
    #[archive_attr(derive(bytecheck::CheckBytes))]
    struct Test {
        a: u64,
        e: TestE,
    }
    
    #[derive(Archive, Serialize, Deserialize, Clone, Debug)]
    #[archive_attr(derive(bytecheck::CheckBytes))]
    enum TestE {
        Foo { b: [u8; 4], a: [u8; 4] },
        Bar { a: [u8; 4], b: [u8; 4] },
    }
    

    When serialized on i686 the hex output is ffffffffffffffff000f0e0d0c01020304000000.

    When serialized on x86_64 the hex output is ffffffffffffffff000f0e0d0c0102030400000000000000.

    When deserializing the i686 output on x86_64 you get the following error:

    thread 'main' panicked at 'calledResult::unwrap()on anErrvalue: CheckBytesError(ContextError(ArchiveError(OutOfBounds { base: 0x55a349ea9b70, offset: -4, range: 0x55a349ea9b70..0x55a349ea9b84 })))', src/main.rs:13:42

    I have posted test code here: https://github.com/sphw/rkyv-issue/blob/master/src/main.rs

    You can test the issue on an x86_64 Linux machine with the following:

    ➜  rkyv-issue git:(master) cargo run --target i686-unknown-linux-gnu
        Finished dev [unoptimized + debuginfo] target(s) in 0.01s
         Running `target/i686-unknown-linux-gnu/debug/rkyv-test`
    HEX BYTES = ffffffffffffffff000f0e0d0c01020304000000
    Test { a: 18446744073709551615, e: Foo { b: [15, 14, 13, 12], a: [1, 2, 3, 4] } }
    ➜  rkyv-issue git:(master) cargo run
        Finished dev [unoptimized + debuginfo] target(s) in 0.01s
         Running `target/debug/rkyv-test`
    HEX BYTES = ffffffffffffffff000f0e0d0c0102030400000000000000
    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: CheckBytesError(ContextError(ArchiveError(OutOfBounds { base: 0x55a349ea9b70, offset: -4, range: 0x55a349ea9b70..0x55a349ea9b84 })))', src/main.rs:13:42
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    All credit to @sadroeck for discovering this.

    Reviewed by sphw at 2022-05-21 05:33
  • 15. `MapKV` support for `HashMap` and `BTreeMap`

    The Map wrapper can be used to map elements of Options and Vecs, but not HashMaps or BTreeMaps. A wrapper that allowed this would enable projections into these containers as well.

    Reviewed by djkoloski at 2022-05-19 23:21
  • 16. ArrayString support

    It would be useful to have fixed-length string support such as arraystring::ArrayString or arrayvec::ArrayString. This would make it safe to mutate values that contain strings in-place.

    Reviewed by mockfox at 2022-04-29 10:22
A binary encoder / decoder implementation in Rust.
A binary encoder / decoder implementation in Rust.

Bincode A compact encoder / decoder pair that uses a binary zero-fluff encoding scheme. The size of the encoded object will be the same or smaller tha

Jun 23, 2022
An impish, cross-platform binary parsing crate, written in Rust
An impish, cross-platform binary parsing crate, written in Rust

libgoblin Documentation https://docs.rs/goblin/ changelog Usage Goblin requires rustc 1.40.0. Add to your Cargo.toml [dependencies] goblin = "0.3" Fea

Jun 24, 2022
Safe and ergonomic Rust-Mach bindings.

mach-rs This project aims to provide safe and ergonomic bindings to Mach APIs for the Rust programming language. License Copyright (c) 2021 Umang Ragh

May 9, 2022
A HTTP Archive format (HAR) serialization & deserialization library, written in Rust.

har-rs HTTP Archive format (HAR) serialization & deserialization library, written in Rust. Install Add the following to your Cargo.toml file: [depende

Jan 22, 2022
archive-rs provides a generic way of dealing with multiple archive and compression formats in Rust

archive-rs A Rust crate that aims to provide a generic way of dealing with multiple archive and compression formats by providing a generic abstraction

Nov 21, 2021
Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

Jun 24, 2022
Simple async codec for rkyv. Reuses streaming buffer for maximum speed

rkyv_codec Simple async codec for rkyv. Reuses streaming buffer for maximum speed! This crate provides a makeshift adaptor for streaming &Archived<Obj

Jun 14, 2022
Rust wrapper for Eclipse iceoryx™ - true zero-copy inter-process-communication
Rust wrapper for Eclipse iceoryx™ - true zero-copy inter-process-communication

iceoryx-rs Experimental rust wrapper for the iceoryx IPC middleware. clone and build The iceoryx repo is include as git submodule, therefore keep in m

Apr 26, 2022
Membrane is an opinionated crate that generates a Dart package from a Rust library. Extremely fast performance with strict typing and zero copy returns over the FFI boundary via bincode.

Membrane is an opinionated crate that generates a Dart package from a Rust library. Extremely fast performance with strict typing and zero copy returns over the FFI boundary via bincode.

Jun 7, 2022
Zero-Copy reading and writing of geospatial data.

GeoZero Zero-Copy reading and writing of geospatial data. GeoZero defines an API for reading geospatial data formats without an intermediate represent

Jun 19, 2022
A zero-copy parser for the contents of the __unwind_info section of a mach-O binary.

A parser for Apple's Compact Unwinding Format, which is used in the __unwind_info section of mach-O binaries.

May 31, 2022
RISC Zero is a zero-knowledge verifiable general computing platform based on zk-STARKs and the RISC-V microarchitecture.

RISC Zero WARNING: This software is still experimental, we do not recommend it for production use (see Security section). RISC Zero is a zero-knowledg

Jun 24, 2022
Msgpack serialization/deserialization library for Python, written in Rust using PyO3, and rust-msgpack. Reboot of orjson. msgpack.org[Python]

ormsgpack ormsgpack is a fast msgpack library for Python. It is a fork/reboot of orjson It serializes faster than msgpack-python and deserializes a bi

Jun 16, 2022
WebAssembly serialization/deserialization in rust

parity-wasm Low-level WebAssembly format library. Documentation Rust WebAssembly format serializing/deserializing Add to Cargo.toml [dependencies] par

Jun 19, 2022
serde-like serialization and deserialization of static Rust types in XML

static-xml static-xml is a serde-like serialization and deserialization library for XML, currently written as a layer on top of xml-rs. Status: in ear

Jun 5, 2022
Deser: an experimental serialization and deserialization library for Rust

deser: an experimental serialization and deserialization library for Rust Deser is an experimental serialization system for Rust. It wants to explore

Jun 15, 2022
Custom deserialization for fields that can be specified as multiple types.

serde-this-or-that Custom deserialization for fields that can be specified as multiple types. This crate works with Cargo with a Cargo.toml like: [dep

Apr 26, 2022
Rust crate which provides direct access to files within a Debian archive

debarchive This Rust crate provides direct access to files within a Debian archive. This crate is used by our debrep utility to generate the Packages

Dec 18, 2021
A Web-App written in Rust with Yew, using the same SyntaxHighlighter from Google Code Archive as planetb.ca

PlanetB SyntaxHighlighter About This is a small app, providing static files to have a frontend to format your code so you can paste it with styles to

Mar 20, 2022
A static mail HTML archive for the 21st century, written in Rust

?? Crabmail ?? self-hosted / github mirror A static mail HTML archive for the 21st century, written in Rust. Includes helpful "modern" features that e

Jun 22, 2022