rkyv (archive) is a zero-copy deserialization framework for Rust

rkyv

Last update: Jan 5, 2023

Overview

rkyv (archive) is a zero-copy deserialization framework for Rust

Resources

Learning Materials

The rkyv book covers the motivation, architecture, and major features of rkyv
The rkyv discord is a great place to get help with specific issues and meet other people using rkyv

Documentation

rkyv, the core library
rkyv_dyn, which adds trait object support to rkyv
rkyv_typename, a type naming library

Benchmarks

The rust serialization benchmark is a shootout style benchmark comparing many rust serialization solutions. It includes special benchmarks for zero-copy serialization solutions like rkyv.

Sister Crates

bytecheck, which rkyv uses for validation
ptr_meta, which rkyv uses for pointer manipulation
rend, which rkyv uses for endian-agnostic features

Example

::default(); serializer.serialize_value(&value).unwrap(); let bytes = serializer.into_serializer().into_inner(); let archived = unsafe { archived_root::
(&bytes[..]) }; assert_eq!(archived.int, value.int); assert_eq!(archived.string, value.string); assert_eq!(archived.option, value.option); let deserialized: Test = archived.deserialize(&mut Infallible).unwrap() assert_eq!(deserialized, value); ">
use rkyv::{
    archived_root,
    ser::{serializers::AllocSerializer, Serializer},
    Archive, Deserialize, Infallible, Serialize,
};

#[derive(Archive, Deserialize, Serialize, Debug, PartialEq)]
struct Test {
    int: u8,
    string: String,
    option: Option<Vec<i32>>,
}

let value = Test {
    int: 42,
    string: "hello world".to_string(),
    option: Some(vec![1, 2, 3, 4]),
};

let mut serializer = AllocSerializer::<256>::default();
serializer.serialize_value(&value).unwrap();
let bytes = serializer.into_serializer().into_inner();

let archived = unsafe { archived_root::
    (
    &bytes[..]) };

    assert_eq!(archived.int, value.int);

    assert_eq!(archived.string, value.string);

    assert_eq!(archived.option, value.option);


    let deserialized: Test 
    = archived.
    deserialize(
    &
    mut Infallible).
    unwrap()

    assert_eq!(deserialized, value);
   

Thanks

Thanks to all the sponsors that keep development sustainable. Special thanks to the following sponsors for going above and beyond supporting rkyv:

Platinum Sponsors

Dusk Network is the first privacy blockchain for financial applications. Our mission is to enable any size enterprise to collaborate at scale, meet compliance requirements, and ensure that transaction data remains confidential.

Bronze Sponsors

Comments

UnsafeCell in Archived type incompatible with miri

I'm attempting to create a mutex type that can be used in its Archived form, to facilitate safe mutation of archived data mmap'd into multiple processes.

I have it implemented, and some basic tests working, but noticed that rkyv::archived_root results in an error in miri. rkyv::archived_root_mut works ok. Maybe there's a fundamental reason for that, and the answer is "just use rkyv::archived_root_mut", but I think it might be worth digging into a bit more.

I created a smaller test case that seems to show the problem is more fundamentally around UnsafeCell, which would be needed for interior mutability in any Archive type.

#![deny(unsafe_op_in_unsafe_fn)]

use std::cell::UnsafeCell;

#[repr(transparent)]
struct UnsafeU32 { 
    val: UnsafeCell<u32>,
}

impl<S> rkyv::Serialize<S> for UnsafeU32 
where 
    S: rkyv::Fallible + ?Sized,
{
    fn serialize(&self, serializer: &mut S) -> Result<Self::Resolver, S::Error> {
        let val = unsafe { &*self.val.get() };
        val.serialize(serializer)
    }
}

impl rkyv::Archive for UnsafeU32 {
    type Archived = Self;
    type Resolver = <u32 as rkyv::Archive>::Resolver;

    unsafe fn resolve(&self, pos: usize, resolver: Self::Resolver, out: *mut Self::Archived) {
        unsafe { (&*self.val.get()).resolve(pos, resolver, out as *mut u32) };
    }
}

#[test]
fn test_rkyv_miri() {
    type T = UnsafeU32;
    let original: T = UnsafeU32 {
        val: UnsafeCell::new(10),
    };
    let bytes = rkyv::to_bytes::<_, 256>(&original).unwrap();

    // The archived mutex can be used to mutate the data in place.
    let archived = unsafe { rkyv::archived_root::<T>(&bytes[..]) };
    assert_eq!(unsafe { *archived.val.get() }, 10);
}

The test passes natively, but under miri fails:

$ MIRIFLAGS=-Zmiri-backtrace=full cargo +nightly miri test
Preparing a sysroot for Miri (target: x86_64-unknown-linux-gnu)... done
    Finished test [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/miri/x86_64-unknown-linux-gnu/debug/deps/repro-982c58b2cf8c397a)

running 1 test
test test_rkyv_miri ... error: Undefined Behavior: trying to reborrow from <185489> for SharedReadWrite permission at alloc77718[0x0], but that tag only grants SharedReadOnly permission for this location
  --> /home/jnewsome/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/util/mod.rs:67:5
   |
67 |     &*bytes.as_ptr().add(pos).cast()
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |     |
   |     trying to reborrow from <185489> for SharedReadWrite permission at alloc77718[0x0], but that tag only grants SharedReadOnly permission for this location
   |     this error occurs as part of a reborrow at alloc77718[0x0..0x4]
   |
   = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
   = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <185489> was created by a retag at offsets [0x0..0x4]
  --> src/lib.rs:38:29
   |
38 |     let archived = unsafe { rkyv::archived_root::<T>(&bytes[..]) };
   |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: backtrace:
   = note: inside `rkyv::archived_value::<UnsafeU32>` at /home/jnewsome/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/util/mod.rs:67:5
   = note: inside `rkyv::archived_root::<UnsafeU32>` at /home/jnewsome/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/util/mod.rs:148:5
note: inside `test_rkyv_miri` at src/lib.rs:38:29
  --> src/lib.rs:38:29
   |
38 |     let archived = unsafe { rkyv::archived_root::<T>(&bytes[..]) };
   |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure at src/lib.rs:30:1
  --> src/lib.rs:30:1
   |
29 |   #[test]
   |   ------- in this procedural macro expansion
30 | / fn test_rkyv_miri() {
31 | |     type T = UnsafeU32;
32 | |     let original: T = UnsafeU32 {
33 | |         val: UnsafeCell::new(10),
...  |
39 | |     assert_eq!(unsafe { *archived.val.get() }, 10);
40 | | }
   | |_^
   = note: inside `<[closure@src/lib.rs:30:1: 40:2] as std::ops::FnOnce<()>>::call_once - shim` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:248:5
   = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:248:5
   = note: inside `test::__rust_begin_short_backtrace::<fn()>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:572:5
   = note: inside closure at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:563:30
   = note: inside `<[closure@test::run_test::{closure#1}] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:248:5
   = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1935:9
   = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
   = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492:40
   = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456:19
   = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
   = note: inside `test::run_test_in_process` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:595:18
   = note: inside closure at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:489:39
   = note: inside `test::run_test::run_test_inner` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:527:13
   = note: inside `test::run_test` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:559:28
   = note: inside `test::run_tests::<[closure@test::run_tests_console::{closure#2}]>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:302:17
   = note: inside `test::run_tests_console` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:293:5
   = note: inside `test::test_main` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:113:15
   = note: inside `test::test_main_static` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:132:5
   = note: inside `main`
   = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:248:5
   = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122:18
   = note: inside closure at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145:18
   = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:280:13
   = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492:40
   = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456:19
   = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
   = note: inside closure at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:48
   = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492:40
   = note: inside `std::panicking::r#try::<isize, [closure@std::rt::lang_start_internal::{closure#2}]>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456:19
   = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
   = note: inside `std::rt::lang_start_internal` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:20
   = note: inside `std::rt::lang_start::<()>` at /home/jnewsome/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:144:17
   = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

opened by sporksmith 15

Bus Error while deserializing an f64

Hi, I am having a Bus Error while deserializing one of my structs and I can't wrap my head around why.

A bit of context first: I am using rkyv to serialize some data and throw it into and UDP socket. It might be important to note that the sending device has a x86_64 CPU and the other one an ARMv7 CPU (so 32bits) but I'm not using any unsized types or usize/isize so it shouldn't matter. I am using the strict feature of the crate.

I have these nested structs that I want to send:

#[derive(Debug, Clone, Archive, Serialize, Deserialize)]
pub enum NoveltyBeatsModePacket {
    Data(NoveltyBeatsModeData),
    Abort,
    Goodbye(GoodbyeData),
}
#[derive(Debug, Clone, Archive, Serialize, Deserialize)]
pub struct NoveltyBeatsModeData {
    pub novelty: NoveltyModeData,
    pub beat: bool,
}
#[derive(Debug, Clone, Archive, Serialize, Deserialize)]
pub struct NoveltyModeData {
    pub value: f64,
    pub peak: f64,
}

And I have this code for deserialisation:

let packet = unsafe {
    archived_value::<NoveltyBeatsModePacket>(&self.deserialize_scratch[..len], 0)
};
let packet: NoveltyBeatsModePacket = packet.deserialize(&mut AllocDeserializer).unwrap(); // <-- this produces a Bus Error

After a bit of debugging I managed to get this stacktrace

#0  0x0058896c in rkyv::core_impl::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211) at /home/lucas/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.4.1/src/core_impl/mod.rs:120
#1  0x00581f78 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:37
#2  0x00582204 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff211, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:50
#3  0x00582390 in rswave_common::packets::_::{{impl}}::deserialize<rkyv::de::deserializers::AllocDeserializer> (self=0x7efff209, deserializer=0x7effedac) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_common/src/packets.rs:43
#4  0x0057ea74 in rswave_server::net::NetHandler::recv (self=0x7efff1d8) at /mnt/c/Users/Lucas/Documents/Dev/rswave/rswave_server/src/net.rs:147

So it seams that the Bus Error is thrown by the deserialization of one of the f64 done by the impl_primitve macro. I am quite new to this library and don't know how do diagnose this issue any further so I am asking for your help.

I should add that for the first packet the deserialization works but not for the next one, and the data received is the same as the data sent.

bug

opened by icanwalkonwater 15

Validation tests failing on Apple M1

test validation::test_alloc::tests::basic_functionality ... FAILED

failures:

---- validation::test_alloc::tests::basic_functionality stdout ----
thread 'validation::test_alloc::tests::basic_functionality' panicked at 'called `Result::unwrap()`
on an `Err` value: ContextError(ArchiveError(Unaligned { ptr: 0x16b97e316, align: 4 }))’,
rkyv_test/src/validation/test_alloc.rs:128:16

I didn’t investigate further but do let me know if I can help.

bug

opened by b8591340 13

Add optional support for granular endianness

I tried to #[derive(Archive)] a u64_be newtype and noticed it was missing, so this PR adds support for optional (feature "endian") granular endianness with non-multibyte impl_primitive and impl_atomic.

I would have asked first, but it was a trivial change. I went by the existing impls assuming they matched rend’s API, so try and give a look and see if something needs to be added or adjusted.

rkyv is such a nice project by the way, keep up the great work!

opened by b8591340 13

Cannot have `validate` and `no_std` with `-Z build-std`

I'm using a custom rustc (esp-rs/rust to be specific) and can't afford to use the standard library. Evidently, bytecheck and ptr_meta have std feature enabled by default (here and here) which leads to this error:

   ...
   Compiling ptr_meta v0.1.4
error[E0658]: use of unstable library feature 'restricted_std'
  |
  = help: add `#![feature(restricted_std)]` to the crate attributes to enable

For more information about this error, try `rustc --explain E0658`.
error: could not compile `ptr_meta` due to previous error

... which indicates that a crate is not doing #![no_std]. I can extern crate alloc though.

Enabled features:

[dependencies.rkyv]
version = "0.7.22"
default_features = false
features = ["strict", "size_32", "validation", "alloc"]

This library already amazes me. Thanks!

opened by mamins1376 10

Unaligned Buffer, only in Debug mode
We have generated and stored a large data structure using rkyv.

let mut serializer = AllocSerializer::<4096>::default(); serializer.serialize_value(&data).unwrap(); let buf = serializer.into_serializer().into_inner(); file.write_all(&buf).await?;

Then when deserializing it with

unsafe { archived_root::<$proto>(&include_bytes!(location)[..]) }

If we do this in debug mode we get the following panic:

assertion failed: `(left == right)` left: `2`, right: `0`: unaligned buffer, expected alignment 8 but found alignment 4

This assert is a debug_assert, so if we compile it in release mode, we get no panic BUT as far as we can tell the data is being deserialized correctly and is usable without errors or data consistency issues.

The assertion seems to be checking for pointer alignment of some kind, as it all seems fine in release the alignment issue being reported by the debug assert seems strange to me.

How best can we investigate this further?
opened by LLBlumire 10
Is it possible to implement Serialize/Deserialize for serde_json Value type?

Hello, Is it possible to implement Serialize/Deserialize for serde_json Value type?

For example, it doesn't work for bincode because of required hints for deserelization.
question

opened by madmaxio 10
Upgrade from 0.6 to 0.7 and archive(copy)
Hi,

I am in the process of upgrading from 0.6 to 0.7 and I am having an issue with archive(copy) . It seems that the copy feature is now behind a feature flag and requires nightly? Not sure if that is the case or it is a different feature?

My same code is below:

#[derive(Debug, Copy, Clone, PartialEq, rkyv::Archive, rkyv::Serialize)] #[archive(copy)] pub struct Value { id: u32, value: f64, } #[derive(Debug, rkyv::Archive, rkyv::Serialize)] struct ValueIndex { items: Vec<Value> }
question
opened by d00z3l 9

Cannot archive as Self with PhantomData

Looks like types containing PhantomData currently can't be archived as Self.

#[derive(rkyv::Archive)]
#[archive(as = "Self")]
struct Test<T> {
    phantom: std::marker::PhantomData<T>,
}

The above code currently (rkyv 0.7.39) results in

error[E0308]: mismatched types
   --> src/main.rs:7:10
    |
7   | #[derive(Archive)]
    |          ^^^^^^^
    |          |
    |          expected associated type, found struct `PhantomData`
    |          arguments to this function are incorrect
    |
    = note: expected raw pointer `*mut <PhantomData<T> as Archive>::Archived`
               found raw pointer `*mut PhantomData<T>`
    = help: consider constraining the associated type `<PhantomData<T> as Archive>::Archived` to `PhantomData<T>` or calling a method that returns `<PhantomData<T> as Archive>::Archived`
    = note: for more information, visit https://doc.rust-lang.org/book/ch19-03-advanced-traits.html

Since PhantomData archives into itself I expected this to work.

Probably caused by some complications in the Archive derive macro?

opened by MaxOhn 8

Flattening RelPtr fields in a Rkyv Archive data struct

If I have a struct using #[derive(rkyv::Archive)], a new struct with "Archive" in the name is created. This struct has the same fields as the normal struct, except that it wraps them in RelPtr and related types.

This is fine if your code is the one and only user of these structs. However, what if you need to pass the structs to third-party code, like as a function argument?

I can call .deserialize() to get back the normal struct without the RelPtrs, but that defeats the purpose of Rkyv.

Since Rkyv supports trait objects, I could create a trait object to wrap my struct, and serialize the trait object. However, then I am permanently committing to dynamic dispatch.

What I was thinking was whether there's a way to flatten the RelPtrs without cloning any data. We could walk the whole data structure, flattening the RelPtrs to normal pointers, and return a data struct with all borrowed data and a corresponding lifetime parameter.

For example, ArchivedString would become &str, ArchivedVec<T> would become &Vec<T>, and so on. Bonus points if this could be Cow<str> and Cow<Vec<T>>.

This wouldn't work for all possible data structs. In particular, when we hit a complex type like ArchivedVec<T>, it is necessary that T does not contain any more fields that need to be flattened, because we want to point to the Vec<T> that lives in Rkyv's memory block. I think in general, we would only support nested tuples and structs; vectors, hash maps, and so on would need to be leaves. So it would work for fixed-size types like i32, but may not work with String. (It would be really nice to figure out a way to have at least String or &str work in vectors.)

I could write such a conversion function manually for all my Rkyv types, but that is cumbersome, and it feels like it belongs in the core library, at least as a feature.

Thoughts?

CC @Manishearth @zbraniecki
enhancement

opened by sffc 8
Provide `#[archive(as = "...")]` for users to provide their own archive types
With the changes around ArchiveCopy (#119) the idea of a type being archiveable as itself needs to be changed too.

Adding #[archive(self)] to a type should:

Add T: Archive<Archived = Self> bounds for each of the fields

Error on any uses of #[archive_attr] or other attributes except #[archive(copy_safe)]

Ensure that enums have an appropriate repr

Not generate an archived version of the type

enhancement
opened by djkoloski 7

Empty ArchivedHashMap panics

I found that a get on an empty archived hashmap panics, at least on my system. As best I can tell it has to do with using 0 as the modulus divisor which results in a divide by 0 panic. Below is sample code that reproduces the error.

#[derive(serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "snake_case")]
#[derive(rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(compare(PartialEq))]
#[archive_attr(derive(bytecheck::CheckBytes, Debug))]
#[derive(Clone, PartialEq, Default)]
pub struct BadMap {
    map: HashMap<u32, u32>,
}

#[test]
fn empty_hashmap_panic() {
    let map = BadMap::default();

    let bytes = rkyv::to_bytes::<_, 256>(&map).unwrap();

    let map_archive = rkyv::check_archived_root::<BadMap>(&bytes[..]).unwrap();

    assert!(map_archive.map.get(&123).is_none());
}

and the resulting stack trace:

running 1 test
thread 'rkyv::empty_hashmap_panic' panicked at 'attempt to calculate the remainder with a divisor of zero', /Users/zschoenb/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/collections/hash_index/mod.rs:66:30
stack backtrace:
   0: rust_begin_unwind
             at /rustc/83356b78c4ff3e7d84e977aa6143793545967301/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/83356b78c4ff3e7d84e977aa6143793545967301/library/core/src/panicking.rs:65:14
   2: core::panicking::panic
             at /rustc/83356b78c4ff3e7d84e977aa6143793545967301/library/core/src/panicking.rs:114:5
   3: rkyv::collections::hash_index::ArchivedHashIndex::index
             at /Users/zschoenb/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/collections/hash_index/mod.rs:66:30
   4: rkyv::collections::hash_map::ArchivedHashMap<K,V>::find
             at /Users/zschoenb/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/collections/hash_map/mod.rs:62:9
   5: rkyv::collections::hash_map::ArchivedHashMap<K,V>::get
             at /Users/zschoenb/.cargo/registry/src/github.com-1ecc6299db9ec823/rkyv-0.7.39/src/collections/hash_map/mod.rs:118:9
   6: rust_scratch::rkyv::empty_hashmap_panic
             at ./src/rkyv.rs:258:13
   7: rust_scratch::rkyv::empty_hashmap_panic::{{closure}}
             at ./src/rkyv.rs:251:26
   8: core::ops::function::FnOnce::call_once
             at /rustc/83356b78c4ff3e7d84e977aa6143793545967301/library/core/src/ops/function.rs:507:5
   9: core::ops::function::FnOnce::call_once
             at /rustc/83356b78c4ff3e7d84e977aa6143793545967301/library/core/src/ops/function.rs:507:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test rkyv::empty_hashmap_panic ... FAILED

failures:

failures:
    rkyv::empty_hashmap_panic

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.03s

opened by zach-schoenberger 0

Add optional support for serde's data model

Seeing how rkyv's type system is open, it should be possible to support the serde data model on top of rkyv. This adaptor would allow integration with the serde ecosystem and could be a path to greater rkyv adoption. Once this integration exists, the serde ecosystem crates could even opt into enhanced rkyv support.

opened by LegNeato 0

`PartialEq` impl for `ArchivedResult` leads to confusing error messages

This isn't really an issue with rkyv, and is maybe more of a diagnostic issue in rust (1.65.0), but I wanted to document it here anyways.

The following example leads to a confusing error message:

extern crate rkyv;

struct Foo {}

fn main() {
    let x: Result<i32, Foo> = Ok(0);
    let y: Result<i32, Foo> = Ok(1);

    assert!(x != y);
}

error[E0308]: mismatched types
 --> src/main.rs:9:18
  |
9 |     assert!(x != y);
  |                  ^ expected enum `ArchivedResult`, found enum `Result`
  |
  = note: expected enum `ArchivedResult<_, _>`
             found enum `Result<i32, Foo>`
help: try wrapping the expression in a variant of `ArchivedResult`
  |
9 |     assert!(x != rkyv::result::ArchivedResult::Ok(y));
  |                  +++++++++++++++++++++++++++++++++ +
9 |     assert!(x != rkyv::result::ArchivedResult::Err(y));
  |                  ++++++++++++++++++++++++++++++++++ +

For more information about this error, try `rustc --explain E0308`.
error: could not compile `rkyv-test` due to previous error

The real reason for the error is that Foo doesn't implement PartialEq, but this is hidden behind a confusing rkyv-related error message. It's obvious what the problem is in this simple example, but can be confusing in larger code where rkyv isn't even involved.

rkyv has the definition:

impl<T, U: PartialEq<T>, E, F: PartialEq<E>> PartialEq<Result<T, E>> for ArchivedResult<U, F>

Without the extern crate rkyv in the above example, we get a much more helpful error message:

error[E0369]: binary operation `!=` cannot be applied to type `Result<i32, Foo>`
 --> src/main.rs:9:15
  |
9 |     assert!(x != y);
  |             - ^^ - Result<i32, Foo>
  |             |
  |             Result<i32, Foo>
  |
note: an implementation of `PartialEq` might be missing for `Foo`
 --> src/main.rs:3:1
  |
3 | struct Foo {}
  | ^^^^^^^^^^ must implement `PartialEq`
help: consider annotating `Foo` with `#[derive(PartialEq)]`
  |
3 | #[derive(PartialEq)]
  |

For more information about this error, try `rustc --explain E0369`.
error: could not compile `rkyv-test` due to previous error

opened by stevenengler 0

fails to build on 32-bit

While packaging stuff for debian I noticed some build failures on 32-bit architectures.

The first is in rkyv_derive and happens with default features.

error: literal out of range for `usize`
   --> rkyv_derive/src/archive.rs:785:41
    |
785 |                         4_294_967_296..=18_446_744_073_709_551_615 => IntRepr::U64,
    |                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: the literal `18_446_744_073_709_551_615` does not fit into the type `usize` whose range is `0..=4294967295`

My fix was to change the match statement to.

let int_repr = match data.variants.len() as u128 {

This works, but I'm not sure what if-any impact it has on optimisation of the code. Possiblly something with cfg attributes would be more efficient.

After fixing that, I ran into a second issue in the rkyv crate, but only when the (non-default) size_64 option is enabled.

error[E0599]: the method `as_ptr` exists for struct `rel_ptr::RelPtr<T, i64>`, but its trait bounds were not satisfied
   --> rkyv/src/boxed.rs:19:27
    |
19  |         unsafe { &*self.0.as_ptr() }
    |                           ^^^^^^ method cannot be called on `rel_ptr::RelPtr<T, i64>` due to unsatisfied trait bounds
    |
   ::: rkyv/src/rel_ptr/mod.rs:267:1
    |
267 | pub struct RelPtr<T: ArchivePointee + ?Sized, O> {
   | ------------------------------------------------ method `as_ptr` not found for this
   |
    = note: the following trait bounds were not satisfied:
            `i64: Offset`

And a bunch more similar errors, I don't know if it makes any sense to build rkyv with the size_64 option on a system that can't actually handle data structures that size.

opened by plugwash 1

Mutating a BTreeMap or OrderedMap backed with memmap.

I've been looking for a straightforward way to have a BTreeMap or OrderedMap backed with a memmaped buffer, and it sounds like this is possible with rkyv. If I have a dataset that is ~60GB, I would like to have file-based indices that I can hash to or range query.

Does rkyv make sense for this use case?

In the docs, there is a reference to using memmapped files here:

https://rkyv.org/architecture/relative-pointers.html

I am not finding any examples where this is being done.

Would it be possible to have an ArchivedBTreeMap backed by a &mut [u8] that came from memmap2?

opened by hallahan 6
"variants never constructed" lint regression in Rust 1.64.0
I'm using a fairly old version of rkyv (0.4.3) and wondering if this lint warning is also appearing for users in later versions of rkyv:

warning: variants `Sha512Result`, `Sha512Trunc256Result`, `SuspendError`, `Uninitialized` and `IdMismatch` are never constructed --> services\engine-sha512\src\api.rs:5:5 | 3 | #[derive(Debug, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] | ------------- variants in this enum 4 | pub enum Sha2Result { 5 | Sha512Result([u8; 64]), | ^^^^^^^^^^^^ 6 | Sha512Trunc256Result([u8; 32]), | ^^^^^^^^^^^^^^^^^^^^ 7 | SuspendError, | ^^^^^^^^^^^^ 8 | Uninitialized, | ^^^^^^^^^^^^^ 9 | IdMismatch, | ^^^^^^^^^^ | = note: `#[warn(dead_code)]` on by default

This just started appearing in Rust 1.64.0; in Rust 1.63.0 and older, such warnings were not emitted.

We try to keep our builds 100% free of warnings, so we can actually see warnings that matter when the builds happen. Unfortunately, we use a lot of rkyv so with this regression our build logs are overwhelmed with Rust complaining about Archive-derived enums not being used.

I tried to actually construct the derived enums with some code like this:

pub fn use_rust_1_64_enums(&self, val: u32) -> ArchivedSha2Result { match val { 0 => ArchivedSha2Result::Sha512Result([0; 64]), 1 => ArchivedSha2Result::Sha512Trunc256Result([0; 32]), 2 => ArchivedSha2Result::IdMismatch, 3 => ArchivedSha2Result::SuspendError, _ => ArchivedSha2Result::Uninitialized, } }

But even if I call this code and pass the result on to something else, Rust is insisting that we never construct that enum type (tried creating both the ArchivedSha2Result and the Sha2ResultResolver and casting the results to something the compiler shouldn't disregard). I'm guessing maybe the derive creates some extra enum types that I'm not aware of or perhaps it does something exciting with the discriminants that cause the rust linter to think variants are not being used.

Furthermore, if I add #[allow(dead_code)] to the enum definition, it still throws the warning; adding #[allow(dead_code)] to every element has the same result, more warnings.

The only way I've been able to silence this is to use the "big hammer" of #![allow(dead_code)] to the entire file.

So, the current work-around I have is to move all rkyv-derive types to its own file, and then drop the file-wide silencer on top.

This is an acceptable work-around; but now I am left with lingering curiosity!

If this is also happening in the latest rkyv, maybe this is a compiler regression? If it's not happening, something else subtle going on that I'm not understanding in order to construct the derived variants so I can make the warning go away...?
bug
opened by bunnie 3

Releases(v0.7.39)

v0.7.39(Jun 26, 2022)
#278 Adds null checks for pointers returned from alloc and realloc.

Fixes some places in the Archive derive macro that was using ::rkyv as the crate path instead of a path provided through #[archive(crate = "...")].

Thanks to @robinhundt for filing an issue!
Source code(tar.gz)
Source code(zip)
v0.7.38(May 3, 2022)
#263 Adds the new generic Map wrapper (thanks to @zyansheep)

#264 Fixes some incorrect DeserializeWith impls for the AsBox and Raw wrapper types

#265 Adds support for validating CStr and CString

#268 Adds the new Skip wrapper to skip serializing fields

#269 Updates support for the uuid crate to 1.0 (thanks to @Dylan-DPC)

Thanks to everyone who contributed issues and PRs!
Source code(tar.gz)
Source code(zip)
v0.7.37(Mar 30, 2022)

This bugfix release properly gates Arc support behind the has_atomics flag to fix rkyv on thumbv6m-none-eabi and other platforms without atomics support running under no_std.

Thanks to @moosingin3space for the bug report.
Source code(tar.gz)
Source code(zip)
v0.7.36(Mar 15, 2022)

This bugfix release introduces the Unsafe wrapper type for serializing and deserializing Cell and UnsafeCell.
Source code(tar.gz)
Source code(zip)
v0.7.35(Mar 3, 2022)

This bugfix release corrects documentation on AlignedVec::into_boxed_slice that was not correctly updated with the previous release.

Thanks to @ralfjung for spotting the issue.
Source code(tar.gz)
Source code(zip)
v0.7.34(Mar 3, 2022)

This bugfix release corrects some minor potential UB in AlignedVec::into_boxed_slice. It is a second location of a similar issue patched in 0.7.29.

Thanks to @saethlin for identifying and reporting the issue securely.
Source code(tar.gz)
Source code(zip)
v0.7.33(Feb 19, 2022)

This bugfix release corrects macro hygiene when using the derive macros inside of a macro_rules! context (#252).

Thanks to @DarrenTsung for reporting the issue.
Source code(tar.gz)
Source code(zip)
v0.7.32(Feb 16, 2022)
This bugfix release addresses the following issues:

Adds the new Raw wrapper for maximum ser/de and validation performance on raw bytes

Implements PartialEq for ArchivedDuration (#247)

Improves error messaging for unaligned buffer errors (#245)

Fixes an issue where the derive macro could generate ambiguous impl code (#251)

Updates hashbrown support to 0.12 (#248)

Thanks everyone who reported issues and to @petar-dambovaliev and @kastolars for filing pull requests!
Source code(tar.gz)
Source code(zip)
v0.7.31(Jan 26, 2022)

This bugfix release extends Send and Sync impls to a few more types.

Thanks to @nagisa for filing an issue!
Source code(tar.gz)
Source code(zip)
v0.7.30(Jan 26, 2022)
This bugfix release addresses the following issues:

#241 Support for niching nonzero integers has been added. Using #[with(Niche)] on Option<NonZero*> fields will only use as much space as the underlying integer type while archived.

#240 SystemTime can now be serialized and deserialized using #[with(UnixTimestamp)], which converts the time to a duration since the UNIX epoch.

#242 SharedSerializeMapError has been made Send and Sync.

#243 Malformed arguments to #[archive(...)] and #[archive_attr(...)] will now properly error. Previously, malformed arguments caused the attribute to be silently ignored.

Thanks to @nagisa, @seritools, @kassens, and @ok-nick for reporting issues and filing feature requests.
Source code(tar.gz)
Source code(zip)
v0.7.29(Jan 8, 2022)
This bugfix release addresses the following issues:

Validation failure when using BTreeMap as a struct member in certain situations (#239)

Soundness hole in AlignedVec::into_vec when the resulting Vec is dropped

General docs improvements

Thanks to @Jasper-Bekkers for filing an issue!
Source code(tar.gz)
Source code(zip)
v0.7.28(Dec 19, 2021)

This bugfix release removes doctest dependencies on the validation feature.

Thanks to @remilauzier for filing an issue!
Source code(tar.gz)
Source code(zip)
v0.7.27(Dec 18, 2021)
0.7.27 is a bugfix release that fixes some issues and adds a few new features:

#234 Adds direct support for the hashbrown crate. This allows users to archive hash maps and hash sets without also needing std.

New helper functions have been added to make serialization and deserialization easier than ever: to_bytes, from_bytes_unchecked, and from_bytes. These are utility functions to make working with serialization and deserialization easier, but may not provide the best possible performance for individual cases.

A usage-tracking scratch space adapter has been added for performance tuning.

A debug assertion message regarding buffer alignment has been fixed to provide the correct error message.

The HeapScratch struct can now be used with const parameter N=0.

Examples have been updated.

Thanks to @Miaxos for feature requests and a PR!
Source code(tar.gz)
Source code(zip)
v0.7.26(Dec 8, 2021)

This bugfix release adds validation support for indexmap types. Thanks to @nagisa for the feature request.
Source code(tar.gz)
Source code(zip)
v0.7.25(Dec 3, 2021)

This bugfix release improves atomics support detection to provide atomics support to more targets.

Thanks to @tobz for the PR.
Source code(tar.gz)
Source code(zip)
v0.7.24(Nov 29, 2021)

This bugfix release fixes #227 by removing the custom pad impl for BufferSerializer. This will cause the serializer to always write zeroes to padding to avoid surprises when reusing buffer memory. Thanks to @Diomendius for reporting an issue.
Source code(tar.gz)
Source code(zip)
v0.7.23(Nov 24, 2021)

0.7.23 is a bugfix release that fixes some unused code warnings on platforms without atomic support.
Source code(tar.gz)
Source code(zip)
v0.7.22(Nov 20, 2021)

This bugfix release improves the documentation for generated types. Thanks to @dvc94ch for the suggestion!
Source code(tar.gz)
Source code(zip)
v0.7.21(Nov 20, 2021)
This bugfix release fixes #223 by expanding the CopyOptimize wrapper to support boxed slices. You can now do:

#[derive(Archive, Serialize)] pub struct Example<'a> { #[with(CopyOptimize, RefAsBox)] slice: &'a [u32], }

and serialize a field as a box with copy optimization. Thanks to @tobz for the feature request.
Source code(tar.gz)
Source code(zip)
v0.7.20(Oct 23, 2021)
rkyv 0.7.20 is a bugfix release that addresses the following issues:

#212 All types can now be boxed with the new AsBox and RefAsBox wrappers.

#210 Copy optimizations are now available on stable with the new CopyOptimize wrapper. Please read the safety documentation carefully.

Thanks to everyone who filed issues and made feature requests.
Source code(tar.gz)
Source code(zip)
v0.7.19(Sep 29, 2021)
rkyv 0.7.19 is a bugfix release that addresses the following issues:

#207 Adds support for the smallvec crate

#208 Fixed validation logic for empty BTreeMaps

Thanks to everyone who filed issues and made feature requests.
Source code(tar.gz)
Source code(zip)
v0.7.18(Sep 15, 2021)

rkyv 0.7.18 addresses a deficiency in the bounds(...) argument to the archive attribute. Bounds can now be provided for the Archive, Serialize, and Deserialize traits individually.
Source code(tar.gz)
Source code(zip)
v0.7.17(Sep 14, 2021)

rkyv 0.7.17 adds the new Niche wrapper, which allows fields of the form Option<Box<T>> to use less space.
Source code(tar.gz)
Source code(zip)
v0.7.16(Sep 3, 2021)

rkyv 0.7.16 extends support to rend types and adds a compile error when multiple mutually exclusive features are active.

This release also contains rkyv_dyn 0.7.6, which fixes a few things that broke during rkyv's evolution.

Thanks to @b8591340 especially for filing issues and PRs.
Source code(tar.gz)
Source code(zip)
v0.7.15(Aug 30, 2021)

This bugfix release extends HashSet support to hash sets with custom hashers. Thanks to @mustafa-guler for the PR!
Source code(tar.gz)
Source code(zip)
v0.7.14(Aug 27, 2021)

This bugfix release implements DoubleEndedIterator for the Option and Result iterators (#191, thanks @l4l for providing a PR) and fixes raw identifier support (#191, thanks @dovahcrow for reporting the issue).
Source code(tar.gz)
Source code(zip)
v0.7.13(Aug 20, 2021)

This bugfix release fixes a panic related to buffers that are undersized for root objects. Thanks to @krl for reporting the issue.
Source code(tar.gz)
Source code(zip)
v0.7.12(Aug 20, 2021)
This bugfix release address the following issues:

#188 The Self type is now properly supported while deriving implementations for Archive, Serialize, and Deserialize.

#189 A new ArchivedResult type has been added, enabling support for archiving Result.

Thanks to @krl for filing issues!
Source code(tar.gz)
Source code(zip)
v0.7.11(Aug 12, 2021)

This bugfix release fixes a panic while archiving and accessing empty BTreeMaps.

Thanks to @ianoc for the bug report and unit test!
Source code(tar.gz)
Source code(zip)
v0.7.10(Aug 9, 2021)

This bugfix release fixes tinyvec integration.

Thanks to @Kixiron for reporting the issue!
Source code(tar.gz)
Source code(zip)