Hi I tried to run this project after installing capnproto and flatbuffers but I'm getting a bunch of compilation errors while calling cargo bench:

error[E0432]: unresolved imports `rust_serialization_benchmark::datasets::log::Log`, `rust_serialization_benchmark::datasets::log::Logs`
  --> benches/bench.rs:41:55
   |
41 |     use rust_serialization_benchmark::datasets::log::{Log, Logs};
   |                                                       ^^^  ^^^^ no `Logs` in `datasets::log`
   |                                                       |
   |                                                       no `Log` in `datasets::log`
   |                                                       help: a similar name exists in the module: `log`

error[E0432]: unresolved import `rust_serialization_benchmark::datasets::log::ArchivedAddress`
   --> benches/bench.rs:140:17
    |
140 |             use rust_serialization_benchmark::datasets::log::ArchivedAddress;
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `ArchivedAddress` in `datasets::log`

error[E0432]: unresolved imports `rust_serialization_benchmark::datasets::mesh::Mesh`, `rust_serialization_benchmark::datasets::mesh::Triangle`
   --> benches/bench.rs:188:56
    |
188 |     use rust_serialization_benchmark::datasets::mesh::{Mesh, Triangle};
    |                                                        ^^^^  ^^^^^^^^ no `Triangle` in `datasets::mesh`
    |                                                        |
    |                                                        no `Mesh` in `datasets::mesh`
    |                                                        help: a similar name exists in the module: `mesh`

error[E0432]: unresolved imports `rust_serialization_benchmark::datasets::minecraft_savedata::Player`, `rust_serialization_benchmark::datasets::minecraft_savedata::Players`
   --> benches/bench.rs:316:70
    |
316 |     use rust_serialization_benchmark::datasets::minecraft_savedata::{Player, Players};
    |                                                                      ^^^^^^  ^^^^^^^ no `Players` in `datasets::minecraft_savedata`
    |                                                                      |
    |                                                                      no `Player` in `datasets::minecraft_savedata`

error[E0432]: unresolved import `rust_serialization_benchmark::datasets::minecraft_savedata::ArchivedGameType`
   --> benches/bench.rs:404:17
    |
404 |             use rust_serialization_benchmark::datasets::minecraft_savedata::ArchivedGameType;
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `ArchivedGameType` in `datasets::minecraft_savedata`

error[E0433]: failed to resolve: could not find `cp` in `log`
  --> benches/bench.rs:79:70
   |
79 |             .get_root::<rust_serialization_benchmark::datasets::log::cp::logs::Reader>()
   |                                                                      ^^ could not find `cp` in `log`

error[E0433]: failed to resolve: could not find `fb` in `log`
  --> benches/bench.rs:98:62
   |
98 |                 rust_serialization_benchmark::datasets::log::fb::Logs,
   |                                                              ^^ could not find `fb` in `log`

error[E0433]: failed to resolve: could not find `fb` in `log`
   --> benches/bench.rs:108:82
    |
108 |                 flatbuffers::root::<rust_serialization_benchmark::datasets::log::fb::Logs>(bytes)
    |                                                                                  ^^ could not find `fb` in `log`

error[E0433]: failed to resolve: could not find `cp` in `mesh`
   --> benches/bench.rs:224:71
    |
224 |             .get_root::<rust_serialization_benchmark::datasets::mesh::cp::mesh::Reader>()
    |                                                                       ^^ could not find `cp` in `mesh`

error[E0433]: failed to resolve: could not find `fb` in `mesh`
   --> benches/bench.rs:241:63
    |
241 |                 rust_serialization_benchmark::datasets::mesh::fb::Mesh,
    |                                                               ^^ could not find `fb` in `mesh`

error[E0433]: failed to resolve: could not find `fb` in `mesh`
   --> benches/bench.rs:249:83
    |
249 |                 flatbuffers::root::<rust_serialization_benchmark::datasets::mesh::fb::Mesh>(bytes)
    |                                                                                   ^^ could not find `fb` in `mesh`

error[E0433]: failed to resolve: could not find `cp` in `minecraft_savedata`
   --> benches/bench.rs:351:106
    |
351 |         let data = message_reader.get_root::<rust_serialization_benchmark::datasets::minecraft_savedata::cp::players::Reader>().unwrap();
    |                                                                                                          ^^ could not find `cp` in `minecraft_savedata`

error[E0433]: failed to resolve: could not find `fb` in `minecraft_savedata`
   --> benches/bench.rs:367:77
    |
367 |                 rust_serialization_benchmark::datasets::minecraft_savedata::fb::Players,
    |                                                                             ^^ could not find `fb` in `minecraft_savedata`

error[E0433]: failed to resolve: could not find `fb` in `minecraft_savedata`
   --> benches/bench.rs:375:77
    |
375 |                 rust_serialization_benchmark::datasets::minecraft_savedata::fb::Players,
    |                                                                             ^^ could not find `fb` in `minecraft_savedata`

error[E0412]: cannot find type `LogsSchema` in module `rust_serialization_benchmark::datasets::log`
   --> benches/bench.rs:178:79
    |
178 |     data: alkahest::Unpacked<'a, rust_serialization_benchmark::datasets::log::LogsSchema>,
    |                                                                               ^^^^^^^^^^ not found in `rust_serialization_benchmark::datasets::log`

error[E0412]: cannot find type `MeshSchema` in module `rust_serialization_benchmark::datasets::mesh`
   --> benches/bench.rs:308:80
    |
308 |     mesh: alkahest::Unpacked<'a, rust_serialization_benchmark::datasets::mesh::MeshSchema>,
    |                                                                                ^^^^^^^^^^ not found in `rust_serialization_benchmark::datasets::mesh`

error[E0412]: cannot find type `PlayersSchema` in module `rust_serialization_benchmark::datasets::minecraft_savedata`
   --> benches/bench.rs:439:69
    |
439 |         rust_serialization_benchmark::datasets::minecraft_savedata::PlayersSchema,
    |                                                                     ^^^^^^^^^^^^^ not found in `rust_serialization_benchmark::datasets::minecraft_savedata`

error[E0609]: no field `logs` on type `&_`
  --> benches/bench.rs:58:25
   |
58 |         for log in data.logs.iter() {
   |                         ^^^^

error[E0609]: no field `logs` on type `&_`
   --> benches/bench.rs:133:29
    |
133 |             for log in data.logs.iter() {
    |                             ^^^^

error[E0599]: no method named `as_ref` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:142:30
    |
142 |             for i in 0..logs.as_ref().logs.len() {
    |                              ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `logs` is a function, perhaps you wish to call it

error[E0599]: no method named `as_mut` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:143:36
    |
143 |                 let mut log = logs.as_mut().logs_pin().index_pin(i);
    |                                    ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `logs` is a function, perhaps you wish to call it

error[E0609]: no field `triangles` on type `&_`
   --> benches/bench.rs:205:30
    |
205 |         for triangle in data.triangles.iter() {
    |                              ^^^^^^^^^

error[E0609]: no field `triangles` on type `&_`
   --> benches/bench.rs:272:34
    |
272 |             for triangle in mesh.triangles.iter() {
    |                                  ^^^^^^^^^

error[E0599]: no method named `as_ref` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:277:30
    |
277 |             for i in 0..mesh.as_ref().triangles.len() {
    |                              ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `mesh` is a function, perhaps you wish to call it

error[E0599]: no method named `as_mut` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:278:41
    |
278 |                 let mut triangle = mesh.as_mut().triangles_pin().index_pin(i);
    |                                         ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `mesh` is a function, perhaps you wish to call it

error[E0609]: no field `players` on type `&_`
   --> benches/bench.rs:333:28
    |
333 |         for player in data.players.iter() {
    |                            ^^^^^^^

error[E0609]: no field `players` on type `&_`
   --> benches/bench.rs:399:32
    |
399 |             for player in data.players.iter() {
    |                                ^^^^^^^

error[E0599]: no method named `as_ref` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:406:33
    |
406 |             for i in 0..players.as_ref().players.len() {
    |                                 ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `players` is a function, perhaps you wish to call it

error[E0599]: no method named `as_mut` found for struct `Pin<&mut _>` in the current scope
   --> benches/bench.rs:407:42
    |
407 |                 let mut player = players.as_mut().players_pin().index_pin(i);
    |                                          ^^^^^^ method not found in `Pin<&mut _>`
    |
    = note: the method was found for
            - `Pin<P>`
    = note: `players` is a function, perhaps you wish to call it

Some errors have detailed explanations: E0412, E0432, E0433, E0599, E0609.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `rust_serialization_benchmark` due to 29 previous errors
warning: build failed, waiting for other jobs to finish...
warning: `rust_serialization_benchmark` (lib test) generated 81 warnings (81 duplicates)
error: build failed

any idea why?

Thanks for this benchmark!

I like BARE's dense encoding quite a bit, so thought it'd be interesting to see how it stacks up against the competition. I've only run the benchmark for a few of the libraries that I've been considering (partially because some of the other benchmarks don't compile, or need external tools which weren't easy to figure out how to install).

EDIT: Here are the results for the formats I was particularly interested in. BARE seems to have an advantage when it comes to size, which is interesting to know :).

log

This data set is composed of HTTP request logs that are small and contain many strings.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 614.03 us | 2.9164 ms | 765778 | 312739 | 264630 | | bincode | 474.82 us | 2.2372 ms | 1045784 | 374305 | 311761 | | cbor | 1.6595 ms | 5.9777 ms | 1407835 | 407372 | 324081 | | rkyv | 183.76 us | 1.3184 ms* 1.9008 ms* | 1011488 | 392809 | 331932 |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 1.0035 ns* 591.06 us* | 9.5201 us* 600.61 us* | 44.896 us |

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 29.93% | 45.21% | 100.00% | 100.00% | 100.00% | | bincode | 38.70% | 58.93% | 73.23% | 83.55% | 84.88% | | cbor | 11.07% | 22.06% | 54.39% | 76.77% | 81.66% | | rkyv | 100.00% | 100.00%* 69.36%* | 75.71% | 79.62% | 79.72% |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 100.00%* 0.00%* | 100.00%* 1.59%* | 100.00% |

mesh

This data set is a single mesh. The mesh contains an array of triangles, each of which has three vertices and a normal vector.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 4.1581 ms | 7.6070 ms | 6000003 | 5380817 | 5345900 | | bincode | 4.9309 ms | 3.3888 ms | 6000008 | 5380823 | 5345890 | | cbor | 36.111 ms | 59.945 ms | 13122324 | 7527423 | 6759658 | | rkyv | 474.21 us | 329.20 us* 318.65 us* | 6000008 | 5380822 | 5345892 |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 1.0050 ns* 12.374 ns* | 35.804 us* 34.760 us* | 192.25 us |

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 11.40% | 4.19% | 100.00% | 100.00% | 100.00% | | bincode | 9.62% | 9.40% | 100.00% | 100.00% | 100.00% | | cbor | 1.31% | 0.53% | 45.72% | 71.48% | 79.09% | | rkyv | 100.00% | 96.80%* 100.00%* | 100.00% | 100.00% | 100.00% |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 100.00%* 8.12%* | 97.08%* 100.00%* | 100.00% |

minecraft_savedata

This data set is composed of Minecraft player saves that contain highly structured data.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 659.83 us | 2.7228 ms | 356311 | 213270 | 198488 | | bincode | 473.08 us | 1.9039 ms | 569975 | 240897 | 232423 | | cbor | 1.6809 ms | 5.5620 ms | 1109821 | 347812 | 274526 | | rkyv | 303.12 us | 1.1217 ms* 1.6373 ms* | 596952 | 254571 | 219976 |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 495.72 us* | 255.19 ns* 494.63 us* | 2.7913 us |

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

| Format / Lib | Serialize | Deserialize | Size | Zlib | Zstd | |---|--:|--:|--:|--:|--:| | bare | 45.94% | 41.20% | 100.00% | 100.00% | 100.00% | | bincode | 64.07% | 58.92% | 62.51% | 88.53% | 85.40% | | cbor | 18.03% | 20.17% | 32.11% | 61.32% | 72.30% | | rkyv | 100.00% | 100.00%* 68.51%* | 59.69% | 83.78% | 90.23% |

Zero-copy deserialization speed

| Format / Lib | Access | Read | Update | |---|--:|--:|--:| | rkyv | 100.00%* | 100.00%* 0.05%* | 100.00% |

Cloned the repo and ran:

$ cargo bench

Machine: AMD64 / Mx-linux.

Had a compilation error due to build.rs script.

  --- stderr                                                                          
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Failed, description: "Error while trying to execute `capnp compile`: Failed: Permission denied (os erro
r 13).  Please verify that version 0.5.2 or higher of the capnp executable is installed on your system. See https://capnproto.org/install.html" }', build.rs:9:10                             
  stack backtrace:    
...
...
17:     0x563db33e0c69 - build_script_build::main::h48173f533a2f1a5e
                                 at /home/prataprc/tmp/rust_serialization_benchmark/build.rs:3:5 
...
...

Can't decipher the reasoning behind the tools/ directory. Looks like they contain windows executable. Let me know whether I am missing something here. Thanks,

This is where we start to get into somewhat murky waters. FlatBuffers supports buffer validation, but the dependency needs to get upgraded to 0.8. Cap'n Proto supports validation, but it's lazy so we would need some way of distinguishing that. Maybe splitting the access/read benchmarks into validated/unvalidated for FlatBuffers and rkyv and leaving the Cap'n Proto results as validated.

I've been maintaining several benchmark repos based off of template-benchmarks-rs. I've noticed there are several other benchmark repos that are hard to know about.

I've created rosetta-rs org as a place for these repos if the authors so choose to

• Increase discoverability
• Improve bus factor

See the meta repo for more details

Interested in collaborating?

serde_json is good but simd-json is probably more representative of what end users will actually be comparing against. It's also just a good benchmark to have since it's so popular.

This is the intended way to deserialize abomonation structures, so adding a bench that performs the clone but notes that it's just a clone and not anything more complicated would be a good way to add more context to the benchmark.

In many cases, data is zlib-compressed anyway. Adding a statistic on zlib compression may increase or decrease the relative performance of serialization frameworks.

In this function you use the unsafe keyword for some safe expressions.

We need to mark unsafe operations more precisely using unsafe keyword. Keeping unsafe blocks small can bring many benefits. For example, when mistakes happen, we can locate any errors related to memory safety within an unsafe block. This is the balance between Safe and Unsafe Rust. The separation is designed to make using Safe Rust as ergonomic as possible, but requires extra effort and care when writing Unsafe Rust.

Hope this PR can help you. Best regards. References https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html

I noticed that the structs you're using with serde-based deserializers actually have an owned String in them. However, bincode doesn't need to copy strings (I'm not aware about the others).

I'm actually using this in zoxide right now. Here's my struct definitions:

#[derive(Debug, Deserialize, Serialize)]
pub struct DirList<'a>(#[serde(borrow)] Vec<Dir<'a>>);

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Dir<'a> {
    #[serde(borrow)]
    pub path: Cow<'a, str>,
    pub rank: Rank,
    pub last_accessed: Epoch,
}

Since the necessary bytes are already in the Vec<u8> that you're trying to deserialize, bincode will just give you a pointer to it instead of copying a whole new String into memory, which is much faster. You can verify this by checking that the deserialized struct actually contains a Cow::Borrowed. You might be able to use a regular &str too, although I'm not sure how bincode would handle different endianness in that case.

P.S. thanks for your fantastic work on rkyv and these benchmarks! Not only is rkyv's performance amazing, it's ideas like structver that would make this a serious contender for anyone wanting a serialization library. I'm currently using a poor man's structver myself, so the idea of rkyv handling this for me sounds absolutely great.

Adding a benchmark including I/O time would be useful. Usually serialization/deserialization involves network or disk I/O, and the medium affects the overall performance.

For example, it is unclear from the table in README.md whether rkyv/speedy are faster than prost in real scenarios because the disk read is faster with protobuf. Probably rkyv/speedy are faster than protobuf in LAN. Are they faster in SSD? We do not know.

These additions will help readers make a quick choice on the library.

• HDD with low IOPS or RPM + Serialization/Deseiralization
• HDD with high IOPS or RPM + Serialization/Deseiralization
• SSD + Serialization/Deseiralization
• Network 10/100/1000 Mbps + Serialization/Deseiralization

It looks like we could be carrying around scratch space and reusing it based on this issue: https://github.com/google/flatbuffers/issues/6511

I think that's a little much to ask considering how much serialization code is already written for this benchmark, but it's worth keeping in mind.

Enabling profile-guided optimization will provide some numbers that are the best they can be for each framework. It might be worth separating these out from the general numbers so users can get an idea of how much they stand to gain for their efforts.

It would be useful to see benchmarks on memory usage and code size.

dhat is a Rust crate that provides profiling of heap allocations. One could also use Valgrind. Heap is the most important, but it would also be useful to see stats on stack and static memory.

For code size, I like to follow the instructions on min-sized-rust to remove a lot of the noise in the binary.

I may be able to help with this.