This issue was automatically generated. Feel free to close without ceremony if you do not agree with re-licensing or if it is not possible for other reasons. Respond to @cmr with any questions or concerns, or pop over to #rust-offtopic on IRC to discuss.

You're receiving this because someone (perhaps the project maintainer) published a crates.io package with the license as "MIT" xor "Apache-2.0" and the repository field pointing here.

TL;DR the Rust ecosystem is largely Apache-2.0. Being available under that license is good for interoperation. The MIT license as an add-on can be nice for GPLv2 projects to use your code.

Why?

The MIT license requires reproducing countless copies of the same copyright header with different names in the copyright field, for every MIT library in use. The Apache license does not have this drawback. However, this is not the primary motivation for me creating these issues. The Apache license also has protections from patent trolls and an explicit contribution licensing clause. However, the Apache license is incompatible with GPLv2. This is why Rust is dual-licensed as MIT/Apache (the "primary" license being Apache, MIT only for GPLv2 compat), and doing so would be wise for this project. This also makes this crate suitable for inclusion and unrestricted sharing in the Rust standard distribution and other projects using dual MIT/Apache, such as my personal ulterior motive, the Robigalia project.

Some ask, "Does this really apply to binary redistributions? Does MIT really require reproducing the whole thing?" I'm not a lawyer, and I can't give legal advice, but some Google Android apps include open source attributions using this interpretation. Others also agree with it. But, again, the copyright notice redistribution is not the primary motivation for the dual-licensing. It's stronger protections to licensees and better interoperation with the wider Rust ecosystem.

How?

To do this, get explicit approval from each contributor of copyrightable work (as not all contributions qualify for copyright, due to not being a "creative work", e.g. a typo fix) and then add the following to your README:

## License

Licensed under either of

 * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
 * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

### Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any
additional terms or conditions.

and in your license headers, if you have them, use the following boilerplate (based on that used in Rust):

// Copyright 2016 bincode developers
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.

It's commonly asked whether license headers are required. I'm not comfortable making an official recommendation either way, but the Apache license recommends it in their appendix on how to use the license.

Be sure to add the relevant LICENSE-{MIT,APACHE} files. You can copy these from the Rust repo for a plain-text version.

And don't forget to update the license metadata in your Cargo.toml to:

license = "MIT/Apache-2.0"

I'll be going through projects which agree to be relicensed and have approval by the necessary contributors and doing this changes, so feel free to leave the heavy lifting to me!

Contributor checkoff

To agree to relicensing, comment with :

I license past and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.

Or, if you're a contributor, you can check the box in this repo next to your name. My scripts will pick this exact phrase up and check your checkbox, but I'll come through and manually review this issue later as well.

• [x] @TyOverby
• [x] @bitonic
• [x] @erickt
• [x] @crhino
• [x] @jmesmon
• [x] @mikedilger
• [x] @tedsta
• [x] @nyx
• [x] @csherratt
• [x] @badboy
• [x] @jeremyjh
• [x] @reem
• [x] @mohtar
• [x] @SimonSapin

I was about to update the library to the latest version of rustc-serialize (now with associated types), and I noticed the last commit.

Is it necessary, considering that the size limit can be easily enforce through the writer -- e.g. with a BufWriter?

I'm not sure if this is the right place for this, but

I am unable to use bincode to serialize and deserialize structs where the serialize and deserialize traits are implemented instead of derived. For example, I copied the Point struct, and the serialize and deserialize implementations used in the "Serliazation without macros" section of the readme here, https://github.com/serde-rs/serde#serialization-without-macros and attempted to use bincode to serialize and deserialize a Point variable. I was unable to do so and got this error

thread '

' panicked at 'called Result::unwrap() on an Err value: SyntaxError', ../src/libcore/result.rs:738 Process didn't exit successfully: target/debug/tcpLocal (exit code: 101)

If I instead derive the serialize and deserialize traits, I can serialize and deserialize the Point variable fine. Using the copied implementations of serialize and deserialize, I can serialize and deserialize from JSON just fine.

Is this expected? Did I miss something where only serializing with macros is supported in bincode?

• Thanks

Hello!

I am working with a protocol that is 99% compatible with the data structures I already use, bar one thing: it uses c-style strings. I know that bincode is technically its own protocol but if we were to support different string formats it would save me 100 lines of custom code...

...So here's 140 lines of custom code that adds a flag similar to the varint flag that allows the user to select between 'vec-style' encoding and 'c-style' encoding. I have marked it as a draft because I wanted to make sure you were open to merging something like this before I finished it.

Thanks! :christmas_tree:

I see that you've blocked updates of byteorder due to concern for Rust 1.18 compatibility (#381, #380). However, in doing so, you're causing problems for all Rust versions:

error: failed to select a version for `byteorder`.
    ... required by package `bincode v1.3.2`

I know your change isn't technically a semver-breaking, but because of the way Cargo resolves dependencies, it is breaking builds (I have byteorder 1.3.2 elsewhere in the dependency tree, and cargo update can't deal with it).

Paradoxically, in a good-faith attempt to avoid breaking anyone, you've caused more breakage.

Please don't cause pain for all Rust users out of concern for users who use an outdated Rust version.

I didn't see no_std support in the documentation / cargo.toml / issues so I thought I would open one. This library seems perfect for no_std serialization in microcontrollers. What are the biggest barriers to supporting no_std? Would you welcome a PR that did so?

Thanks!

To revisit #236, it would be more convenient for users if i128 support was enabled automatically on compatible compilers. We already argued in July that this was the best option for Rand.

The linked PR shows how this can be achieved without any breakage (except for old nightlies, which are generally not possible to ensure support for anyway).

When serializing a struct with a HashMap it's easy to end up with two values that are equal to each other but serialize to different output. An example that fails sometimes:

#[macro_use] extern crate serde_derive;
extern crate bincode;

use std::collections::HashMap;

#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct MyStruct {
    map: HashMap<u64, u64>,
}

fn main() {
    let mut map = HashMap::new();
    map.insert(0, 0);
    map.insert(1, 1);

    let s1 = MyStruct{map,};
    let e1: Vec<u8> = bincode::serialize(&s1).unwrap();
    let s2: MyStruct = bincode::deserialize(&e1).unwrap();
    let e2: Vec<u8> = bincode::serialize(&s2).unwrap();
    
    println!("struct1 is {:?}", s1);
    println!("struct2 is {:?}", s2);

    assert_eq!(s1, s2);
    assert_eq!(e1, e2);
}

This is most likely because rust is seeding the hash with different values on different runs and bincode serializes in hash order and not key order. This makes it hard to use bincode to hash a struct for deduplication/content addressing purposes.

I have not released this yet but I will in the next few days. https://github.com/serde-rs/serde/pull/1044

trait Serializer {
    /* ... */

    fn is_human_readable(&self) -> bool { true }
}

trait Deserializer<'de> {
    /* ... */

    fn is_human_readable(&self) -> bool { true }
}

Bincode should override these to false to get more compact representation for types like IpAddr and Datetime.

The issue is in bincode_derive crate, where we are adding extra debug info that are leaking enum names in the final binary (which imply less optimization with a bigger binary size, and internal structure leak). I think we should use a proper feature flag to disable this. I'm working on a pull request to address that issue.

It seems there is only one leak in the code in this part of the code. I agree this debug line can be important, but should be optional for release build if required. A feature flag could be used to address that situation, and maybe it is useful for other cases like in #594.

Background

Collections said: The length value is based on your IntEncoding.

But in fact, array length only supports write_fixed_array_length and skip_fixed_array_length.

let arr: [u8; 5] = [10, 20, 30, 40, 50];
let encoded = bincode::encode_to_vec(arr, bincode::config::legacy()).unwrap();
assert_eq!(encoded.as_slice(), &[
    5, 0, 0, 0, 0, 0, 0, 0, // The length, as a u64
    10, 20, 30, 40, 50, // the bytes
]);

let encoded = bincode::encode_to_vec(arr, bincode::config::legacy().skip_fixed_array_length()).unwrap();
assert_eq!(encoded.as_slice(), &[
    // no length
    10, 20, 30, 40, 50, // the bytes
]);

Proposal

We can support variant encodes for array length, which can make our array smaller, especially for arrays smaller than 256 (The cases happened in databend which will operate many small slices)

let arr: [u8; 5] = [10, 20, 30, 40, 50];
let encoded = bincode::encode_to_vec(arr, bincode::config::legacy().write_variant_array_length()).unwrap();
assert_eq!(encoded.as_slice(), &[
    5, // The length, as a single byte
    10, 20, 30, 40, 50, // the bytes
]);

Reference

Add a new const bool in trait InternalArrayLengthConfig:

pub trait InternalArrayLengthConfig {
    const SKIP_FIXED_ARRAY_LENGTH: bool;
    const WIRTE_VARIANT_ARRAY_LENGTH: bool;
}

Implement a config struct:

#[doc(hidden)]
#[derive(Copy, Clone)]
pub struct WriteVariantArrayLength {}

impl InternalArrayLengthConfig for WriteVariantArrayLength {
    const SKIP_FIXED_ARRAY_LENGTH: bool = false;
    const WIRTE_VARIANT_ARRAY_LENGTH: bool = true;
}

Alternative

If API break is allowed, we can make InternalArrayLengthConfig refer to an enum:

enum InternalArrayLengthOption {
    SKIP_FIXED_ARRAY_LENGTH,
    WIRTE_VARIANT_ARRAY_LENGTH,
    WIRTE_FIXED_ARRAY_LENGTH,
}

pub trait InternalArrayLengthConfig {
    const SKIP_FIXED_ARRAY_LENGTH: InternalArrayLengthOption;
}

Other contexts

We have implemented the same feature on bincode v1 on our fork: https://github.com/datafuse-extras/bincode

Rust 1.66 was just released, along with Explicit discriminants on enums with fields. The given example does not compile when adding bincode (2.0.0-rc.2) derive macros to the enum:

$ cargo -V
# cargo 1.66.0 (d65d197ad 2022-11-15)

#[derive(Decode, Encode)]
#[repr(u8)]
enum Foo {
    A(u8),
    B(i8),
    C(bool) = 42,
}

# error: Invalid rust syntax, expected ident, got Some(Punct { ch: '=', spacing: Alone, span: #0 bytes(334..335) })

I am trying to create custom Encode/Decode implementations for types in other crates that I'd like to use. However, I cant access the configuration as done on the impls.rs file of the encoder. How can I access the config (endianess, fixed/varint, etc) to write the data from the external type according to the config of the encoder/decoder?

We just realized after blissfully having used deserialize_from() for quite a while that it is potentially very dangerous when not used in conjunction with with_limit(). And unless you carefully read through the Bincode readme, you won't be aware of the existence of this issue.

I've spent some time looking into why this is as unsafe as it is and realized that it's due to the fact that the std::io::Read trait provides zero information about the remaining number of bytes. So all Bincode can do is blindly allocate however many bytes the u64 value it just decoded tells it to. Enforcing a default limit is similarly problematic, as it may both break some clients that are not aware of it and still result in unsafely large allocations for other clients.

But then it dawned on me that one could modify the Bincode API so that it requires an explicit limit whenever using deserialize_from(). E.g. by replacing deserialize_from(reader) with deserialize_from(reader, limit). This would immediately make it obvious to Bincode users that they need to make a trade-off between convenience and safety.

Those that only ever deal with trusted inputs can always go with usize::MAX. And those that (like us) are actually dealing with a reader on top of a slice, can simply get rid of the reader (and the need to pick an explicit limit) and use deserialize() instead.

Just an idea, but it feels to me like a net improvement over the status quo.

With the transition to the bincode specific encode and decode traits in the v2 release candidate, it appears there is currently no mechanism available for stateful deserialization comparable to Serde's DeserializeSeed trait.

See: https://github.com/serde-rs/serde/blob/f52d134c14f6904010d0a59107987b050b36b811/serde/src/de/mod.rs#L770

The following trait definitions could be included to provide this functionality.

pub trait DecodeSeed: Sized {
    type Value;

    fn decode_seed<D: Decoder>(self, decoder: &mut D) -> Result<Self::Value, DecodeError>;
}

pub trait BorrowDecodeSeed<'de>: Sized {
    type Value;

    fn borrow_decode_seed<D: BorrowDecoder<'de>>(self, decoder: &mut D) -> Result<Self::Value, DecodeError>;
}

For example, utilizing the DecodeSeed trait defined above, the SequenceSeed struct could be defined below:

struct SequenceSeed<'a>(&'a mut Vec<f64>);

impl<'a> DecodeSeed for SequenceSeed<'a> {
    type Value = u64;

    #[inline]
    fn decode_seed<D: bincode::de::Decoder>(self, decoder: &mut D) -> Result<Self::Value, bincode::error::DecodeError> {
        let size: u64 = bincode::Decode::decode(decoder)?;

        for _ in 0..size {
            let element = bincode::Decode::decode(decoder)?;
            self.0.push(element)
        }

        Ok(size)
    }
}

Utilizing SequenceSeed, we can append each decoded element to a pre-allocated vector.

// Encode
let values = vec![34.5, 345.3, 40.3, 100.55, 60.5];
let buffer = bincode::encode_to_vec(&values, config::standard()).expect("Failed to encode");

// Construct decoder
let slice_reader = bincode::de::read::SliceReader::new(&buffer[..]);
let mut decoder = bincode::de::DecoderImpl::new(slice_reader, config::standard());

// Construct SequenceSeed utilizing pre-allocated vector
let mut seq = Vec::with_capacity(5);
let seed = SequenceSeed(&mut seq);

// Decode using SequenceSeed's DecodeSeed implementation
let len = seed.decode_seed(&mut decoder).expect("Failed to decode");

println!("{len}");  // 5
println!("{seq:?}") // [34.5, 345.3, 40.3, 100.55, 60.5]

I feel this functionality provides many opportunities for optimizations and would be a useful addition.

error[E0425]: cannot find value `field_0` in this scope
  --> bug/src/lib.rs:4:29
   |
1  | / macro_rules! dv {
2  | |     ($($t:item),+) => {
3  | |         $(
4  | |             #[derive(Clone, bincode::Encode, bincode::Decode)]
   | |                             ^^^^^^^^^^^^^^^
   | |                             |
   | |                             not found in this scope
   | |                             in this derive macro expansion (#2)
...  |
7  | |     }
8  | | }
   | |_- in this expansion of `dv!` (#1)
...
11 | / dv! {
12 | |     pub struct Foo {},
13 | |     pub enum Bar {
14 | |         A(Foo),
15 | |         B,
16 | |     }
17 | | }
   | |_- in this macro invocation (#1)
   |
  ::: /home/lxb/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode_derive-2.0.0-rc.1/src/lib.rs:9:1
   |
9  |   pub fn derive_encode(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
   |   ------------------------------------------------------------------------------- in this expansion of `#[derive(bincode::Encode)]` (#2)

macro_rules! dv {
    ($($t:item),+) => {
        $(
            #[derive(Clone, bincode::Encode, bincode::Decode)]
            $t
        )+
    }
}

// Doesn't compile
dv! {
    pub struct Foo {},
    pub enum Bar {
        A(Foo),
        B,
    }
}

// Compiles
#[derive(Clone, bincode::Encode, bincode::Decode)]
pub struct Foo2 {}
#[derive(Clone, bincode::Encode, bincode::Decode)]
pub enum Bar2 {
    A(Foo2),
    B,
}

[package]
name = "bug"
version = "0.1.0"
edition = "2021"

[dependencies]
bincode = { version = "2.0.0-rc.1", default-features = false, features = ["derive"] }

This doesn't seem to happen with Decode

Currently, the readme states:

Bincode 2.0 is still in development and does not yet have a targeted MSRV. Once 2.0 is fully released the MSRV will be locked. After this point any changes to the MSRV are considered a breaking change for semver purposes.

I would suggest to specify the MSRV in Cargo.toml by using rust-version field as soon as a MSRV is selected so tools like cargo and clippy work better with the code.