CBOR (binary JSON) for Rust with automatic type based decoding and encoding.

I've been using this code on the c2rust project pretty extensively for the last year after bringing it up in #12 It'd be neat to fold it back into the project!

I use the following code to unpack a packet from network:

  let mut dec = Decoder::from_reader(io::Cursor::new(buf));
  match dec.decode::<Packet>().next() {...}

And next() returns None when packet was truncated by some reason. Is it works as intended? I would like to have some more explicit error.

I use cbor 0.3.14 if that matters.

Is there a better interface? Should this behavior be documented?

Hi,

May I ask if you could decouple dependencies? For a basic, simple and concise implementation, I think it would be great if there are no dependencies.

For more features, maybe you can make a separate crate...

Thank you,

I'm interested in using this crate to decode CBOR structures that happen to use indefinite length encodings. I've locally patched the code to support this locally. I'd be willing to submit a pull-request soon.

I'm interested to know what you plans are for supporting indefinite lengths in general. For my use-case it is sufficient for me to add support to read_array, read_map, read_bytes, and read_string in decoder.rs. Are there some other considerations?

I can't tell if this is user error, or an issue with rust-cbor. I'm using CBorTagEncode to encode some custom types. When these custom types include a u64, I'm getting an unexpectedEOF during decode.

A fully self-contained reproducer is below. Using rustc 1.10.0-nightly (4ec5ce5e4 2016-05-12)

Thanks!

extern crate cbor;
extern crate rustc_serialize;

use cbor::{CborTagEncode};
use rustc_serialize::{Decodable, Decoder, Encodable, Encoder};

pub enum TestTypes {
    TestA(u32),
    TestB(u64)
}


fn encode<T: Encodable>(t: &T) -> Vec<u8> {
    let mut e = cbor::Encoder::from_memory();
    e.encode(vec!(t)).unwrap();
    e.into_bytes()
}
fn decode<T: Decodable>(bytes: Vec<u8>) -> T {
    let mut d = cbor::Decoder::from_bytes(bytes);

    let mut items = d.decode();
    let maybe_result = items.next();
    let result = maybe_result.unwrap();
    result.unwrap()
}

impl Encodable for TestTypes {
    fn encode<E: Encoder>(&self, e: &mut E) -> Result<(), E::Error> {
        match self {
            &TestTypes::TestA(ref val) => CborTagEncode::new(100_000_01, val).encode(e),
            &TestTypes::TestB(ref val) => CborTagEncode::new(100_000_02, val).encode(e),
        }
    }
}

impl Decodable for TestTypes {
    fn decode<D: Decoder>(d: &mut D) -> Result<TestTypes, D::Error> {
        let tag = try!(d.read_u64());
        match tag {
            100_000_01 => Ok(TestTypes::TestA(try!(Decodable::decode(d)))),
            100_000_02 => Ok(TestTypes::TestB(try!(Decodable::decode(d)))),
            _ => panic!("Unexpected tag")
        }
    }
}



fn main() {
    let a = TestTypes::TestA(1234u32);
    let b = TestTypes::TestB(1234u64);

    let a_encoded = encode(&a);
    println!("TestA(u32) encoded as {} bytes: {:?}", a_encoded.len(), a_encoded);
    let b_encoded = encode(&b);
    println!("TestB(u64) encoded as {} bytes: {:?}", b_encoded.len(), b_encoded);


    let aa: TestTypes = decode(a_encoded);
    let bb: TestTypes = decode(b_encoded);
}

achin@bigbox ~/tmp/06/cbor_test $ env RUST_BACKTRACE=1 cargo run
     Running `target/debug/cbor_test`
TestA(u32) encoded as 8 bytes: [218, 0, 152, 150, 129, 25, 4, 210]
TestB(u64) encoded as 8 bytes: [218, 0, 152, 150, 130, 217, 4, 210]

thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedEOF', ../src/libcore/result.rs:785
stack backtrace:
   1:     0x55b614483c60 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
   2:     0x55b6144869fb - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
   3:     0x55b614486683 - std::panicking::default_hook::hc2c969e7453d080c
   4:     0x55b61447de78 - std::panicking::rust_panic_with_hook::hfe203e3083c2b544
   5:     0x55b614486c41 - std::panicking::begin_panic::h4889569716505182
   6:     0x55b61447eaaa - std::panicking::begin_panic_fmt::h484cd47786497f03
   7:     0x55b614486bde - rust_begin_unwind
   8:     0x55b6144bc9ff - core::panicking::panic_fmt::h257ceb0aa351d801
   9:     0x55b614458eb8 - core::result::unwrap_failed::hf334f4fb1663f17e
                        at ../src/libcore/macros.rs:29
  10:     0x55b614475032 - _<std..result..Result<T, E>>::unwrap::h627180c2294af4f7
                        at ../src/libcore/result.rs:723
  11:     0x55b61445ee81 - cbor_test::decode::hf50c8635c0e15470
                        at src/main.rs:24
  12:     0x55b614458a35 - cbor_test::main::h6878392f0e5b8935
                        at src/main.rs:60
  13:     0x55b6144862c8 - std::panicking::try::call::hc5e1f5b484ec7f0e
  14:     0x55b61449079b - __rust_try
  15:     0x55b61449073e - __rust_maybe_catch_panic
  16:     0x55b614485cf3 - std::rt::lang_start::h61f4934e780b4dfc
  17:     0x55b614475109 - main
  18:     0x7f442ddfa61f - __libc_start_main
  19:     0x55b614458528 - _start
  20:                0x0 - <unknown>
error: Process didn't exit successfully: `target/debug/cbor_test` (exit code: 101)

Hi,

I'm using read_data_item directly and it would be cool if it were public.

pub(crate) fn decode(bytes: Vec<u8>) -> Result<Ipld, IpldError> {
    let mut d = Decoder::from_bytes(bytes);
    let cbor: Cbor = d.read_data_item(None)?;
    cbor_to_ipld(cbor)
}

This is exactly the same class of error that has hit bincode recently. https://github.com/TyOverby/bincode/issues/41

I have an issue with a solution open on rustc-serialize.

Fixes #8 and adds a test. Only set Encoder::tag while encoding the tag value itself, not when encoding the contained data. Also, the tag setting is now only set locally and so nested tags do not overwrite the state of Encoder::tag (not really a problem before).

Here, in lib.rc, the documentation states:

This type is only useful when your manually expecting the structure of a CBOR data item.

What are you trying to say here? "when you're manually inspecting the structure", or something else?

This addresses #13 by adding signed/unsigned integer map key support (not sure it makes sense to support absolutely every CBOR type as map keys). It's a first pass, so this is more of a feedback request to see if you think this is going in the right direction. I can also add more tests.

Unless I'm misunderstanding, rust-cbor doesn't support non-string keys in CBOR maps. From my reading of the rfc, types other than strings may be used as keys in maps.

For instance, the hex bytes a10126 should decode as a map with one entry: a key of an unsigned integer of value 1 mapping to a value of the negative integer -7. Right now the decoder throws a TypeMismatch error expecting a Unicode string.

I have a CBOR encoder/decoder for Lua which uses the canonicalization recommendations from the RFC, specifically that floating point values should be stored in the smallest representation that does not result in loss of information. This provides a decent saving on structures that have a lot of floating point values many of which are just small integers. Unfortunately this fails spectacularly for every value except 0.0 with the f16 decoding behaviour in rust-cbor.

An example correct decoding routine is available in Appendix D of the RFC. The C version focuses on breaking down the f16 and building a new f32 float with ldexp. The Python version uses bit manipulation to convert the f16 directly into an f32.

When using a fixed size buffer, the encoder should fail if it needs to write more data than will fit in the buffer. Currently, the encoder writes as much as it can and stops.

It seems like this can be attributed to using io::BufWriter internally. Here's a small reproduction of the core bug:

extern crate bytes;

use std::io::{self, Write};

fn main() {
    let mut buf = io::BufWriter::new(bytes::ByteBuf::mut_with_capacity(1));
    buf.write_all(&[1, 2, 3, 4, 5]).unwrap();
}

This succeeds presumably because the bytes were indeed successfully written to the buffer. However, if one calls buf.flush(), then an error is reported because the entire contents of the buffer could not be written.

If we look at an example from #6, this finishes with output len: 1, contents: [161]. Namely, no error is reported:

extern crate bytes;
extern crate cbor;
extern crate rustc_serialize;

use std::io::Read;

use cbor::Encoder;

#[derive(RustcDecodable, RustcEncodable, Debug)]
struct Test {
    value: String,
}

fn main() {
    let mut buf = bytes::ByteBuf::mut_with_capacity(1);
    {
        let mut enc = Encoder::from_writer(&mut buf);
        enc.encode(&[Test {
            value: "hello world hello world hello world".to_string()
        }]).unwrap();
    }
    let nbytes = buf.bytes().len();
    println!("len: {}, contents: {:?}", nbytes, buf.bytes());
}

This is because the data from the buffer is flushed when the encoder is dropped. This should produce an error, but errors are ignored in destructors.

If we change the code above to (same, but with a call to enc.flush()):

extern crate bytes;
extern crate cbor;
extern crate rustc_serialize;

use std::io::Read;

use cbor::Encoder;

#[derive(RustcDecodable, RustcEncodable, Debug)]
struct Test {
    value: String,
}

fn main() {
    let mut buf = bytes::ByteBuf::mut_with_capacity(1);
    {
        let mut enc = Encoder::from_writer(&mut buf);
        enc.encode(&[Test {
            value: "hello world hello world hello world".to_string()
        }]).unwrap();
        enc.flush().unwrap();
    }
    let nbytes = buf.bytes().len();
    println!("len: {}, contents: {:?}", nbytes, buf.bytes());
}

Then it will indeed fail because the underlying buffer flush will fail.

cc @tailhook

Hello!

This PR adds initial support for serde. One of the cool things about it is that it can directly deserialize into a structure instead of having to go through cbor::Decoder to parse out a Cbor type and then deserialize from that into a cbor::RustcDecoder. It currently passes the tests and benchmarks pretty well (although it's still slower than serde_json for some unknown reason):

     Running target/release/bench-d76ac9f7d864696a

running 16 tests
test decode_medium_cbor        ... bench:  15,151,437 ns/iter (+/- 1,104,258) = 26 MB/s
test decode_medium_direct_cbor ... bench:   4,477,213 ns/iter (+/- 511,434) = 89 MB/s
test decode_medium_json        ... bench:  20,373,353 ns/iter (+/- 2,205,575) = 27 MB/s
test decode_small_cbor         ... bench:       1,315 ns/iter (+/- 37) = 30 MB/s
test decode_small_direct_cbor  ... bench:       2,558 ns/iter (+/- 345) = 15 MB/s
test decode_small_json         ... bench:       1,880 ns/iter (+/- 367) = 29 MB/s
test encode_medium_cbor        ... bench:     945,316 ns/iter (+/- 16,675) = 422 MB/s
test encode_medium_json        ... bench:   7,628,975 ns/iter (+/- 339,718) = 74 MB/s
test encode_medium_tojson      ... bench:  14,429,221 ns/iter (+/- 1,244,292) = 39 MB/s
test encode_small_cbor         ... bench:       3,460 ns/iter (+/- 359) = 11 MB/s
test encode_small_json         ... bench:         841 ns/iter (+/- 318) = 66 MB/s
test encode_small_tojson       ... bench:       1,213 ns/iter (+/- 202) = 46 MB/s
test read_medium_cbor          ... bench:   9,462,908 ns/iter (+/- 1,028,207) = 42 MB/s
test read_medium_json          ... bench:  14,876,985 ns/iter (+/- 2,433,638) = 38 MB/s
test read_small_cbor           ... bench:         785 ns/iter (+/- 193) = 50 MB/s
test read_small_json           ... bench:       1,226 ns/iter (+/- 26) = 45 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 16 measured

     Running target/release/bench_serde-7de34a6bd2bc4f3a

running 8 tests
test deserialize_medium_cbor ... bench:   6,434,673 ns/iter (+/- 412,666) = 62 MB/s
test deserialize_medium_json ... bench:   4,808,781 ns/iter (+/- 1,297,244) = 117 MB/s
test deserialize_small_cbor  ... bench:       2,666 ns/iter (+/- 891) = 15 MB/s
test deserialize_small_json  ... bench:         439 ns/iter (+/- 16) = 127 MB/s
test serialize_medium_cbor   ... bench:   1,421,697 ns/iter (+/- 475,204) = 281 MB/s
test serialize_medium_json   ... bench:   4,730,402 ns/iter (+/- 1,666,983) = 120 MB/s
test serialize_small_cbor    ... bench:       3,479 ns/iter (+/- 491) = 11 MB/s
test serialize_small_json    ... bench:         441 ns/iter (+/- 122) = 126 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 8 measured

     Running target/release/cbor-8d79df739e5d6e87

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

Along the way I also changed how variants are serialized. Instead of {"variant": "Name", "fields": [...]}, it's now {"Name": [...]}. This is much more efficient for serde to parse because it doesn't require potentially buffering the field argument.

Be aware though that this does add a testing dependency to serde_macros, which is currently nightly-only. Check out how serde_tests in https://githb.com/serde-rs/serde if you're interested in a method that's compatible with stable rust.

I should start using serde for the automatic serialization component of this crate.

I don't think this means removing support for rustc-serialize just yet though.