Cap'n Proto for Rust

Overview

Cap'n Proto for Rust

Build Status

documentation

blog

Introduction

Cap'n Proto is a type system for distributed systems.

With Cap'n Proto, you describe your data and interfaces in a schema file, like this:

@0x986b3393db1396c9;

struct Point {
    x @0 :Float32;
    y @1 :Float32;
}

interface PointTracker {
    addPoint @0 (p :Point) -> (totalPoints :UInt64);
}

You can then use the capnp tool to generate code in a variety of programming languages. The generated code lets you produce and consume values of the types you've defined in your schema.

Values are encoded in a format that is suitable not only for transmission over a network and persistence to disk, but also for zero-copy in-memory traversal. That is, you can completely skip serialization and deserialization! It's in this sense that Cap'n Proto is "infinity times faster" than alternatives like Protocol Buffers.

In Rust, the generated code for the example above includes a point::Reader<'a> struct with get_x() and get_y() methods, and a point::Builder<'a> struct with set_x() and set_y() methods. The lifetime parameter 'a is a formal reminder that point::Reader<'a> and point::Builder<'a> contain borrowed references to the raw buffers that contain the encoded messages. Those underlying buffers are never actually copied into separate data structures.

The generated code for the example above also includes a point_tracker::Server trait with an add_point() method, and a point_tracker::Client struct with an add_point_request() method. The former can be implemented to create a network-accessible object, and the latter can be used to invoke a possibly-remote instance of a PointTracker.

Features

Crates

capnp Runtime library for dealing with Cap'n Proto messages. crates.io
capnpc Rust code generator plugin, including support for hooking into a build.rs file in a cargo build. crates.io
capnp-futures Support for asynchronous reading and writing of Cap'n Proto messages. crates.io
capnp-rpc Object-capability remote procedure call system. crates.io

Examples

addressbook serialization, RPC

Who is using capnproto-rust?

Unimplemented / Future Work

Comments
  • Remove need for `reborrow()`

    Remove need for `reborrow()`

    I looked at the 2014 blog post, which says:

    The Builder types of capnproto-rust also need to provide an exclusivity guarantee. Recall that if Foo is a struct defined in a Cap'n Proto schema, then a foo::Builder<'a> provides access to a writable location in arena-allocated memory that contains a Foo in Cap'n Proto format. To protect access to that memory, a foo::Builder<'a> ought to behave as if it were a &'a mut Foo, even though the Foo type cannot directly exist in Rust (because Cap'n Proto struct layout differs from Rust struct layout).

    The solution it comes up with is to use reborrow() everywhere. But that's not necessary, since it can just take &mut self in the accessors instead of taking an owned value; that both enforces exclusive access and allows reusing the builder later without needing a reborrow() call.

    opened by jyn514 22
  • Sync ReadLimiter

    Sync ReadLimiter

    As mentioned in https://github.com/capnproto/capnproto-rust/issues/191 and https://github.com/capnproto/capnproto-rust/issues/121, capnp::message::Reader<capnp::serialize::OwnedSegments> isn't Sync because of the Cell<u64> in the ReadLimiter.

    This solution will only work with platforms that have AtomicU64 and only works from 1.34 onward. This modification could be put behind a feature if this affects any user of the crate.

    Let me know !

    opened by appaquet 22
  • Support for 2018 edition

    Support for 2018 edition

    Currently capnpc generates incompatible code with rust 2018.

    Seems to be an issue with using :: to refer to the crate root. This has been changed to crate:: in 2018.

    2015 Edition:

    #[inline]
    pub fn get_currency(self) -> ::std::result::Result<::account_capnp::Currency,::capnp::NotInSchema> {
      ::capnp::traits::FromU16::from_u16(self.reader.get_data_field::<u16>(16))
    }
    

    2018 Edition:

    #[inline]
    pub fn get_currency(self) -> ::std::result::Result<crate::account_capnp::Currency,::capnp::NotInSchema> {
      ::capnp::traits::FromU16::from_u16(self.reader.get_data_field::<u16>(16))
    }
    
    opened by FallingSnow 21
  • Add no_std feature

    Add no_std feature

    Hey there! I've been wanting to have capnproto-rust available on no_std for awhile (per #71), and I've given it a few shots in the past but I think now I'm actually pretty close to nailing it. I wanted to open up a PR to get some feedback on my approach and ask some questions about things I've gotten hung up on.

    Things I've done:

    • Added a no_std feature flag to the capnp crate. This feature is used in a few #[cfg(...)] conditional compilations around the codebase.
    • Included the core_io crate as an optional dependency to be included when the no_std feature flag is set. This crate is mostly autogenerated in that it applies patches against std::io to take out all std usages and replace them using the alloc crate.
    • Replaced all references to std:: within the capnp crate to use core:: instead. In capnp's lib.rs file, there is a conditional import based on #[cfg(feature = "no_std")]. If no_std is enabled, then the name core:: references core_io:: items. If no_std is not enabled, then core:: references std:: items. A similar strategy handles deciding between e.g. std::string and alloc::string and between std::str and alloc::str.

    Problems I'm having now:

    It seems that everything in the capnp crate is now building properly both when no_std is enabled and disabled. However, when I add the capnpc::CompilerCommand::new().file("schema.capnp").run().unwrap(); buildscript to a no_std project of mine, there's an interesting compilation problem. capnp needs to be compiled as no_std in order to comply with my no_std application, but since that is the case, the std-compiled capnpc seems to now be linking against the no_std-compiled capnp, whereas I think it should build a separate instance of capnp (std-compiled) to link against. The problem that this creates is that e.g. in capnpc/src/lib.rs:81: let mut p = command.spawn()?;, the spawn() call returns a Result in which the error type is std::io::Error. Typically, when capnp is std-compiled, it would implement From<std::io::Error> for capnp::Error. However, since it is linking against the no_std-compiled version of capnp, the implementation that is actually implemented is From<core_io::Error> for capnp::Error. It seems that I can get around this by simply mapping the error each time capnpc deals with an io::Result, e.g. like this:

    let mut p = command.spawn().map_err(|err| capnp::Error::failed(err.to_string()))?;
    

    There are only perhaps five or so instances of this problem, but I figured I should ask if anybody has any suggestions on a strategy to take. I don't think it makes sense to force the no_std requirements to leak up into capnpc, but I also don't necessarily want to destroy ergonomics by forcing the client to use map_err everywhere. I think I'll probably work on adding a commit that does do that, just in order to get a working solution together, but any alternative suggestions are certainly welcome!

    Edit:

    I forgot to mention, since the core_io crate is built by applying patches against certain builds of std::io, you'll need a fixed version of nightly rust to build it using the no_std feature. You should be able to satisfy it using this override which corresponds to the last patch of the core_io crate:

    rustup override set nightly-2019-07-01
    
    opened by nicholastmosher 20
  • implement FlatArrayMessageReader

    implement FlatArrayMessageReader

    Hi! In http://youtu.be/A65w-qoyTYg?t=14m57s you say that reading from a Cap'n Proto message is comparable to a struct field access, or at least I've got that impression from your talk. Now, testing it with a simple benchmark

    #[bench] fn capnp_unpack (bencher: &mut Bencher) {
      bencher.bytes = 3;
    
      let encoded: Vec<u8> = {
        let mut builder = MallocMessageBuilder::new_default();
        { let mut cache_value = builder.init_root::<cache_value::Builder>();
          cache_value.set_value (b"foo"); }
        let mut buf = Vec::with_capacity (128);
        capnp::serialize::write_message (&mut buf, &builder) .unwrap();
        buf};
    
      bencher.iter (|&:| {
        let mut bytes = encoded.as_slice();
        let reader = capnp::serialize::new_reader (&mut bytes, capnp::ReaderOptions::new()) .unwrap();
        let value = reader.get_root::<cache_value::Reader>();
        assert! (value.get_value() == b"foo");
      });}
    

    which uses the following schema

    struct CacheValue {
      expires @0 :UInt64;
      value @1 :Data;
      tags @2 :List(Text);
    }
    

    shows that reading from a Cap'n Proto message is much more expensive than a struct access. I get 3931 ns/iter from it. (In fact, the LMDB database where I keep this value is faster at returning it from the database (616 ns/iter!) than Cap'n Proto is at decoding it).

    Theoretically I'd imagine that the decoding should involve just some memory accesses, but examining the new_reader code I see a couple of Vec allocations there.

    So, Am I doing something wrong? Is it a state of the art Cap'n Proto performance or will it be improved?

    opened by ArtemGr 18
  • async message reader with explicit continuations

    async message reader with explicit continuations

    This is a sketch of a possible solution to #38. The strategy taken here is to return a enum type of Complete or Continue whenever reading or writing a message could block. Complete signals that the operation completed successfully, while Continue signals that the operation must be tried again, and carries along with it any necessary continuation state.

    The initial commit only includes a read_message implementation with continuations. The next step will be to implement a read_message_continue which takes the continuation and Read object and continues reading the message. I believe write_message can be implemented in much the same way, and perhaps with less impact to the code. Finally, I think packed serialization could be extended in much the same way.

    The motivation to use continuations instead of a higher level abstraction like Futures or Promises is that Rust doesn't have a standard implementation of these abstractions, so adapting the read/write machinery to one specific implementation would not benefit the others. My goal is that this continuation API is a stepping stone to implementing async message reading and writing in any higher level async abstraction.

    I'd love to get feedback on this strategy. I will continue filling it out in the meantime.

    opened by danburkert 17
  • `write_message` now takes ownership of the `Write` argument

    `write_message` now takes ownership of the `Write` argument

    A recent commit changed write_message in serialize.rs from

    pub fn write_message<W, A>(write: &mut W, message: &message::Builder<A>) -> ::std::io::Result<()>
    

    to

    pub fn write_message<W, A>(mut write: W, message: &message::Builder<A>) -> Result<()>
    

    I am wondering why this changed.

    It means you can no longer call write_message in a loop with a reference to a Write. E.g.

    pub fn transcode<I,O>(input: I, output: &mut O, packed: bool)
        -> Result<(),Box<dyn Error>>
        where I: Read, O: Write
    {
        let write_message = if packed { serialize_packed::write_message } else { serialize::write_message };
    
        let input = BufReader::new(input);
        for line in input.lines() {
            let line = line?;
            let e: Event = serde_json::from_str(&line)?;
            let bin = e.to_capnp();
            write_message(output, &bin)?
        }
        Ok(())
    }
    

    Now gives an error:

    17 | pub fn transcode<I,O>(input: I, output: &mut O, packed: bool)
       |                                             ------ move occurs because `output` has type `&mut O`, which does not implement the `Copy` trait
    ...
    28 |         write_message(output, &bin)?
       |                       ^^^^^^ value moved here, in previous iteration of loop
    
    opened by eliaslevy 16
  • Support for

    Support for "unwrapping" clients to get at the underlying server.

    I'm experimenting with implementing Sandstorm's WebSite interface from web-publishing.capnp, and am unsure how to deal with the way uploading large blobs works. See:

    https://github.com/sandstorm-io/sandstorm/blob/d366557c18faa085410334908a240ec88f97910b/src/sandstorm/web-publishing.capnp#L45-L46

    and:

    https://github.com/sandstorm-io/sandstorm/blob/d366557c18faa085410334908a240ec88f97910b/src/sandstorm/web-publishing.capnp#L81-L88

    The interface suggests a flow where:

    1. uploadBlob hands back a bytestream and blob to the caller
    2. The caller writes the data to the bytestream
    3. The caller passes the blob back to the server
    4. The server then somehow identifies that this is one of the blobs it had previously handed out.

    The last point suggests needing to be able to 'unwrap' a client to get at an underlying server (if the server is inside the local vat). I don't know how to do this this with the rust bindings, and am not sure whether it is in fact possible? Afaik the Go implementation doesn't do this either, and I just added this to the Haskell implementation a couple days ago. I'm not actually 100% sure whether the C++ implementation can do this, or if so, how (@kentonv?)

    Can the rust implementation do this? If not, how difficult do you think it would be to add?

    opened by zenhack 15
  • support for no_std environments

    support for no_std environments

    I came across this protocol and it looks absolutely perfect for microcontrollers! I understand some features support dynamically sized types, so cannot be supported on the stack, but was wondering if a subset of this library could support no_std?

    If it is possible, but would have to be a separate crate then that would also be useful to know.

    Thanks!

    opened by vitiral 15
  • Using capnproto with non-blocking input/output streams

    Using capnproto with non-blocking input/output streams

    It's prohibitively difficult to use capnproto-rust with non-blocking input/output streams (Read/Write instances that can return WouldBlock. I've been looking at the codebase the last few days to see what it would take to add support for resumable serialization/deserialization. I believe by adding a WouldBlock variant to the capnproto Error type which holds intermediate state it would be possible. So for instance:

    enum WouldBlockState {
        ReadPacked(..),
        Read(..),
        WritePacked(..),
        Write(..),
    }
    
    enum Error {
        Decode { .. },
        WouldBlock(WouldBlockState),
        Io(io::Error),
    }
    

    with additional read/write methods to resume reading a message after a WouldBlock error:

    
    pub fn resume_read_message(WouldBlockState, &mut InputStream, ReaderOptions) -> Result<..>;
    pub fn resume_read_message_packed(WouldBlockState, &mut BufferedInputStream, ReaderOptions) -> Result<..>;
    pub fn resume_write_message(WouldBlockState, &mut OutputStream, ReaderOptions) -> Result<..>;
    pub fn resume_write_message_packed(WouldBlockState, &mut BufferedOutputStream, ReaderOptions) -> Result<..>;
    

    I want to get your thoughts on this before diving too far into the implementation.

    opened by danburkert 14
  • simple function for loading messages?

    simple function for loading messages?

    I think this could be great for python (and other) FFI so I have been trying to figure out how to load &[u8] into an actual object. This is what I have so far but I think it's probably misguided.

    
    #[no_mangle]
    pub extern fn test_capnproto(thing: *const u8, length: size_t) {
        let array: &[u8] = unsafe{ slice::from_raw_parts(thing, length as usize) };
        let mut message = MallocMessageBuilder::new_default();
        let mut test_date = message.init_root::<date::Builder>();
        { // scope for builder 
            let b = test_date.borrow();
            for (i, elem) in array.iter().enumerate() {
                b.set_data_field(i, *elem);
            }
        }
    
        println!("{:?}", test_date);
    }
    

    Any tips? Perhaps something to do with read_message? I couldn't figure it out from the examples

    opened by waynenilsen 13
  • Error { kind: Failed, description:

    Error { kind: Failed, description: "Received a new call on in-use question id 0" }

    Hello.

    I'm seeing an error happen sporadically when I run my tests on CI, and I've been trying to reproduce it locally with little luck.

    Basically I have two components: F (capnproto-rust) and E (pycapnp). F calls E and passes it, among other things, a capability for it to call F. The test where this error happens is one where there the F->E call timeouts and it needs to retry. RPC calls start throwing the error "Error { kind: Failed, description: "Received a new call on in-use question id 0" }", and shortly after the RpcSystem promise also returns with the same error.

    I know I don't have a lot of information to give, but I was wondering if someone could explain to me in which circumstances I can obtain the above error so that I can more easily reproduce it. It seems like a bug in the rpc library that should be fixed. From looking at the code I get the impression that what causes the error is the call E->F, even though I only notice the issue afterwards when I try to do more rpc calls from F.

    opened by edevil 4
  • non-idiomatic getters

    non-idiomatic getters

    the 'getters' in this library (both internal and generated) are non-idiomatic. In Rust, it is considered non-idiomatic to prefix getters with get_.

    see RFC #344

    obviously fixing this would be a BREAKING CHANGE

    opened by danieleades 1
  • Ported reconnect API

    Ported reconnect API

    This is more or less a straight port of the reconnect API from capnp-c++.

    I say more or less because I also needed the setTarget functionality from CapRedirector in Sandstorm and so have included that as part of the API.

    To keep the API consistent with the existing new_client and new_promise_client functions the auto_reconnect and lazy_auto_reconnect functions work with generic clients that implement FromClientHook.

    opened by griff 0
Owner
Cap'n Proto
Cap'n Proto
MessagePack implementation for Rust / msgpack.org[Rust]

RMP - Rust MessagePack RMP is a pure Rust MessagePack implementation. This repository consists of three separate crates: the RMP core and two implemen

Evgeny Safronov 840 Dec 30, 2022
A Rust ASN.1 (DER) serializer.

rust-asn1 This is a Rust library for parsing and generating ASN.1 data (DER only). Installation Add asn1 to the [dependencies] section of your Cargo.t

Alex Gaynor 85 Dec 16, 2022
Implementation of Bencode encoding written in rust

Rust Bencode Implementation of Bencode encoding written in rust. Project Status Not in active developement due to lack of time and other priorities. I

Arjan Topolovec 32 Aug 6, 2022
Encoding and decoding support for BSON in Rust

bson-rs Encoding and decoding support for BSON in Rust Index Overview of BSON Format Usage BSON Values BSON Documents Modeling BSON with strongly type

mongodb 304 Dec 30, 2022
Rust library for reading/writing numbers in big-endian and little-endian.

byteorder This crate provides convenience methods for encoding and decoding numbers in either big-endian or little-endian order. Dual-licensed under M

Andrew Gallant 811 Jan 1, 2023
A Gecko-oriented implementation of the Encoding Standard in Rust

encoding_rs encoding_rs an implementation of the (non-JavaScript parts of) the Encoding Standard written in Rust and used in Gecko (starting with Fire

Henri Sivonen 284 Dec 13, 2022
Character encoding support for Rust

Encoding 0.3.0-dev Character encoding support for Rust. (also known as rust-encoding) It is based on WHATWG Encoding Standard, and also provides an ad

Kang Seonghoon 264 Dec 14, 2022
Rust implementation of CRC(16, 32, 64) with support of various standards

crc Rust implementation of CRC(16, 32, 64). MSRV is 1.46. Usage Add crc to Cargo.toml [dependencies] crc = "2.0" Compute CRC use crc::{Crc, Algorithm,

Rui Hu 120 Dec 23, 2022
A CSV parser for Rust, with Serde support.

csv A fast and flexible CSV reader and writer for Rust, with support for Serde. Dual-licensed under MIT or the UNLICENSE. Documentation https://docs.r

Andrew Gallant 1.3k Jan 5, 2023
A HTTP Archive format (HAR) serialization & deserialization library, written in Rust.

har-rs HTTP Archive format (HAR) serialization & deserialization library, written in Rust. Install Add the following to your Cargo.toml file: [depende

Sebastian Mandrean 25 Dec 24, 2022
A HTML entity encoding library for Rust

A HTML entity encoding library for Rust Example usage All example assume a extern crate htmlescape; and use htmlescape::{relevant functions here}; is

Viktor Dahl 41 Nov 1, 2022
pem-rs pem PEM jcreekmore/pem-rs [pem] — A Rust based way to parse and encode PEM-encoded data

pem A Rust library for parsing and encoding PEM-encoded data. Documentation Module documentation with examples Usage Add this to your Cargo.toml: [dep

Jonathan Creekmore 30 Dec 27, 2022
PROST! a Protocol Buffers implementation for the Rust Language

PROST! prost is a Protocol Buffers implementation for the Rust Language. prost generates simple, idiomatic Rust code from proto2 and proto3 files. Com

Dan Burkert 17 Jan 8, 2023
Rust implementation of Google protocol buffers

rust-protobuf Protobuf implementation in Rust. Written in pure rust Generate rust code Has runtime library for generated code (Coded{Input|Output}Stre

Stepan Koltsov 2.3k Dec 31, 2022
tnetstring serialization library for rust.

TNetStrings: Tagged Netstrings This module implements bindings for the tnetstring serialization format. API let t = tnetstring::str("hello world"); le

Erick Tryzelaar 16 Jul 14, 2019
A TOML encoding/decoding library for Rust

toml-rs A TOML decoder and encoder for Rust. This library is currently compliant with the v0.5.0 version of TOML. This library will also likely contin

Alex Crichton 1k Dec 30, 2022
A fast, performant implementation of skip list in Rust.

Subway A fast, performant implementation of skip list in Rust. A skip list is probabilistic data structure that provides O(log N) search and insertion

Sushrut 16 Apr 5, 2022
A Rust PAC for the RP2040 Microcontroller

rp2040-pac - PAC for Raspberry Pi RP2040 microcontrollers This is a Peripheral Access Crate for the Raspberry Pi RP2040 dual-core Cortex-M0+ microcont

rp-rs 120 Nov 23, 2022
Pure Rust port of CRFsuite: a fast implementation of Conditional Random Fields (CRFs)

crfs-rs Pure Rust port of CRFsuite: a fast implementation of Conditional Random Fields (CRFs) Currently only support prediction, model training is not

messense 24 Nov 23, 2022