I looked at the 2014 blog post, which says:

The Builder types of capnproto-rust also need to provide an exclusivity guarantee. Recall that if Foo is a struct defined in a Cap'n Proto schema, then a foo::Builder<'a> provides access to a writable location in arena-allocated memory that contains a Foo in Cap'n Proto format. To protect access to that memory, a foo::Builder<'a> ought to behave as if it were a &'a mut Foo, even though the Foo type cannot directly exist in Rust (because Cap'n Proto struct layout differs from Rust struct layout).

The solution it comes up with is to use reborrow() everywhere. But that's not necessary, since it can just take &mut self in the accessors instead of taking an owned value; that both enforces exclusive access and allows reusing the builder later without needing a reborrow() call.

As mentioned in https://github.com/capnproto/capnproto-rust/issues/191 and https://github.com/capnproto/capnproto-rust/issues/121, capnp::message::Reader<capnp::serialize::OwnedSegments> isn't Sync because of the Cell<u64> in the ReadLimiter.

This solution will only work with platforms that have AtomicU64 and only works from 1.34 onward. This modification could be put behind a feature if this affects any user of the crate.

Let me know !

Currently capnpc generates incompatible code with rust 2018.

Seems to be an issue with using :: to refer to the crate root. This has been changed to crate:: in 2018.

2015 Edition:

#[inline]
pub fn get_currency(self) -> ::std::result::Result<::account_capnp::Currency,::capnp::NotInSchema> {
  ::capnp::traits::FromU16::from_u16(self.reader.get_data_field::<u16>(16))
}

2018 Edition:

#[inline]
pub fn get_currency(self) -> ::std::result::Result<crate::account_capnp::Currency,::capnp::NotInSchema> {
  ::capnp::traits::FromU16::from_u16(self.reader.get_data_field::<u16>(16))
}

Hey there! I've been wanting to have capnproto-rust available on no_std for awhile (per #71), and I've given it a few shots in the past but I think now I'm actually pretty close to nailing it. I wanted to open up a PR to get some feedback on my approach and ask some questions about things I've gotten hung up on.

Things I've done:

• Added a no_std feature flag to the capnp crate. This feature is used in a few #[cfg(...)] conditional compilations around the codebase.
• Included the core_io crate as an optional dependency to be included when the no_std feature flag is set. This crate is mostly autogenerated in that it applies patches against std::io to take out all std usages and replace them using the alloc crate.
• Replaced all references to std:: within the capnp crate to use core:: instead. In capnp's lib.rs file, there is a conditional import based on #[cfg(feature = "no_std")]. If no_std is enabled, then the name core:: references core_io:: items. If no_std is not enabled, then core:: references std:: items. A similar strategy handles deciding between e.g. std::string and alloc::string and between std::str and alloc::str.

Problems I'm having now:

It seems that everything in the capnp crate is now building properly both when no_std is enabled and disabled. However, when I add the capnpc::CompilerCommand::new().file("schema.capnp").run().unwrap(); buildscript to a no_std project of mine, there's an interesting compilation problem. capnp needs to be compiled as no_std in order to comply with my no_std application, but since that is the case, the std-compiled capnpc seems to now be linking against the no_std-compiled capnp, whereas I think it should build a separate instance of capnp (std-compiled) to link against. The problem that this creates is that e.g. in capnpc/src/lib.rs:81: let mut p = command.spawn()?;, the spawn() call returns a Result in which the error type is std::io::Error. Typically, when capnp is std-compiled, it would implement From<std::io::Error> for capnp::Error. However, since it is linking against the no_std-compiled version of capnp, the implementation that is actually implemented is From<core_io::Error> for capnp::Error. It seems that I can get around this by simply mapping the error each time capnpc deals with an io::Result, e.g. like this:

let mut p = command.spawn().map_err(|err| capnp::Error::failed(err.to_string()))?;

There are only perhaps five or so instances of this problem, but I figured I should ask if anybody has any suggestions on a strategy to take. I don't think it makes sense to force the no_std requirements to leak up into capnpc, but I also don't necessarily want to destroy ergonomics by forcing the client to use map_err everywhere. I think I'll probably work on adding a commit that does do that, just in order to get a working solution together, but any alternative suggestions are certainly welcome!

Edit:

I forgot to mention, since the core_io crate is built by applying patches against certain builds of std::io, you'll need a fixed version of nightly rust to build it using the no_std feature. You should be able to satisfy it using this override which corresponds to the last patch of the core_io crate:

rustup override set nightly-2019-07-01

Hi! In http://youtu.be/A65w-qoyTYg?t=14m57s you say that reading from a Cap'n Proto message is comparable to a struct field access, or at least I've got that impression from your talk. Now, testing it with a simple benchmark

#[bench] fn capnp_unpack (bencher: &mut Bencher) {
  bencher.bytes = 3;

  let encoded: Vec<u8> = {
    let mut builder = MallocMessageBuilder::new_default();
    { let mut cache_value = builder.init_root::<cache_value::Builder>();
      cache_value.set_value (b"foo"); }
    let mut buf = Vec::with_capacity (128);
    capnp::serialize::write_message (&mut buf, &builder) .unwrap();
    buf};

  bencher.iter (|&:| {
    let mut bytes = encoded.as_slice();
    let reader = capnp::serialize::new_reader (&mut bytes, capnp::ReaderOptions::new()) .unwrap();
    let value = reader.get_root::<cache_value::Reader>();
    assert! (value.get_value() == b"foo");
  });}

which uses the following schema

struct CacheValue {
  expires @0 :UInt64;
  value @1 :Data;
  tags @2 :List(Text);
}

shows that reading from a Cap'n Proto message is much more expensive than a struct access. I get 3931 ns/iter from it. (In fact, the LMDB database where I keep this value is faster at returning it from the database (616 ns/iter!) than Cap'n Proto is at decoding it).

Theoretically I'd imagine that the decoding should involve just some memory accesses, but examining the new_reader code I see a couple of Vec allocations there.

So, Am I doing something wrong? Is it a state of the art Cap'n Proto performance or will it be improved?

This is a sketch of a possible solution to #38. The strategy taken here is to return a enum type of Complete or Continue whenever reading or writing a message could block. Complete signals that the operation completed successfully, while Continue signals that the operation must be tried again, and carries along with it any necessary continuation state.

The initial commit only includes a read_message implementation with continuations. The next step will be to implement a read_message_continue which takes the continuation and Read object and continues reading the message. I believe write_message can be implemented in much the same way, and perhaps with less impact to the code. Finally, I think packed serialization could be extended in much the same way.

The motivation to use continuations instead of a higher level abstraction like Futures or Promises is that Rust doesn't have a standard implementation of these abstractions, so adapting the read/write machinery to one specific implementation would not benefit the others. My goal is that this continuation API is a stepping stone to implementing async message reading and writing in any higher level async abstraction.

I'd love to get feedback on this strategy. I will continue filling it out in the meantime.

A recent commit changed write_message in serialize.rs from

pub fn write_message<W, A>(write: &mut W, message: &message::Builder<A>) -> ::std::io::Result<()>

to

pub fn write_message<W, A>(mut write: W, message: &message::Builder<A>) -> Result<()>

I am wondering why this changed.

It means you can no longer call write_message in a loop with a reference to a Write. E.g.

pub fn transcode<I,O>(input: I, output: &mut O, packed: bool)
    -> Result<(),Box<dyn Error>>
    where I: Read, O: Write
{
    let write_message = if packed { serialize_packed::write_message } else { serialize::write_message };

    let input = BufReader::new(input);
    for line in input.lines() {
        let line = line?;
        let e: Event = serde_json::from_str(&line)?;
        let bin = e.to_capnp();
        write_message(output, &bin)?
    }
    Ok(())
}

Now gives an error:

17 | pub fn transcode<I,O>(input: I, output: &mut O, packed: bool)
   |                                             ------ move occurs because `output` has type `&mut O`, which does not implement the `Copy` trait
...
28 |         write_message(output, &bin)?
   |                       ^^^^^^ value moved here, in previous iteration of loop

I'm experimenting with implementing Sandstorm's WebSite interface from web-publishing.capnp, and am unsure how to deal with the way uploading large blobs works. See:

https://github.com/sandstorm-io/sandstorm/blob/d366557c18faa085410334908a240ec88f97910b/src/sandstorm/web-publishing.capnp#L45-L46

and:

https://github.com/sandstorm-io/sandstorm/blob/d366557c18faa085410334908a240ec88f97910b/src/sandstorm/web-publishing.capnp#L81-L88

The interface suggests a flow where:

1. uploadBlob hands back a bytestream and blob to the caller
2. The caller writes the data to the bytestream
3. The caller passes the blob back to the server
4. The server then somehow identifies that this is one of the blobs it had previously handed out.

The last point suggests needing to be able to 'unwrap' a client to get at an underlying server (if the server is inside the local vat). I don't know how to do this this with the rust bindings, and am not sure whether it is in fact possible? Afaik the Go implementation doesn't do this either, and I just added this to the Haskell implementation a couple days ago. I'm not actually 100% sure whether the C++ implementation can do this, or if so, how (@kentonv?)

Can the rust implementation do this? If not, how difficult do you think it would be to add?

I came across this protocol and it looks absolutely perfect for microcontrollers! I understand some features support dynamically sized types, so cannot be supported on the stack, but was wondering if a subset of this library could support no_std?

If it is possible, but would have to be a separate crate then that would also be useful to know.

Thanks!

It's prohibitively difficult to use capnproto-rust with non-blocking input/output streams (Read/Write instances that can return WouldBlock. I've been looking at the codebase the last few days to see what it would take to add support for resumable serialization/deserialization. I believe by adding a WouldBlock variant to the capnproto Error type which holds intermediate state it would be possible. So for instance:

enum WouldBlockState {
    ReadPacked(..),
    Read(..),
    WritePacked(..),
    Write(..),
}

enum Error {
    Decode { .. },
    WouldBlock(WouldBlockState),
    Io(io::Error),
}

with additional read/write methods to resume reading a message after a WouldBlock error:

pub fn resume_read_message(WouldBlockState, &mut InputStream, ReaderOptions) -> Result<..>;
pub fn resume_read_message_packed(WouldBlockState, &mut BufferedInputStream, ReaderOptions) -> Result<..>;
pub fn resume_write_message(WouldBlockState, &mut OutputStream, ReaderOptions) -> Result<..>;
pub fn resume_write_message_packed(WouldBlockState, &mut BufferedOutputStream, ReaderOptions) -> Result<..>;

I want to get your thoughts on this before diving too far into the implementation.

I think this could be great for python (and other) FFI so I have been trying to figure out how to load &[u8] into an actual object. This is what I have so far but I think it's probably misguided.

#[no_mangle]
pub extern fn test_capnproto(thing: *const u8, length: size_t) {
    let array: &[u8] = unsafe{ slice::from_raw_parts(thing, length as usize) };
    let mut message = MallocMessageBuilder::new_default();
    let mut test_date = message.init_root::<date::Builder>();
    { // scope for builder 
        let b = test_date.borrow();
        for (i, elem) in array.iter().enumerate() {
            b.set_data_field(i, *elem);
        }
    }

    println!("{:?}", test_date);
}

Any tips? Perhaps something to do with read_message? I couldn't figure it out from the examples

the 'getters' in this library (both internal and generated) are non-idiomatic. In Rust, it is considered non-idiomatic to prefix getters with get_.

see RFC #344

obviously fixing this would be a BREAKING CHANGE

This is more or less a straight port of the reconnect API from capnp-c++.

I say more or less because I also needed the setTarget functionality from CapRedirector in Sandstorm and so have included that as part of the API.

To keep the API consistent with the existing new_client and new_promise_client functions the auto_reconnect and lazy_auto_reconnect functions work with generic clients that implement FromClientHook.