Rust implementation of Google protocol buffers

Stepan Koltsov

Last update: Dec 31, 2022

Overview

rust-protobuf

Protobuf implementation in Rust.

Written in pure rust
Generate rust code
Has runtime library for generated code (Coded{Input|Output}Stream impl)
Supports both Protobuf versions 2 and 3

List of crates

rust-protobuf — repository provides multiple crates:

protobuf — protobuf runtime
protobuf-codegen — protobuf codegen engine and protoc-gen-rust plugin for protoc command
protoc — programmatically work with protoc command
protoc-rust — codegen which can be invoked programmatically using protoc binary (e. g. from build.rs)
protobuf-codegen-pure — pure rust codegen
protoc-bin-vendored — protoc binary packaged as crate, can be used with protoc or protoc-rust crates

About versions and branches

2.*.* is the latest stable version. 2.*.* versions follow semver conventions
versions below 2 are no longer supported

See CHANGELOG.md for a list of changes and compatility issues between versions.

How to generate rust code

There are several ways to generate rust code from .proto files:

Invoke protoc programmatically with protoc-rust crate (protoc-rust crate). Reliable, but depends on protoc binary.
Use pure rust protobuf parser and code generator (protobuf-codegen-pure crate). Convenient, but may generate incorrect code or fail in complex definitions.
Use protoc-gen-rust plugin for Google's protoc command. This is useful to generate code manually or when using external build system.

Generated code

Have a look at generated files, used internally in rust-protobuf:

descriptor.rs for descriptor.proto (that is part of Google protobuf)

Rustdoc

docs.rs hosts rustdoc for protobuf.

Getting help

Feel free to open an issue if you need help with rust-protobuf.

Copy-on-write

Rust-protobuf can be used with bytes crate.

To enable Bytes you need to:

Enable with-bytes feature in rust-protobuf:

[dependencies]
protobuf = { version = "2", features = ["with-bytes"] }

Enable bytes option

with Customize when codegen is invoked programmatically:

With stable rust-protobuf:

protoc_rust::run(protoc_rust::Args {
    ...
    customize: Customize {
        carllerche_bytes_for_bytes: Some(true),
        carllerche_bytes_for_string: Some(true),
        ..Default::default()
    },
 });

With rust-protobuf from master:

protoc_rust::Args::new()
    ...
    .customize(Customize {
        carllerche_bytes_for_bytes: Some(true),
        carllerche_bytes_for_string: Some(true),
        ..Default::default()
    })
    .run()?;

or in .proto file:

import "rustproto.proto";

option (rustproto.carllerche_bytes_for_bytes_all) = true;
option (rustproto.carllerche_bytes_for_string_all) = true;

With these options enabled, fields of type bytes or string are generated as Bytes or Chars respectively. When CodedInputStream is constructed from Bytes object, fields of these types get subslices of original Bytes object, instead of being allocated on heap.

serde_derive support

(Only in master, not released yet)

Rust-protobuf can be used with serde.

To enable serde you need to:

Enable serde option

with Customize when codegen is invoked programmatically:

with stable rust-protobuf:

protoc_rust::run(protoc_rust::Args {
    ...
    customize: Customize {
        serde_derive: Some(true),
        ..Default::default()
    },
});

with rust-protobuf from master:

protoc_rust::Args::new()
    ...
    .customize(Customize {
        serde_derive: Some(true),
        ..Default::default()
    })
    .run()?;

or in .proto file:

import "rustproto.proto";

option (rustproto.serde_derive_all) = true;

You may now Serialize and Deserialize messages:

let my_message = MyMessage::new();
serde_json::to_string(&my_message).unwrap();

Related projects

quick-protobuf — alternative protobuf implementation in Rust
prost — another protobuf implementation in Rust
serde-protobuf
grpc-rust — implementation of gRPC based on this library
grpc-rs — another gRPC implementation for Rust

Comments

Implement Deserialize and Serialize

What

This is a work in progress PR for beginning to allow rust-protobuf enums to implement Serialize / Deserialize from serde.

EDIT

Adding support for messages as well, not just enums.

Related issue: #242

opened by chinedufn 37
stream: use static dispatch for CodedInputStream

This pr use static dispatch for CodedInputStream. According to the perf test, there are some significant improvements in various cases.

master:

test1: write: 77 ns per iter test1: read: 97 ns per iter test1: read reuse: 92 ns per iter test_repeated_bool: write: 69 ns per iter test_repeated_bool: read: 165 ns per iter test_repeated_bool: read reuse: 125 ns per iter test_repeated_packed_int32: write: 102 ns per iter test_repeated_packed_int32: read: 217 ns per iter test_repeated_packed_int32: read reuse: 165 ns per iter test_repeated_messages: write: 235 ns per iter test_repeated_messages: read: 1171 ns per iter test_repeated_messages: read reuse: 327 ns per iter test_optional_messages: write: 205 ns per iter test_optional_messages: read: 535 ns per iter test_optional_messages: read reuse: 325 ns per iter test_strings: write: 118 ns per iter test_strings: read: 376 ns per iter test_strings: read reuse: 172 ns per iter test_small_bytearrays: write: 274 ns per iter test_small_bytearrays: read: 251 ns per iter test_small_bytearrays: read reuse: 130 ns per iter test_large_bytearrays: write: 20139 ns per iter test_large_bytearrays: read: 10893 ns per iter test_large_bytearrays: read reuse: 7678 ns per iter

opt-is:

test1: write: 78 ns per iter test1: read: 59 ns per iter test1: read reuse: 52 ns per iter test_repeated_bool: write: 86 ns per iter test_repeated_bool: read: 114 ns per iter test_repeated_bool: read reuse: 65 ns per iter test_repeated_packed_int32: write: 134 ns per iter test_repeated_packed_int32: read: 134 ns per iter test_repeated_packed_int32: read reuse: 71 ns per iter test_repeated_messages: write: 199 ns per iter test_repeated_messages: read: 1003 ns per iter test_repeated_messages: read reuse: 166 ns per iter test_optional_messages: write: 190 ns per iter test_optional_messages: read: 362 ns per iter test_optional_messages: read reuse: 165 ns per iter test_strings: write: 123 ns per iter test_strings: read: 246 ns per iter test_strings: read reuse: 109 ns per iter test_small_bytearrays: write: 259 ns per iter test_small_bytearrays: read: 219 ns per iter test_small_bytearrays: read reuse: 93 ns per iter test_large_bytearrays: write: 19579 ns per iter test_large_bytearrays: read: 12098 ns per iter test_large_bytearrays: read reuse: 6702 ns per iter

This pr break Message into Message and CodedMessage, which will require users to use CodedMessage explicitly once they upgrade protobuf. Old generated code will not work if this pr is merged.

I have no idea what format rules this project follows, so I haven't formatted the code yet, any suggestions?

I also changed CodedOutputStream to static dispatch too, but it's not included in the pr, because I see performance decrease in some cases. I will fire another pr once I figure it out.

opened by BusyJay 28
Use `allow(clippy::all)` instead of `allow(clippy)` as reccomended.
Closes #331 .

Note this currently requires a nightly only feature. Users would have to be using something like:

#![cfg_attr(feature = "cargo-clippy", feature(tool_lints))]
opened by Hoverbear 27
protoc-gen-rust does not follow common file placement conventions
Instead of outputting *.rs files to the "conventional" (per cpp, go, and python) directories -- relative to the source proto files -- the generated files are placed relative to the command invocation site.

When I run the following example command:

protoc --rust_out . ./first/second/example.proto

Instead of

./first/second/example.rs

We see

./example.rs

A more complete example with several generators:

> protoc --rust_out . ./first/second/example.proto > protoc --cpp_out . ./first/second/example.proto > protoc --go_out . ./first/second/example.proto > protoc --python_out . ./first/second/example.proto > protoc --java_out . ./first/second/example.proto > tree ├── example.rs # Rust ├── first │ └── second │ ├── example_pb2.py # Python │ ├── example.pb.cc # Cpp │ ├── example.pb.go # Go │ ├── example.pb.h # Cpp │ └── example.proto # Proto └── org └── pubref └── rules_protobuf └── examples └── CommonProto.java # Java

It appears that only we and the Java rules break from "convention".

I understand this change would likely break many existing users, so I'd understand if this change was rejected for that reason. Example short change diff

EDIT:

To clarify, the primary reason this is an issue for me (and hasn't been for anyone else yet presumably) is because I'm using this protoc extension with bazel instead of cargo+build.rs file, specifically the rules_protobuf "library". That library has an expectation built in that generated source files are placed in a directory structure adjacent to the underlying proto files. It seems that assumption holds for all protoc-gen-* except us and java.
help-wanted
opened by acmcarther 25
Input Stream "Raw Bytes" Optimizations
Speed up handling of nontrivial "raw_bytes" reads within the input stream (a big need we have at dropbox).

Switch to direct reader-to-slice I/O and bypass the buffer for read sizes greater than the buffer

Switch to bytes crate for managing buffering, bytes remaining, start/end positions, using copy_nonoverlapping to slices.. it takes care of a lot of that for us and eliminates some moving pieces.

Add a performance test for 1k byte arrays and 100k byte arrays.

Results:

1k byte arrays are ~4x faster

100k byte arrays are ~10x faster

Ran tests, and have done lots of loaded profiles on internal daemons here at Dropbox.
opened by jamwt 19
Expose enum for `one_of` fields
I see that rust-protobuf maps protobuf one_of fields to a Rust enum internally, which makes sense. However, it doesn't expose the enum publicly from the Message type, so you can't write nice match-using code for it, and have to resort to chained if-statements.

It'd be nice to be able to write something like:

message A { one_of things { string s = 1; int32 i = 2; } }

fn foo(a : A) { match a.get_things() { A_oneof_things::s(s) => {}, A_oneof_things::i(i) => {}, } }
opened by luser 17

Obtain concrete type of an enum

I have this compiled protobuf code.

#[derive(PartialEq,Clone,Default)]
pub struct Request {
    // message oneof groups
    value: ::std::option::Option<Request_oneof_value>,
    // special fields
    unknown_fields: ::protobuf::UnknownFields,
    cached_size: ::protobuf::CachedSize,
}

// see codegen.rs for the explanation why impl Sync explicitly
unsafe impl ::std::marker::Sync for Request {}

#[derive(Clone,PartialEq)]
pub enum Request_oneof_value {
    echo(RequestEcho),
    flush(RequestFlush),
    info(RequestInfo),
    set_option(RequestSetOption),
    deliver_tx(RequestDeliverTx),
    check_tx(RequestCheckTx),
    commit(RequestCommit),
    query(RequestQuery),
    init_chain(RequestInitChain),
    begin_block(RequestBeginBlock),
    end_block(RequestEndBlock),
}

In my client code I am reading a buffer into the generic Request struct.

How can I obtain the concrete type that my Request.

Ideally I would like something like this:

    match message.value.unwrap() {
        types::Request_oneof_value::begin_block => {
            println!("begin_block");
        },
        types::Request_oneof_value::check_tx => {
            println!("check_tx");
        },
        types::Request_oneof_value::commit => {
            println!("commit");
        },
        types::Request_oneof_value::deliver_tx => {
            println!("deliver_tx");
        },
        types::Request_oneof_value::echo => {
            println!("echo");
        },
        types::Request_oneof_value::end_block => {
            println!("end_block");
        },
        types::Request_oneof_value::flush => {
            println!("flush");
        },
        types::Request_oneof_value::info => {
            println!("info");
        },
        types::Request_oneof_value::init_chain => {
            println!("init_chain");
        },
        types::Request_oneof_value::query => {
            println!("query");
        },
        types::Request_oneof_value::set_option => {
            println!("set_option");
        }
    }

opened by adrianbrink 16

Decode unknown enum values
Currently rust-protobuf results in error when decoding unknown enum values.

Protobuf 3 requires that message should preserve unknown enum values, not simply drop them.

Proposal

struct<E : ProtobufEnum> ProtobufEnumOrUknown<E> { ... }

will be introduced in the protobuf library.

Generated fields will be of that type:

color: ProtobufEnumOrUnknown<Color>; // for protobuf3 signular enums color: Option<ProtobufEnumOrUnknown<Color>>; // for protobuf2 singular enums colors: Vec<ProtobufEnumOrUnknown<Color>>; // for repeated enums

Getters and setters will work with enums as previously (e. g. Color).
opened by stepancheg 14

Cannot include generated code

Hi,

following cargo's documentation I've tried to generate my protobufs at build time (through a build.rs script) and include them in my main.rs using:

...
mod model {
    include!(concat!(env!("OUT_DIR"), "/model.rs"));
}
...

However, the (rustc) compilation then fails with:

.../target/debug/build/pubmonitor-42b07ea6e237e2f6/out/model.rs:3:3: 3:4 error: an inner attribute is not permitted in this context
.../target/debug/build/pubmonitor-42b07ea6e237e2f6/out/model.rs:3 #![allow(dead_code)]
                                                                    ^
.../target/debug/build/pubmonitor-42b07ea6e237e2f6/out/model.rs:3:3: 3:4 help: place inner attribute at the top of the module or block
...

It seems that module level attributes (e.g. #![allow(dead_code)] here) are not allowed in included content. Would there be a way to avoid these in the generated code by turning them into item-level attributes where needed?

Many thanks, P.

opened by xitep 14

rustfmt ignore deprecated
Now in Rust 1.44, the

#![cfg_attr(rustfmt, rustfmt_skip)]

tags are now deprecated in favour of #![rustfmt::skip], I tried to replace the tags but for some reason it doesn't work with the autogen code, and I didn't dig deeper.

To replicate, I think it's just sufficient when updating to 1.44 and run a cargo fmt and it will format all autogen files in this repo or otherwise.
opened by austinabell 13
Implement Hash and Eq on protobuf messages

For our use case it'd be really convenient to have the Hash and Ord trait implemented on protobuf messages. Mainly to be able to do some caching using protobuf keys in a HashMap or BTreeMap.

I see a way to do this by generating these traits using only the actual protobuf fields, and not the specials fields like unknown_fields.

Is this something you'd be interested in merging?

opened by thijsc 13
Add From and Into support for SystemTime using references

Enable conversions between Timestamp and SystemTime using references without cloning, and add support for SystemTime::from(Timestamp) in addition to Into (which is derived from the From implementations)

Refer to the note in https://doc.rust-lang.org/std/convert/trait.Into.html

opened by neilisaac 1
MessageDyn: Do not omit zero values on fields containing oneof
Omitting zeros on singular fields only make sense if it's not in oneof For example, omitting zero on the following oneof will make it ambiguous

message IntOrString { oneof int_or_string { int32 int = 1; string str = 2; } }

if either int or str is zero, both of the fields will be missing. Now it's not possible to know which field was originally filled
opened by mingyuchi 0
How to install?

How is this intended to be installed (particularly on windows).

I'm trying to build a Rust project that requires this, and I took a stab in the dark with "cargo install protobuf", but no dice.

opened by stevedefazio 1
Add support for wrapping message in Option
MessageField has a noticeable overhead where there are lots of small nested messages being created and dropped frequently, because it wraps messages in Box.

This PR add an option option_for_message to rustproto that allows to wrap a message in an Option in generated code.

Also adding a simple benchmark shows the improvement.

# Results in Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz test clone_and_drop_message_field ... bench: 148 ns/iter (+/- 24) test clone_and_drop_message_field_with_option ... bench: 36 ns/iter (+/- 5)

I would appreciate feedback!
opened by overvenus 0