Easy protocol definitions in Rust

Overview

protocol

Build Status Crates.io MIT licensed

Documentation

Easy protocol definitions in Rust.

This crate adds a custom derive that can be added to types, allowing structured data to be sent and received from any IO stream.

Networking is built-in, with special support for TCP and UDP.

The protocol you define can be used outside of networking too - see the Parcel::from_raw_bytes and Parcel::raw_bytes methods.

This crate also provides:

  • TCP and UDP modules for easy sending and receiving of Parcels
  • A generic middleware library for automatic transformation of sent/received data
    • Middleware has already been written to support compression
    • Custom middleware can be implemented via a trait with two methods

Checkout the examples folder for usage.

Usage

Add this to your Cargo.toml:

[dependencies]
protocol = "3.1"
protocol-derive = "3.1"

Under the hood

The most interesting part here is the protocol::Parcel trait. Any type that implements this trait can then be serialized to and from a byte stream. All primitive types, standard collections, tuples, and arrays implement this trait.

This crate becomes particularly useful when you define your own Parcel types. You can use #[derive(Protocol)] to do this. Note that in order for a type to implement Parcel, it must also implement Clone, Debug, and PartialEq.

#[derive(Parcel, Clone, Debug, PartialEq]
pub struct Health(f32);

#[derive(Parcel, Clone, Debug, PartialEq]
pub struct SetPlayerPosition {
    pub position: (f32, f32),
    pub health: Health,
    pub values: Vec<String>,
}

Custom derive

Any user-defined type can have the Parcel trait automatically derived.

Example

#[macro_use] extern crate protocol_derive;
#[macro_use] extern crate protocol;

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Handshake;

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Hello {
    id: i64,
    data: Vec<u8>,
}

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Goodbye {
    id: i64,
    reason: String,
}

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Node {
    name: String,
    enabled: bool
}

#[protocol(discriminant = "integer")]
#[derive(Protocol, Clone, Debug, PartialEq)]
pub enum PacketKind {
    #[protocol(discriminator(0x00))]
    Handshake(Handshake),
    #[protocol(discriminator(0xaa))]
    Hello(Hello),
    #[protocol(discriminator(0xaf))]
    Goodbye(Goodbye),
}

fn main() {
    use std::net::TcpStream;

    let stream = TcpStream::connect("127.0.0.1:34254").unwrap();
    let mut connection = protocol::wire::stream::Connection::new(stream, protocol::wire::middleware::pipeline::default());

    connection.send_packet(&Packet::Handshake(Handshake)).unwrap();
    connection.send_packet(&Packet::Hello(Hello { id: 0, data: vec![ 55 ]})).unwrap();
    connection.send_packet(&Packet::Goodbye(Goodbye { id: 0, reason: "leaving".to_string() })).unwrap();

    loop {
        if let Some(response) = connection.receive_packet().unwrap() {
            println!("{:?}", response);
            break;
        }
    }
}

Enums

Discriminators

Enum values can be transmitted either by their 1-based variant index, or by transmitting the string name of each variant.

NOTE: The default behaviour is to use the variant name as a string (string).

This behaviour can be changed by the #[protocol(discriminant = "<type>")] attribute.

Supported discriminant types:

  • string (default)
    • This transmits the enum variant name as the over-the-wire discriminant
    • This uses more bytes per message, but it very flexible
  • integer
    • This transmits the 1-based variant number as the over-the-wire discriminant
    • If enum variants have explicit discriminators, the
    • Enum variants cannot be reordered in the source without breaking the protocol
#[derive(Protocol, Clone, Debug, PartialEq)]
#[protocol(discriminant = "string")]
pub enum PlayerState {
  Stationary,
  Flying { velocity: (f32,f32,f32) },
  // Discriminators can be explicitly specified.
  #[protocol(discriminator("ArbitraryOverTheWireName"))]
  Jumping { height: f32 },
}

Misc

You can rename the variant for their serialisation.

#[derive(Protocol, Clone, Debug, PartialEq)]
#[protocol(discriminant = "string")]
pub enum Foo {
  Bar,
  #[protocol(name = "Biz")] // the Bing variant will be send/received as 'Biz'.
  Bing,
  Baz,
}
Comments
  • Support array sizes over 32

    Support array sizes over 32

    Hey,

    I'm currently working on a Macho parser and couldn't figure out how to declare sized char arrays (char foo[16]). I tried [u8; 16], with no luck.

    Is that possible right now at all?

    Thanks!

    opened by 19h 8
  • Support for uuid seams broken.

    Support for uuid seams broken.

    I am having issues with getting the feature to use Uuid to work.

    Cargo.toml:

    [dependencies]
    protocol-derive = "3.1.3"
    protocol= { version = "3.1.3", features = ["uuid"] }
    uuid="0.8.1"
    

    Trying to compile something simple as

    use uuid::Uuid;
    use protocol_derive::Protocol;
    
    #[derive(Protocol, Debug, Copy, Clone, PartialEq, Eq)]
    pub struct Foo {
        pub id: Uuid,
    }
    

    And then I got a compiler error:

    #[derive(Protocol, Debug, Copy, Clone, PartialEq, Eq)]
    |        ^^^^^^^^ the trait `id::__impl_protocol___Parcel_FOR_Foo::protocol::HighLevel` is not implemented for `uuid::Uuid`
    

    Am I missing something?

    opened by Superhepper 3
  • Not possible to get discriminant of received packet?

    Not possible to get discriminant of received packet?

    I am interested in retrieving the discriminant of the packet that I have received. Though it seams as this is not possible. And I only wish to ask if I have understood this correctly?

    If we use the example from the documentation. What would hope would be possible would be to do the following:

        loop {
            if let Some(response) = connection.receive_packet().unwrap() {
                println!("{:?}", response.discriminant());
                break;
            }
        }
    

    But to me it seams as the only way one can retrieve discriminators is through the enum that defines the protocol?

    #[macro_use] extern crate protocol_derive;
    #[macro_use] extern crate protocol;
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Handshake;
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Hello {
        id: i64,
        data: Vec<u8>,
    }
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Goodbye {
        id: i64,
        reason: String,
    }
    
    #[protocol(discriminant = "integer")]
    #[derive(Protocol, Clone, Debug, PartialEq)]
    #[repr(u16)]
    pub enum PacketKind {
        #[protocol(discriminator(0x00))]
        Handshake(Handshake),
        #[protocol(discriminator(0xaa))]
        Hello(Hello),
        #[protocol(discriminator(0xaf))]
        Goodbye(Goodbye),
    }
    
    fn main() {
    println!("Disc: {}", Packet::Handshake(Handshake)).discriminant());
    }
    

    In order to get the discriminant of message do I have to write my own function that checks the message type and return the appropriate discriminator by constructing a temporary object and extract it from the temporary object?

    use protocol_derive;
    use protocol::{ByteOrder, Enum, Parcel, Settings};
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Handshake;
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Hello {
        id: i64,
        data: Vec<u8>,
    }
    
    #[derive(Protocol, Clone, Debug, PartialEq)]
    pub struct Goodbye {
        id: i64,
        reason: String,
    }
    
    #[protocol(discriminant = "integer")]
    #[derive(Protocol, Clone, Debug, PartialEq)]
    #[repr(u16)]
    pub enum PacketKind {
        #[protocol(discriminator(0x00))]
        Handshake(Handshake),
        #[protocol(discriminator(0xaa))]
        Hello(Hello),
        #[protocol(discriminator(0xaf))]
        Goodbye(Goodbye),
    }
    
    impl PacketKind {
    pub fn get_discriminator(&self) -> u16 {
    match self {
        PacketKind::Handshake(_) => {
            PacketKind::Handshake(Handshake).discriminator()
        },
        PacketKind::Hello(_) => {
           PacketKind::Hello(Hello {id: Default::default(), data: Default::default() }).discriminant()
        },
        PacketKind::GoodBye(_) => {
            PacketKind::Goodbye(GoodBye {id: Default::default(), reason: "".to_string() }).discriminant()
        },
    }
    

    It seam really weird that the library user would have to go through all this trouble to get the discriminant of the received package or am I missing something?

    opened by Superhepper 3
  • Doesn't work at latest nightly rust toolchain

    Doesn't work at latest nightly rust toolchain

    error[E0271]: type mismatch resolving `<i64 as std::convert::TryFrom<i8>>::Error == std::num::TryFromIntError`
       --> ...\protocol-0.3.1\src\primitives\numerics.rs:102:6
        |
    102 | impl Integer for i64 { }
        |      ^^^^^^^ expected enum `std::convert::Infallible`, found struct `std::num::TryFromIntError`
        |
        = note: expected type `std::convert::Infallible`
                   found type `std::num::TryFromIntError`
    
    opened by CertainLach 2
  • remove panic on unknown discriminator

    remove panic on unknown discriminator

    This panic line caused one of our production systems to crash when trying to decode messages on a noisy channel.

    https://github.com/dylanmckay/protocol/blob/ae808cf8f6b723c1edf38e0d4d731b9240eb76f5/protocol-derive/src/codegen/enums.rs#L66

    Now, I'm happy to fix this and submit a pull request but how? Because this is part of the derived parcel implementation, I'm not sure what the generated code ends up looking like or what can be returned here to replace the panic

    opened by derkbell 1
  • Null terminated strings

    Null terminated strings

    I am currently implementing a protocol that has 17 bytes or things that I don't care for now and then a 0 terminated string. It is datagram based, so the length of the packet is very clear from the start.

    I am not the creator of the protocol, it has been around for 20 years or so.

    Is there any way I can decode that string? What happens if there are bytes left to consume?

    opened by anarelion 1
  • Make protocol work on stable rust and lastest nightly compiler.

    Make protocol work on stable rust and lastest nightly compiler.

    Removed TryFrom in favor of num-traits

    • Allows protocol to work in stable rust
    • Fixes the compiler errors in the latest nightly (caused by https://github.com/rust-lang/rust/pull/44174)
    opened by Kern-- 1
  • Merge the `Type` and `Packet` traits

    Merge the `Type` and `Packet` traits

    Currently, there is the distinction between a Type that can be serialised, and a Packet that can be serialised. They both share almost the exact same definition.

    It would be good to merge the two traits into one. This would allow connections to be build using primitives as packets, and also make everything more consistent.

    enhancement 
    opened by dylanmckay 1
  • Where clauses ignored when deriving Protocol for traits that have where clauses

    Where clauses ignored when deriving Protocol for traits that have where clauses

    Currently

    #[derive(Protocol)]
    enum Event<E: Event> {
       // ...
    }
    

    will work.

    However

    #[derive(Protocol)]
    enum Event<E> where E: Event {
       // ...
    }
    

    does not. protocol-derive only looks at the constraints on the generic types, it currently fails to look at the where clause constraints and add them to the new impl Parcel.

    Something like this needs to be done

    diff --git a/protocol-derive/src/lib.rs b/protocol-derive/src/lib.rs
    index 517c224..dfeda42 100644
    --- a/protocol-derive/src/lib.rs
    +++ b/protocol-derive/src/lib.rs
    @@ -37,7 +37,7 @@ fn impl_parcel(ast: &syn::DeriveInput) -> proc_macro2::TokenStream {
     /// Returns `(generics, where_predicates)`
     fn build_generics(ast: &syn::DeriveInput) -> (Vec<proc_macro2::TokenStream>, Vec<proc_macro2::TokenStream>) {
         let mut where_predicates = Vec::new();
    -    let generics: Vec<_> = ast.generics.params.iter().enumerate().map(|(i, p)| {
    +    let generics: Vec<_> = ast.generics.type_params().enumerate().map(|(i, p)| {
             match p {
                 syn::GenericParam::Type(t) => {
                     let (ident, bounds) = (&t.ident, &t.bounds);
    

    NOTE: this if 1/3rd of a proper fix. A proper fix would also check generics.lifetimes() and generics.const_params and copy the constraints to the new impl. Currently the patch would ignore lifetime parameters and break all traits with lifetime type parameters.

    opened by dylanmckay 0
  • Fixed length

    Fixed length

    Hello , is it possible to set a fixed length for a string field ? I would need something like this :

    #[derive(Protocol, Debug, Clone, PartialEq, Eq)]
    pub struct DatFile {
        #[protocol(fixed_lenght(8))]
        game_version: String,
    }
    
    opened by oknozor 0
  • Variable protocol depending on context:  Custom Settings or Hints.

    Variable protocol depending on context: Custom Settings or Hints.

    The protocol I am implementing has certain quirks. It is a opcode u16 based protocol, so first 2 bytes define the rest, but:

    Some packets can be embebbed into other packets and depending on that, a CRC is added at the end or not.

    Does protocol has a easy way to map this? Is there a way that I can provide custom settings or hints ?

    opened by anarelion 0
  • Does not work with non_blocking communication.

    Does not work with non_blocking communication.

    I have tried to use this with a non_blocking socket and I get some weird error:

    panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }'
    protocol-3.1.3/src/wire/stream/transport/simple.rs:116:30
    

    Is non blocking sockets supported?

    opened by Superhepper 1
  • Docs on what result of `.receive_packet` means

    Docs on what result of `.receive_packet` means

    Returns a Result<Option<P>, Error>, but the docs are a bit terse

    https://docs.rs/protocol/3.1.4/protocol/wire/stream/struct.Connection.html#method.receive_packet

    Attempts to receive a packet

    • Is this a non-blocking call?
    • What does Ok(None) mean here? Is it an error? If this is non-blocking, does it mean try again later?
    opened by deontologician 0
Owner
Dylan McKay
Dylan McKay
A minimalistic encryption protocol for rust async streams/packets, based on noise protocol and snow.

Snowstorm A minimalistic encryption protocol for rust async streams / packets, based on noise protocol and snow. Quickstart Snowstorm allows you to se

Black Binary 19 Nov 22, 2022
A Constrained Application Protocol(CoAP) library implemented in Rust.

coap-rs A fast and stable Constrained Application Protocol(CoAP) library implemented in Rust. Features: CoAP core protocol RFC 7252 CoAP Observe optio

Covertness 170 Dec 19, 2022
A Rust library for parsing the SOME/IP network protocol (without payload interpretation).

someip_parse A Rust library for parsing the SOME/IP network protocol (without payload interpretation). Usage Add the following to your Cargo.toml: [de

Julian Schmid 18 Oct 31, 2022
Rustus - TUS protocol implementation in Rust.

Rustus Tus protocol implementation written in Rust. Features This implementation has several features to make usage as simple as possible. Rustus is r

Pavel Kirilin 74 Jan 1, 2023
Peer-to-peer communications library for Rust based on QUIC protocol

qp2p Crate Documentation MaidSafe website SAFE Dev Forum SAFE Network Forum Overview This library provides an API to simplify common tasks when creati

MaidSafe 337 Dec 14, 2022
Easy-to-use wrapper for WebRTC DataChannels peer-to-peer connections written in Rust and compiling to WASM.

Easy-to-use wrapper for WebRTC DataChannels peer-to-peer connections written in Rust and compiling to WASM.

null 58 Dec 11, 2022
RakNet Protocol implementation by Rust.

rust-raknet RakNet Protocol implementation by Rust. Raknet is a reliable udp transport protocol that is often used for communication between game clie

b23r0 161 Dec 29, 2022
An easy-to-use tunnel to localhost built in Rust. An alternative to ngrok and frp.

rslocal English | 中文 What is rslocal? Rslocal is like ngrok built in Rust, it builds a tunnel to localhost. Project status support http support tcp su

saltbo 220 Jan 7, 2023
A simple tool in Rust to split urls in their protocol, host, port, path and query parts.

rexturl A simple tool to split urls in their protocol, host, port, path and query parts. Install cargo install rexturl or clone the source code and r

Volker Schwaberow 3 Oct 22, 2022
Rust implementation of TCP + UDP Proxy Protocol (aka. MMProxy)

mmproxy-rs A Rust implementation of MMProxy! ?? Rationale Many previous implementations only support PROXY Protocol for either TCP or UDP, whereas thi

Saikō Technology 3 Dec 29, 2022
🥧 Savoury implementation of the QUIC transport protocol and HTTP/3

quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF. It provides a low level API for processing QUIC packet

Cloudflare 7.1k Jan 8, 2023
Easy per application transparent proxy built on cgroup.

cproxy can redirect TCP and UDP traffic made by a program to a proxy, without requiring the program supporting a proxy. Compared to many existi

Xiangru Lian 263 Dec 20, 2022
🤖 brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers

brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers (plain text data). That is, although it can be accessed from a browser, brwrs will not correctly interpret the browser's GET request.

daCoUSB 3 Jul 30, 2021
A multi-protocol network relay

A multi-protocol network relay

zephyr 43 Dec 13, 2022
The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL.

Graph Node The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL. Graph Node is an open s

Mindy.wang 2 Jun 18, 2022
Transfer file in LAN is so easy

txrx Transfer file in LAN is so easy. Purpose Finding machine IP before executing netcat Multiplatform transfer is collapsing Usage Just run $ txrx fi

Konge 4 Sep 5, 2022
easy to use version controll

Verzcon To start off you need to know are you the server or the client Server If you are the server then you should run verzcon --host then it will ma

Arthur Melton 2 Nov 8, 2021
🤖 Autonomous Twitter bot that posts analytics of the APYs available on the Solend Protocol.

Solend APY Twitter Bot Solana Ignition Hackathon 2021 View Demo · Report Bug · Request Feature Table of Contents About The Project Motivation Challeng

Manuel Gil 4 Sep 23, 2022
Yet Another File Transfer Protocol.

yaftp Yet Another File Transfer Protocol. Build & Run $> cargo build --release Features C2C Lightweight Per something per session High performence Res

b23r0 20 Sep 22, 2022