It looks like we're assuming that std::net::Ipv4 matches the libc sockaddr_in (same for IPv6), for example in https://github.com/quinn-rs/quinn/blob/main/quinn/src/platform/unix.rs#L357. However, the standard library never guaranteed this, and in fact is looking at changing this in https://github.com/rust-lang/rust/pull/78802. We should fix our usage.

We have an internal abstraction over the cryptographic protocol used called the crypto::Session. (Note that this trait is currently private, so you'd have to make this public to start work. We'd be happy to merge that, but would likely prefer doing so when something materializes that actually uses that.)

Currently we have a single implementation for rustls. In QUIC v1, only TLS 1.3 is officially supported, so supporting something else is technically outside of the spec and can thus probably only be made to work between two consenting endpoints.

If we wanted to support Noise, that would likely start by writing an implementation of this thread that is based on Noise. I think there are some crates that implement (parts of) Noise -- snow probably makes for a good start.

There was a 2018 attempt at integrating Noise support into a much earlier version of Quinn. Quinn looks basically nothing like it did at the time, but maybe there are still useful bits to look at there.

If you want to work on this, we're usually available for questions on our Gitter channel -- just commenting in this issue should also work fine.

This is a continuation of https://github.com/quinn-rs/quinn/pull/1180. I tried to keep both the changes by @dvc94ch and the original code, however I had to make a monkey fix in cmgs.rs in order to fix tests.

@dvc94ch all good?

Experiment on implement libp2p-tls.

But it need peer's certificates. I see rustls has get_peer_certificates func on Session trait. Consider add it to crypto::Session in quinn-proto and Connection in quinn.

This updates the following:

• tokio: 0.2.0-alpha.6 -> 0.2.2
• futures: 0.3.0-alpha.18 -> 0.3.1
• bytes: 0.4.7 -> 0.5.2
• string: 0.2 -> master
• http: a3a8fcb213bc456e0b7a42cf0e2bd57afa49851b -> 43dffa1eb79f6801e5e07f3338fa56191dc454bb

Tests, examples and benchmarks are minimally changed to keep the PR small.

libp2p needs to know the local and remote socket addresses for every incoming connection as soon as the connection is accepted. Is this a feature that you would be willing to include? If so, I will submit a PR.

Some Arc Shenanigans

To initialize a server/client endpoint, Quinn first requires configuration (e.g. via TransportConfig, ServerConfig, etc.). As mentioned in the documentation, the default options are sufficient for most use cases. Otherwise, we have to go through some hoops with setter methods to achieve the behavior we want.

use quinn::{ServerConfig, TransportConifg, VarInt};
use std::{sync::Arc, time::Duration};

// Configure some special ALPN protocols via `rustls`
let mut server_config = ServerConfig::with_crypto(crypto);

// Create a new transport configuration, and modify its defaults
let mut transport_config = TransportConfig::default();
transport_config
    .keep_alive_interval(Some(Duration::from_secs(30)))
    .max_idle_timeout(Some(VarInt::from(60u32).into()));

// And then... replace the old transport config? This is necessary
// because there's no public constructor for `ServerConfig` that accepts a
// `TransportConfig` at the moment; there is only one for the `crypto` field.
server_config.transport = Arc::new(transport_config);

// Alternatively, we may also modify through the `Arc` directly
// since we're the sole owner of the reference at the moment.
// But, now we're forced to use a nasty `unwrap` in the code...
//
// We may, of course, succumb to the `unsafe` version: `unwrap_unchecked`.
// Although this is technically legal, the Quinn public API documents
// no guarantees about the reference count of the `transport` field. Hence,
// a version bump may unintentionally invoke undefined behavior if we're not careful.
Arc::get_mut(&mut server_config.transport)
    .unwrap()
    .keep_alive_interval(Some(Duration::from_secs(30)))
    .max_idle_timeout(Some(VarInt::from(60u32).into()));

// Finally initialize the endpoint
let (endpoint, incoming) = Endpoint::server(server_config, addr)?;

I hope the example above demonstrates my nitpicks with the current API. This is exactly what I faced in one of my projects which required a custom protocol (mostly for network efficiency's sake because JSON-over-HTTP is too verbose).

My Proposed Solution

Instead, I propose that all fields of the various *Config structs be made public. After all, most of their methods are solely transparent setters. We may as well treat the fields to be public.

To accommodate for default values, we simply use the default spread syntax (..Default::default()) when initializing from the user code.

use quinn::{ServerConfig, TransportConifg, VarInt};
use std::{sync::Arc, time::Duration};

// NOTE: Observe that `mut` is no longer necessary.
let server_config = ServerConfig {
    // Configure some special ALPN protocols via `rustls`
    crypto: Arc::new(rustls::server::ServerConfig { ... }),
    transport: Arc::new(TransportConfig {
        keep_alive_interval: Some(Duration::from_secs(30)),
        max_idle_timeout: Some(VarInt::from(60u32).into()),
        // 🎉 Hooray for default values! 🎉
        ..Default::default()
    }),
    // 🎉 Hooray for **more** default values! 🎉
    // ASIDE: `ServerConfig` does not currently implement `Default`
    // as of Quinn 0.8. This should be addressed before proceeding.
    ..Default::default()
};

// Finally initialize the endpoint
let (endpoint, incoming) = Endpoint::server(server_config, addr)?;

Here, I demonstrate the fact that we no longer have to jump through the hoops of indirection. In particular, direct initialization allows us to remove the Arc shenanigans altogether.

This, to me, is a significantly more ergonomic API than using the original setters. Of course, we don't have to remove the setters from the library. Though, I am not totally against deprecation.

I would love to know your thoughts about this API change. I'd love to send in a PR that addresses this. Just wanted to collect some feedback before committing my time.

On a successful connection in either direction (i.e., either actively via connect or passively via listen) I get two things eventually - one Connection and one IncomingStreams. As a user i expect these to be related at some level since they denote a connection to the peer. However it seems that closing the connection does nothing for the IncomingStreams.

Details:

0. I get a Connection and IncomingStreams from a peer while I'm listening
1. Something happens and I decide I don't want this peer any more
2. I get the Connection object and do Connection::close(...).
3. Nothing happens to the IncomingStreams - it continue to stay indefinitely with tokio event loop - I would have expected it to resolve to completion/failure at this point and destroy itself.
4. Remote peer still tries to send something to us on their connection to us (Which is still alive for them) by trying to open a new stream to us. The peer keeps getting "ConnectionAborted - closed by remote peer" error which is fine and expected.

So the IncomingStreams stream (and possibly any other stream obtained from it which the remote hasn't shutdown/closed/destroyed yet - though I haven't checked this part) uselessly remains unresolved with tokio. I have to now keep extra knowledge about closing these streams when I close the Connection to the peer.

Wouldn't the better/expected design be that when I close Connection (or drop/destroy it) all the related stuff resolve into an error (or anything, but resolve) to gracefully collect all resources ?

Hi, Thanks for the hard work.

I am trying to implement a proxy client/server pair using quinn, I get it working quickly thanks to quinn's succinct APIs, but the throughput is quite low.

The client is running on MacOS, and server on Linux, RTT reported by ping is 170ms, while quinn::Connection::rtt() reports 190ms. I have another C++ proxy implementation using TCP, which runs alot faster with the same network link. I can observe that the server is quite fast at writing data to the client, each write with 6k to 8k bytes for most the time, but the client reads 1k to 3k bytes for each attempt and never catches up with server's write speed. I tried setting SND_BUF/RCV_BUF up to 3 megabytes, but nothing changes, and I don't know what to investigate on. I guess there may be some mistakes in my code, The client code for relaying traffic is as the following:

pub async fn serve(&mut self, local_conn_receiver: &mut Receiver<TcpStream>) -> Result<()> {
    let remote_conn = &self.remote_conn.as_ref().unwrap();
    // accept local connections and build a tunnel to remote for accepted connections
    while let Some(local_conn) = local_conn_receiver.recv().await {
        match remote_conn.open_bi().await {
            Ok((remote_send, remote_recv)) => {
                tokio::spawn(Self::handle_stream(local_conn, remote_send, remote_recv));
            }
            Err(e) => {
                error!("failed to open_bi on remote connection: {}", e);
                break;
            }
        }
    }

    info!("quit!");
    Ok(())
}

async fn handle_stream(
    mut local_conn: TcpStream,
    mut remote_send: SendStream,
    mut remote_recv: RecvStream,
) -> Result<()> {
    info!("open new stream, id: {}", remote_send.id().index());

    let mut local_read_result = ReadResult::Succeeded;
    loop {
        let (mut local_read, mut local_write) = local_conn.split();
        let local2remote = Self::local_to_remote(&mut local_read, &mut remote_send);
        let remote2local = Self::remote_to_local(&mut remote_recv, &mut local_write);

        tokio::select! {
            Ok(result) = local2remote, if !local_read_result.is_eof() => {
                local_read_result = result;
            }
            Ok(result) = remote2local => {
                if let ReadResult::EOF = result {
                    info!("quit stream after hitting EOF, stream_id: {}", remote_send.id().index());
                    break;
                }
            }
            else => {
                info!("quit unexpectedly, stream_id: {}", remote_send.id().index());
                break;
            }
        };
    }
    Ok(())
}

async fn local_to_remote<'a>(
    local_read: &'a mut ReadHalf<'a>,
    remote_send: &'a mut SendStream,
) -> Result<ReadResult> {
    let mut buffer = vec![0_u8; 8192];
    let len_read = local_read.read(&mut buffer[..]).await?;

    if len_read > 0 {
        remote_send.write_all(&buffer[..len_read]).await?;
        Ok(ReadResult::Succeeded)
    } else {
        remote_send.finish().await?;
        Ok(ReadResult::EOF)
    }
}

async fn remote_to_local<'a>(
    remote_recv: &'a mut RecvStream,
    local_write: &'a mut WriteHalf<'a>,
) -> Result<ReadResult> {
    let mut buffer = vec![0_u8; 8192];
    let result = remote_recv.read(&mut buffer[..]).await?;
    if let Some(len_read) = result {
        local_write.write_all(&buffer[..len_read]).await?;
        local_write.flush().await?;
        Ok(ReadResult::Succeeded)
    } else {
        Ok(ReadResult::EOF)
    }
}

quinn-proto is extremely low-level and difficult to use. quinn is much nicer to use, but depends on Tokio. Would it be possible for quinn to also support async_std via a Cargo feature?

I was hoping to get some clarification on how the stateless reset functionality works in quinn/QUIC. My understanding of the stateless reset feature is that if a connection exists between two endpoints A and B and one side of the connection state is lost for some reason (say endpoint A crashed and was restarted) then the side that lost state would be able to inform the other side and the old connection would be closed quickly on the side that didn't lose state. In my limited testing i've found this to not be the case though, and instead (via logs) i'm able to see that the side the lost sate (A) is sending a stateless reset but the other side (B) doesn't seem to handle the reset and close the connection on its side, instead the connection hangs until the configured idle timeout period at which point the connection is timed out.

Is my understanding of how the stateless reset is supposed to work not correct? Is there some configuration that I need to be doing differently in order to have this work properly? (eg do you need to use a non-default EndpointConfig)

Hello, I am implementing quinn_proto within Kompact, a distributed actor framework which has its own event loop(reason I chose quinn_proto instead of quinn_rs) as a part of my master thesis.

I am encountering couple of problems with the API, which I am having a hard time figuring out. The test case I am trying to execute is following: // Sets up two KompactSystems with 2x actor Pingers and actor Pongers and QUIC as the network protocol. One Ponger is registered by UUID, // the other by a custom name. One Pinger communicates with the UUID-registered Ponger, // the other with the named Ponger. Both sets are expected to exchange PING_COUNT ping-pong // messages. This is the test, in mu case In my case I send 10 ping pongs between the actors messages.https://github.com/jodiserla/kompact/blob/5836a4697b04309ffc407c45ece8ebf5a9d959ff/core/tests/dispatch_integration_tests.rs#L339

However, when I execute the test case, the actors send varying number of ping pongs between them and then the test starts to "hang" until the test times out by itself. It seems that QUIC stops on the handle_event() function and doesn't do any further than that - causing the test to hang and timeout. Here is the code for calling the API methods https://github.com/jodiserla/kompact/blob/5836a4697b04309ffc407c45ece8ebf5a9d959ff/core/src/net/quic_endpoint.rs#L142 I also noticed that the endpoint.poll_transmit() doesn't seem to get called either which confuses me as well - https://github.com/jodiserla/kompact/blob/5836a4697b04309ffc407c45ece8ebf5a9d959ff/core/src/net/quic_endpoint.rs#L128 Any feedback on this would be greatly appreciated!

log

sur# cargo build
   Compiling quinn-udp v0.3.2 (/home/build/quinn-0.9.3/quinn-udp)
error[E0425]: cannot find value `IP_RECVTOS` in crate `libc`
    --> quinn-udp/src/unix.rs:94:75
     |
94   |         if let Err(err) = set_socket_option(&*io, libc::IPPROTO_IP, libc::IP_RECVTOS, OPTION_ON) {
     |                                                                           ^^^^^^^^^^ help: a constant with a similar name exists: `IP_RECVIF`
     |
    ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/netbsdlike/openbsd/mod.rs:1068:1
     |
1068 | pub const IP_RECVIF: ::c_int = 30;
     | ---------------------------------- similarly named constant `IP_RECVIF` defined here

error[E0412]: cannot find type `mmsghdr` in crate `libc`
   --> quinn-udp/src/unix.rs:166:26
    |
166 |       let mut msgs: [libc::mmsghdr; BATCH_SIZE] = unsafe { mem::zeroed() };
    |                            ^^^^^^^ help: a struct with a similar name exists: `cmsghdr`
    |
   ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/mod.rs:10:1
    |
10  | / s! {
11  | |     pub struct sockaddr {
12  | |         pub sa_len: u8,
13  | |         pub sa_family: sa_family_t,
...   |
124 | |     }
125 | | }
    | |_- similarly named struct `cmsghdr` defined here

error[E0425]: cannot find function `sendmmsg` in crate `libc`
   --> quinn-udp/src/unix.rs:197:32
    |
197 |         let n = unsafe { libc::sendmmsg(io.as_raw_fd(), msgs.as_mut_ptr(), num_transmits as _, 0) };
    |                                ^^^^^^^^ help: a function with a similar name exists: `sendmsg`
    |
   ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/mod.rs:743:5
    |
743 |     pub fn sendmsg(fd: ::c_int, msg: *const ::msghdr, flags: ::c_int) -> ::ssize_t;
    |     ------------------------------------------------------------------------------- similarly named function `sendmsg` defined here

error[E0412]: cannot find type `mmsghdr` in crate `libc`
   --> quinn-udp/src/unix.rs:296:50
    |
296 |       let mut hdrs = unsafe { mem::zeroed::<[libc::mmsghdr; BATCH_SIZE]>() };
    |                                                    ^^^^^^^ help: a struct with a similar name exists: `cmsghdr`
    |
   ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/mod.rs:10:1
    |
10  | / s! {
11  | |     pub struct sockaddr {
12  | |         pub sa_len: u8,
13  | |         pub sa_family: sa_family_t,
...   |
124 | |     }
125 | | }
    | |_- similarly named struct `cmsghdr` defined here

error[E0425]: cannot find function `recvmmsg` in crate `libc`
   --> quinn-udp/src/unix.rs:308:19
    |
308 |             libc::recvmmsg(
    |                   ^^^^^^^^ help: a function with a similar name exists: `recvmsg`
    |
   ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/mod.rs:748:5
    |
748 |     pub fn recvmsg(fd: ::c_int, msg: *mut ::msghdr, flags: ::c_int) -> ::ssize_t;
    |     ----------------------------------------------------------------------------- similarly named function `recvmsg` defined here

error[E0531]: cannot find unit struct, unit variant or constant `IP_RECVTOS` in crate `libc`
    --> quinn-udp/src/unix.rs:472:73
     |
472  |             (libc::IPPROTO_IP, libc::IP_TOS) | (libc::IPPROTO_IP, libc::IP_RECVTOS) => unsafe {
     |                                                                         ^^^^^^^^^^ help: a constant with a similar name exists: `IP_RECVIF`
     |
    ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.138/src/unix/bsd/netbsdlike/openbsd/mod.rs:1068:1
     |
1068 | pub const IP_RECVIF: ::c_int = 30;
     | ---------------------------------- similarly named constant `IP_RECVIF` defined here

Some errors have detailed explanations: E0412, E0425, E0531.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `quinn-udp` due to 6 previous errors

In our application using UDP, the client is sending some UDP messages to the server and the server get the client address (ip:port) and send the responses back to the client where the client is running a packet receiver doing recvfrom as well on the same client address. We are planning to move it to using QUIC for better QOS control.

Is it possible to do the same in the Quinn's quic implementation? i.e. using the same ip:port for the quic server and client on the same node? I am aware that one can use bi-directional streams to send messages in both directions, but that would require significant changes to our programming model. cc @djc

When enabled on a socket with IP_TRANSPARENT, it populates metadata with the original destination (IP:port) for traffic redirected with TPROXY.

Signed-off-by: Piotr Sikora [email protected]

I did some benchmarking with a quinn based RPC framework. https://github.com/n0-computer/quic-rpc .

I found that unsurprisingly, the packet size has a huge influence on throughput. Here are benchmarks on a linux box with different values for initial_max_udp_payload_size:

The default:

$ ./target/release/bulk --initial-mtu 1200

Client 0 stats:
Overall download stats:

Transferred 1073741824 bytes on 1 streams in 4.48s (228.59 MiB/s)

Stream download metrics:

      │  Throughput   │ Duration 
──────┼───────────────┼──────────
 AVG  │  228.69 MiB/s │     4.48s
 P0   │  228.62 MiB/s │     4.48s
 P10  │  228.75 MiB/s │     4.48s
 P50  │  228.75 MiB/s │     4.48s
 P90  │  228.75 MiB/s │     4.48s
 P100 │  228.75 MiB/s │     4.48s

Largest value that was working for me:

$ ./target/release/bulk --initial-mtu 6000

Client 0 stats:
Overall download stats:

Transferred 1073741824 bytes on 1 streams in 2.18s (468.73 MiB/s)

Stream download metrics:

      │  Throughput   │ Duration 
──────┼───────────────┼──────────
 AVG  │  468.88 MiB/s │     2.18s
 P0   │  468.75 MiB/s │     2.18s
 P10  │  469.00 MiB/s │     2.18s
 P50  │  469.00 MiB/s │     2.18s
 P90  │  469.00 MiB/s │     2.18s
 P100 │  469.00 MiB/s │     2.18s

I found that with a sufficiently large packet size quinn bulk can outperform TCP with default settings, but with the default quinn settings it is slower than TCP (about 2/3).

So it would be nice to ensure that quinn works well in an environment that allows large frames, e.g. a LAN with jumbo frames enabled, or a loopback device with a large MTU.

I think a good way to do this would be to implement a dummy in memory AbstractUdpSocket transport that has basically zero overhead, then make sure quinn works well with large values of initial-mtu up to 65536, or at least up to the 9000 bytes of jumbo frames.

Dear all,

I'm writine some io_uring examples in rust, and I would like to add QUIC. The idea is to try and benchmark some advanced options like batching, zero copy, AF_XDP, eBPF, ....

I guess my best option is to try quinn-proto, as it's advertised for custom event loops. Is there any example, maybe in the tests, maybe a simplified one? Do you think it's doable?

Any advice is welcome, before I will try to dive in the codebase :)