Similar to #145, and again exercised by #144. If there size of the completion queue is smaller than the number of concurrent writers to the submission queue, the runtime just hangs.

Probably not commiting?

This change refactors the runtime to not use reference counting directly in the ops themselves. Instead, ops access the driver via thread local variables.

This is sound because dropping the driver (which happens when it is removed from its thread-local state) blocks the thread until all ops complete, ensuring that we do not free the contents of the driver until after all operations have completed.

Add infrastructure to manage pre-registered buffers, and operation methods File::read_fixed_at and File::write_fixed_at to make use of them. Exclusive access to buffer data between the application and in-flight ops is controlled at runtime.

This is initial API to enable fixed buffers. Future developments may include:

• :white_check_mark: Improved parameter polymorphism with ~#53~ #172;
• An internal linked list of free buffers, to be able to check out the next available buffer from FixedBufRegistry without enumerating or keeping track of indices;
• Support IORING_REGISTER_BUFFERS2/IORING_REGISTER_BUFFERS_UPDATE.

Needs #52 to get tests to pass.

Handles LifeCycle::CompletionList in driver Drop logic

During the first pass over all lifecycles, identifies if CompletionLists are finished or not.

• If they are, it changes the Lifecycle to Completed with null data.
• If they are not, adds them to the cancel list, and converts to Ignored

This means that after this pass, we will never observe CompletionList in the lifecycles. This means the existing termination logic is ok unchanged. The ignored entry will stay until removed from the Lifecycle slab by the call to complete.

Fixes a gap left by #158 and unblocks #123

Linux 6.0 introduced IORING_OP_SEND_ZC. This PR introduces the function send_zc on udp (todo: tcp) sockets which takes advantage of it.

~This is blocked on this PR on io-uring~ Merged

As IORING_OP_SEND_ZC can return multiple CQE's per SQE, this also extends the Lifecycle handling in impl Fut for Op<T> to check for these occurences. If does this using a new method update in Completable, which I added to minimise the scope of the changes - it just falls back to the complete method for implementations which are just single entry.

This approach could probably be used as a the basis of streaming interfaces for some other submission types.

Since being split, is now blocked on #130

Add TCP, UDP and Unix socket support.

• [x] TCP
  • [x] Stream
  • [x] Listener
• [x] UDP socket
• [ ] Unix
  • [ ] Stream
  • [ ] Listener
  • [ ] Datagram

Open to feedback

This fixes an immediate issue where ops like Accept can cause runtime shutdown to hang.

This may need to be revisited once we have a real cancellation story. It may be better to do cancel-on-drop for ops.

Removes FuturesUnordered from benchmarks and alters timing to ignore task spawning, as to hone in on specifically the processing of ops and driver performance.

The owned slice API has a flaw: b"hello".to_vec().slice(1..).slice(..3) produces a Slice<Slice<Vec<u8>>>. It is easy to inadvertently end up with nested Slice wrappers, which adds unnecessary overhead to the buffer accessor methods and makes the results of .begin() and .end() difficult to interpret.

Move the slice method to a new trait IntoSlice, which is used to uniformly convert buffer parameters to a Slice<T>, with the impl of InfoSlice for Slice<T> closing onto Slice<T>.

Instead of IoBuf which can no longer be implemented for Slice, the buffer parameter type in the generic I/O operation methods is bound with IntoSlice to keep the convenient polymorphism, while reducing code bloat in monomorphization when the application passes both buffers and their slices to tokio-uring operations. Consequently, all buffers returned from the operations are wrapped into Slice.

This PR incorporates #52 in order to get tests to pass.

Make the internal infrastructure for fixed buffers more flexible, allowing FixedBuf handles to be backed by alternative collections. Add one new collection as FixedBufPool, enabling dynamic retrieval of free buffers.

Support of multi completion events, originally proposed in https://github.com/tokio-rs/tokio-uring/pull/123, and submitted separately for review as suggested by @Noah-Kennedy

Multi completion events contain a CQE_F_MORE flag, which indicates more events are expected associated with the same submission entry. This leads to a 1:M correspondance, where all ops currently supported are 1:1

This PR modifies struct Op<T> to support both types. It does this by adding the update method to the Completable trait. The semantics of this are:

Take 1 or more completions, update the state of the operation, and return if the operation is Finished, with the result, or if More are required

For single event Ops, the result will always be Finished. As this is the most likely behaviour, the default method is provided.

To support this with the current Lifecycle handling, the Lifecycle::Completed variant now contains 1 or more completions, rather than 1. This is because there is a data race between reading the completion queue, and operations calling poll() to consume Lifecycle::Completed. To manage this, completions are stored in a growable conatiner. smallvec (a vec with small string optimization) has been chosen to avoid allocations in the vast majority of cases. The default size is chosen to be small (under a cache line), by no rigorous method. It may want to be bigger (8 or 16).

Other changes to Lifecycle include the Ignored variant. This now has to wait for all outstanding completions for an Op before being dropped. It just checks the more flag to do this.

There are design decisions here which want discussing.

The major one in my eyes is the decision to have an unbounded collection of completion events. The other option would be to stall the completion queue handling once some predetermined threshold of completion events for a single op is reached. A waker would be registered, and completion queue handling would be woken when the Lifecycle::Completed was consumed by the Op's Poll(). I deemed this unpalatable, as it stalls all Ops

Provide an example where a single binary both calls TCP listen accept and connect, and that tracks the file descriptor values, to show that the tokio_uring accept operation can leak file descriptors if it is canceled at an inopportune time: when the uring accept operation is being run by the kernel, a client makes the connection, but the server side has canceled the future.

It's a small time window where this can happen, but it shows potentially a larger problem when uring operations in flight and having to deal with cleanup when the future is cancelled.

Run with

cargo run --release --example investigate_accept_socket_leak

It will list the open file descriptors to stdout for a few rounds and will finish with an error if file descriptor leaks were evident.

A number of errors got overlooked in #151 and earlier PRs targeting the criterion benchmarks. In particular, the throughput, concurrency, and iteration count values were all wrong.

This change addresses those issues.

Problem

A Rather awkward Builder API presented by tokio_uring::Builder.

Presently the tokio_uring::Builder (TUB) is a thin wrapper over the io_uring::Builder (IUB) but it serves a noble purpose, which is to allow a developer to tweak some settings without having to declare a dependency on io_uring

Focusing on *entries, the IUB allows one to set, as a builder method, the size of the Completion Queue, but not the Submission Queue. The SQ Entries value can only be supplied as a (required) parameter to the Builder::build() method. As a result, the TUB stores 1 parameter, the SQ entries, which it then supplies to the build() function. So really, the TUB isn't really a builder at all, but a convenience wrapper to store and supply the SQ Entries value to the IUB build invocation.

Setting the Completion Queue Size requires the construction of the inner builder:

    tokio_uring::builder()
        .entries(64)
        .uring_builder(
            tokio_uring::uring_builder()
            .setup_cqsize(1024)
        )
        .start(async {
            ...
        })

Options

1. We rename TUB to something other than "Builder" since it's sort of misleading and, at best, incomplete.
2. We alter the IUB API to be consistent with regards to its setup_parameters. Then wrap it with a higher-fidelity Builder API in tokio-uring.
   1. Add setup_sqsize to achieve parity with setup_cqsize .
   2. Give it an intelligent default.
   3. Remove the parameter in the io_uring::Builder::build() method, so that it takes its sqsize from its internal member instead.
   4. Update the TUB API to achieve parity with the IUB API (expose all relevant parameters)

The proposed new API would look something like this:

    tokio_uring::builder()
        .sq_entries(64)
        .cq_entries(1024)
        .iopoll(true)
        .start(async {
            ...
        })

This PR does ~3~ 2 things: ~1. Alters the Runtime crate::Builder API slightly to make it easier to correctly set sq and cq entry counts.~ 2. Updates method documentation to use indicative mood. This is not a huge change, as most public method docs were conformant. 3. Enhances other documentation in a few places for clarity.

This is definitely not in a ready-to-merge state, but I would like to open a conversation about integrating squeue::Entry128, into tokio-uring to unblock integration of io_uring methods that require squeue::Entry128. For context, I am working on implementing a rust-native ublk server library, and would like to make use of tokio-uring.