Rust implementation of reactor pattern for I/O resources

Related tags

Database io-reactor
Overview

I/O reactor

Build Tests Lints codecov

crates.io Docs Apache-2 licensed

Concurrent I/O without rust async problems

This repository provides a set of libraries for concurrent access to I/O resources (file, network, devices etc) which uses reactor pattern with pluggable poll syscall engines. This allows to handle situations like multiple network connections within the scope of a single thread (see C10k problem).

The crate can be used for building concurrent microservice architectures without polling all APIs with asyncs.

The reactor design pattern is an event handling pattern for handling service requests delivered concurrently to a service handler by one or more inputs. The service handler then demultiplexes the incoming requests and dispatches them synchronously to the associated service1.

Core concepts:

  • Resource: any resource that can provide input to or consume output from the system.
  • Runtime: runs an event loop to block on all resources. Sends the resource service when it is possible to start a synchronous operation on a resource without blocking (Example: a synchronous call to read() will block if there is no data to read.
  • Service: custom business logic provided by the application for a given set of resources. Service exposes a Handler API for the reactor runtime. It is also rsponsible for the creation and destruction of the resources.

Documentation

API reference documentation for the library can be accessed at https://docs.rs/io-reactor/.

Licensing

The libraries are distributed on the terms of Apache 2.0 opensource license. See LICENCE file for the license details.

Footnotes

  1. Schmidt, Douglas et al. Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects. Volume 2. Wiley, 2000.

You might also like...
Go like sync.WaitGroup implementation in Rust. (sync/async)

WAG Go like sync.WaitGroup implementation in Rust. (sync/async) | Examples | Docs | Latest Note | wag = "0.3.0" How to use, use wag::WaitGroup; let w

An intentionally-limited Rust implementation of the Redis server with no external dependencies.

lil-redis An intentionally-limited Rust implementation of the Redis server. lil redis is an accessible implementation of a very basic Redis server (wi

Pure Rust implementation of Arbitrum sequencer feed reader with built-in transaction decoding and MEV features

Sequencer-Client (WIP 🚧 ) Pure Rust implementation of Arbitrum sequencer feed reader with built-in transaction decoding and MEV features Design Goal

Simple and flexible queue implementation for Rust with support for multiple backends (Redis, RabbitMQ, SQS, etc.)

Omniqueue Omniqueue is an abstraction layer over queue backends for Rust. It includes support for RabbitMQ, Redis streams, and SQS out of the box. The

Another minimal Raft implementation in Rust.

raft-rs Not my first time implementing Raft. I wrote about another implementation in Go I did. But you don't learn a concept well until you've impleme

Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol).

Ratchet 🚧 Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol). Complete with an optional

Incomplete Redis client and server implementation using Tokio - for learning purposes only

mini-redis mini-redis is an incomplete, idiomatic implementation of a Redis client and server built with Tokio. The intent of this project is to provi

Diesel async connection implementation

A async interface for diesel Diesel gets rid of the boilerplate for database interaction and eliminates runtime errors without sacrificing performance

🦀 REST API client implementation for freee, auto-generated from OpenAPI specification.

freee-rs REST API client implementation for freee, auto-generated from OpenAPI specification. Getting Started Add to your Cargo.toml as follows: [depe

Comments
  • Tidy up error variants

    Tidy up error variants

    There are currently too many variants in reactor::Error, which makes the crate very hard to use, as it's unclear how each of these errors should be handled. Ideally most of the variants have clear actionables, eg. unregistering the socket. Here's what I'd start with.


    These errors come from improper use of the API by the user, and shouldn't be reactor errors. If we're trying to unregister an unknown peer, not returning an error makes unregistering idempotent, which makes the API easier to use. If we're trying to write to an unknown peer, we could return a WriteFailure error instead. Since they aren't fatal and can easily happen due to race conditions, I'd simply log an error and remove the variants:

    reactor::Error::ListenerUnknown(id)
    reactor::Error::TransportUnknown(id)
    reactor::Error::WriteLogicError(id, _)
    

    These are pretty critical and it's not clear how they can happen, but if our listener socket goes down, we probably want to restart the node. I would probably merge the two variants into Error::ListenerFailed, since there isn't really a difference in how they should be handled.

    reactor::Error::ListenerPollError(id, err)
    reactor::Error::ListenerDisconnect(id, _, err)
    

    This error should be whatever the internal Poll implementation returns. Ideally it can be downcasted to the concrete error type. Poll errors should be fatal, so we'd probably abort here.

    reactor::Error::Poll(err)
    

    I'm guessing this one is a clean disconnect, in which case we can handle it and unregister the socket:

    reactor::Error::TransportDisconnect(id, _, err)
    

    These two seem like they can be bundled under Error::Transport, since they'd likely both be handled by disconnecting:

    reactor::Error::TransportPollError(id, err)
    reactor::Error::WriteFailure(id, err)
    
    enhancement 
    opened by cloudhead 1
Releases(v0.1.0)
  • v0.1.0(Jan 23, 2023)

    Concurrent I/O without rust async problems

    This is the first release of a library implementing reactor pattern, which allows concurrent I/O and new types of simple and yet powerful microservice-based applications in rust, without polluting all APIs with asyncs.

    This initial release provides:

    • fully-documented reactor API;
    • support for both listener-type resources (able to spawn more resources, like TcpListeners) and transport-style resources;
    • timer management;
    • multi-threaded controller for the reactor;
    • works with any resource which can be represented as a file descriptor (files, POSIX sockets, pipes, TCP, UDP and other network connections, database handlers, devices and periphery);
    • easy integration with multi-layer network protocols (like SOCKS5 proxy combined with end-to-end encryption);
    • popol poll engine by @cloudhead.

    The creation of the library was inspired by @cloudhead work on nakamoto reactor.

    API reference documentation for the library can be accessed at https://docs.rs/io-reactor/.

    Source code(tar.gz)
    Source code(zip)
Owner
Rust Amplified
Set of lightweight dependency-minimal, derive macros & no-std crates amplifying rust capabilities
Rust Amplified
Awesome books, tutorials, courses, and resources for the Tokio asynchronous runtime ecosystem. ⚡

Awesome Tokio Tokio is an asynchronous runtime for the Rust programming language. It provides the building blocks needed for writing network applicati

Marcus Cvjeticanin 59 Oct 27, 2023
rinflux is Rust based influx client implementation that have been inspired from influx other language implementation, developed with 💖

Unofficial InfluxDB Driver for Rust This library is a work in progress. This means a feature you might need is not implemented yet or could be handled

Workfoxes 1 Apr 7, 2022
Rust implementation of Hyperswarm, a networking stack for connecting peers

hyperswarm-rs Peer to peer networking stack API Docs | Contributing NOTE: This is still in early stages. See the roadmap below. Please feel free to op

Franz Heinzmann 25 Nov 22, 2022
A pure Rust database implementation using an append-only B-Tree file format.

nebari nebari - noun - the surface roots that flare out from the base of a bonsai tree Warning: This crate is early in development. The format of the

Khonsu Labs 194 Jan 3, 2023
A firebase HTTP v1 client implementation in Rust using google-authz

firebase-client A firebase HTTP v1 client implementation in Rust using the google_authz library. Example There are two ways to send notifications, one

Mobiltracker 2 Dec 14, 2022
The spatial message broker and database for real-time multiplayer experiences. Official Rust implementation.

WorldQL Server Rust implementation of WorldQL, the spatial message broker and database for real-time multiplayer experiences Setup Instructions ⚠️ Thi

null 214 Jan 2, 2023
Dataloader-rs - Rust implementation of Facebook's DataLoader using async-await.

Dataloader Rust implementation of Facebook's DataLoader using async-await. Documentation Features Batching load requests with caching Batching load re

cksac 229 Nov 27, 2022
An implementation of the tz database for the time-rs Rust crate.

time-tz An implementation of the tz database for the time-rs Rust crate. This implementation is based off of chrono-tz

null 12 Jul 27, 2022
HTTP 2.0 client & server implementation for Rust.

H2 A Tokio aware, HTTP/2 client & server implementation for Rust. More information about this crate can be found in the crate documentation. Features

null 1.1k Dec 30, 2022
A mini kv database demo that using simplified bitcask storage model with rust implementation

A mini kv database demo that using simplified bitcask storage model with rust implementation.

Wancheng Long 17 Nov 28, 2022