Incomplete Redis client and server implementation using Tokio - for learning purposes only

Related tags

Database mini-redis
Overview

mini-redis

mini-redis is an incomplete, idiomatic implementation of a Redis client and server built with Tokio.

The intent of this project is to provide a larger example of writing a Tokio application.

Disclaimer Please don't use mini-redis in production. This project is intended to be a learning resource, and omits various parts of the Redis protocol because implementing them would not introduce any new concepts. We will not add new features because you need them in your project — use one of the fully featured alternatives instead.

Why Redis

The primary goal of this project is teaching Tokio. Doing this requires a project with a wide range of features with a focus on implementation simplicity. Redis, an in-memory database, provides a wide range of features and uses a simple wire protocol. The wide range of features allows demonstrating many Tokio patterns in a "real world" context.

The Redis wire protocol documentation can be found here.

The set of commands Redis provides can be found here.

Running

The repository provides a server, client library, and some client executables for interacting with the server.

Start the server:

RUST_LOG=debug cargo run --bin mini-redis-server

The tracing crate is used to provide structured logs. You can substitute debug with the desired log level.

Then, in a different terminal window, the various client examples can be executed. For example:

cargo run --example hello_world

Additionally, a CLI client is provided to run arbitrary commands from the terminal. With the server running, the following works:

cargo run --bin mini-redis-cli set foo bar

cargo run --bin mini-redis-cli get foo

Supported commands

mini-redis currently supports the following commands.

The Redis wire protocol specification can be found here.

There is no support for persistence yet.

Tokio patterns

The project demonstrates a number of useful patterns, including:

TCP server

server.rs starts a TCP server that accepts connections, and spawns a new task per connection. It gracefully handles accept errors.

Client library

client.rs shows how to model an asynchronous client. The various capabilities are exposed as async methods.

State shared across sockets

The server maintains a Db instance that is accessible from all connected connections. The Db instance manages the key-value state as well as pub/sub capabilities.

Framing

connection.rs and frame.rs show how to idiomatically implement a wire protocol. The protocol is modeled using an intermediate representation, the Frame structure. Connection takes a TcpStream and exposes an API that sends and receives Frame values.

Graceful shutdown

The server implements graceful shutdown. tokio::signal is used to listen for a SIGINT. Once the signal is received, shutdown begins. The server stops accepting new connections. Existing connections are notified to shutdown gracefully. In-flight work is completed, and the connection is closed.

Concurrent connection limiting

The server uses a Semaphore limits the maximum number of concurrent connections. Once the limit is reached, the server stops accepting new connections until an existing one terminates.

Pub/Sub

The server implements non-trivial pub/sub capability. The client may subscribe to multiple channels and update its subscription at any time. The server implements this using one broadcast channel per channel and a StreamMap per connection. Clients are able to send subscription commands to the server to update the active subscriptions.

Using a std::sync::Mutex in an async application

The server uses a std::sync::Mutex and not a Tokio mutex to synchronize access to shared state. See db.rs for more details.

Testing asynchronous code that relies on time

In tests/server.rs, there are tests for key expiration. These tests depend on time passing. In order to make the tests deterministic, time is mocked out using Tokio's testing utilities.

Contributing

Contributions to mini-redis are welcome. Keep in mind, the goal of the project is not to reach feature parity with real Redis, but to demonstrate asynchronous Rust patterns with Tokio.

Commands or other features should only be added if doing so is useful to demonstrate a new pattern.

Contributions should come with extensive comments targetted to new Tokio users.

Contributions that only focus on clarifying and improving comments are very welcome.

License

This project is licensed under the MIT license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in mini-redis by you, shall be licensed as MIT, without any additional terms or conditions.

Comments
  • Apparent memory leak of spawned tasks

    Apparent memory leak of spawned tasks

    I believe there is some form of memory leak, and this was discussed some in the Discord channel #tokio-users recently. This graph should illustrate it pretty well:

    Memory leak graph

    The orange 6.2MB are a single allocation by tracing-subscriber, the green are RawTasks that are allocated via spawn, and the blue are something in broadcast. The workload is just running while true; do target/release/cli get k; done for a while (perhaps after setting a value for k, but I believe it doesn't actually matter).

    As far as I can tell, the spawned tasks have actually completed. A different implementation (using watch instead of broadcast) fixes it. Another project using broadcast does not have a similar issue. My hunch is that there is some kind of reference cycle, maybe in part because the shutdown coordination is bidirectional, but I have not looked more closely yet.

    opened by jebrosen 6
  • Update Cargo.toml to change `cli` and `server` binary names

    Update Cargo.toml to change `cli` and `server` binary names

    When you run the server like the README says, you get an error:

    $ cargo run --bin server
    
    error: no bin target named `server`
    

    Same happens with cli also.

    I believe this is happening cos how they are declared in the Cargo:

    [[bin]]
    name = "mini-redis-cli"
    path = "src/bin/cli.rs"
    
    [[bin]]
    name = "mini-redis-server"
    path = "src/bin/server.rs"
    

    This PR fixes this. Instead of editing README, I edited the Cargo itself cos I thought it will be easier to type server often than mini-redis-server

    opened by avinassh 4
  • Fix expiration and upppercase check

    Fix expiration and upppercase check

    • While constructing the frame, expiration is not checked: added a check and the additional bytes, i.e. PX ms.
    • When parsing from "EX" or "PX", it accepts only upper case, added to_uppercase() to the comparison.
    opened by gallir 3
  • use structopt instead of Clap

    use structopt instead of Clap

    mini-redis uses the CLI derive pattern. Clap does not yet have a release supporting this pattern. Using structopt allows mini-redis to avoid git dependencies.

    opened by carllerche 2
  • Add (optional) OpenTelemetry + Xray integration (#95)

    Add (optional) OpenTelemetry + Xray integration (#95)

    This introduces all the necessary code to be able to send traces to AWS Xray via tracing-opentelemetry. It can be optionally enabled using the xray feature defined on this crate.

    Also update the README.md with instructions on how to enable and use this.

    opened by bryangarza 1
  • Add (optional) OpenTelemetry + Xray integration

    Add (optional) OpenTelemetry + Xray integration

    This task is to set up the necessary code for shipping traces to AWS Xray (via OpenTelemetry). This will allow us to have a full example that we can point people to when they are looking to get started with Xray.

    This additional code should be compiled only conditionally, and not enabled as a default.

    opened by bryangarza 1
  • Fails with Latest Rust

    Fails with Latest Rust

    I am using Rust and Cargo 1.44.1. When installing mini-redis, I get a build error:

    error[E0658]: use of unstable library feature 'renamed_spin_loop'
     --> /Users/jahred-love/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.2/src/spinwait.rs:9:5
      |
    9 | use core::hint::spin_loop;
      |     ^^^^^^^^^^^^^^^^^^^^^
      |
      = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information
    
    error[E0658]: use of unstable library feature 'renamed_spin_loop'
      --> /Users/jahred-love/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.2/src/spinwait.rs:16:9
       |
    16 |         spin_loop()
       |         ^^^^^^^^^
       |
       = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information
    
       Compiling num_cpus v1.13.1
    error[E0658]: `match` is not allowed in a `const fn`
       --> /Users/jahred-love/.cargo/registry/src/github.com-1ecc6299db9ec823/socket2-0.4.4/src/lib.rs:165:9
        |
    165 | /         match address {
    166 | |             SocketAddr::V4(_) => Domain::IPV4,
    167 | |             SocketAddr::V6(_) => Domain::IPV6,
    168 | |         }
        | |_________^
        |
        = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
    
       Compiling atty v0.2.14
    error: aborting due to 2 previous errors
    
    For more information about this error, try `rustc --explain E0658`.
    error: could not compile `parking_lot_core`.
    
    To learn more, run the command again with --verbose.
    warning: build failed, waiting for other jobs to finish...
    error: aborting due to previous error
    
    For more information about this error, try `rustc --explain E0658`.
    error: could not compile `socket2`.
    
    To learn more, run the command again with --verbose.
    warning: build failed, waiting for other jobs to finish...
    error: failed to compile `mini-redis v0.4.1`, intermediate artifacts can be found at `/var/folders/pz/g8nj0j791bzb4_7qn53833v80000gp/T/cargo-installG3dzIj`
    
    Caused by:
      build failed
    
    opened by Lazarus404 1
  • support PING command for easy testing

    support PING command for easy testing

    Signed-off-by: tison [email protected]

    I'm unsure whether this repo accepts new features, but I meet this requirement when testing redis pipeline feature as

    (echo -e '*1\r\n$4\r\nPING\r\n*1\r\n$4\r\nPING\r\n*1\r\n$4\r\nPING\r\n'; sleep 1) | nc localhost 6379
    
    opened by tisonkun 1
  • Add blocking client

    Add blocking client

    This PR adds a simple blocking Redis client.

    Open questions:

    1. Does the client spawn any tasks, e.g. for periodic pings? If so, a different approach is needed.
    2. Should this be a separate crate?
    3. Should the module be called sync_client instead?

    This is introduced for the purpose of being the basis for an article in the tutorial about using async code in synchronous projects.

    opened by Darksonn 1
  • feat: adds trace events to server

    feat: adds trace events to server

    This PR adds tracing to the server which allows for some debugging.

    start server:

    $ RUST_LOG=debug cargo run --bin server
    

    set hi = there with default redis client

    $ redis-cli set hi there
    OK
    

    server logs:

    Mar 03 12:30:29.565 DEBUG mini_redis::cmd::set: parsed key: hi
    Mar 03 12:30:29.566 DEBUG mini_redis::cmd::set: parsed value: b"there"
    Mar 03 12:30:29.566 DEBUG mini_redis::cmd::set: parsed expiration: None
    Mar 03 12:30:29.566 DEBUG mini_redis::cmd::set: writing frame: Simple("OK")
    

    get value of hi with default redis client

    $ redis-cli get hi
    "there"
    

    server logs:

    Mar 03 12:30:29.572 DEBUG mini_redis::cmd::get: parsed key: hi
    Mar 03 12:30:29.572 DEBUG mini_redis::cmd::get: writing frame: Bulk(b"there")
    
    opened by EverlastingBugstopper 1
  • Fix tiny typo in server.rs

    Fix tiny typo in server.rs

    Thank you for awesome project! I'm learning a lot from this project about async processing in rust.

    I found tiny typo in the run function in server.rs and fixed it.

    opened by shomitarai 0
  • Potential bug in read_frame if we receive 2 frames ready in the buffer?

    Potential bug in read_frame if we receive 2 frames ready in the buffer?

    https://github.com/tokio-rs/mini-redis/blob/c1c3bcb4657ebaba8665e2f404a995be0d93cb26/src/connection.rs#L56

    If we receive 2 complete frames within a single read, we will read the first one and then exit the loop immediately.

    opened by hbina 1
  • add rustfmt.toml && simplify PartialEq on str for Frame

    add rustfmt.toml && simplify PartialEq on str for Frame

    An empty rustfmt.toml in project tells rustfmt to use the default rustfmt settings, which won't be affected by global rustfmt config (like in ~/.config/rustfmt/rustfmt.toml).

    opened by zjp-CN 0
  • The `chat` example is empty

    The `chat` example is empty

    There is a chat.rs example file in the repository but it is empty:

    #[tokio::main]
    async fn main() {
        unimplemented!();
    }
    

    Why put an empty example file that contains nothing (genuine question)?

    opened by ClementNerma 2
  • Add additional `tracing` instrumentation.

    Add additional `tracing` instrumentation.

    This commit lays the ground-work for making mini-redis a better example of an instrumented tokio application. While it does not go so far to turn mini-redis into a fully-featured guide for using tracing, it ensures that people who use mini-redis as a basis for experimenting with different subscribers get a more complete experience out-of-the-box.

    E.g., with tracing-tree a tracing-tree subscriber and level=Debug:

     INFO mini_redis::server accepting inbound connections
    ┐mini_redis::db::purge_expired_tasks 
    ├─┐mini_redis::db::Shared::purge_expired_keys 
    ├─┘
    ┐mini_redis::server::Handler::run peer_addr=127.0.0.1:56574
    ├─┐mini_redis::server::Handler::process_frame 
    │ ├─0ms DEBUG mini_redis::server cmd=Unknown(Unknown { command_name: "config" })
    │ ├─0ms DEBUG mini_redis::cmd::unknown response=Error("ERR unknown command 'config'")
    ├─┘
    ├─┐mini_redis::server::Handler::process_frame 
    │ ├─0ms DEBUG mini_redis::server cmd=Unknown(Unknown { command_name: "config" })
    │ ├─0ms DEBUG mini_redis::cmd::unknown response=Error("ERR unknown command 'config'")
    ├─┘
    ├─┐mini_redis::server::Handler::process_frame 
    │ ├─0ms  WARN mini_redis::connection connection closed abruptly by peer
    ├─┘
    ┘
    ┐mini_redis::server::Handler::run peer_addr=127.0.0.1:56576
    ├─┐mini_redis::server::Handler::process_frame 
    │ ├─0ms DEBUG mini_redis::server cmd=Set(Set { key: "key:000000000027", value: b"VXK", expire: None })
    ├─┘
    ├─┐mini_redis::server::Handler::process_frame 
    │ ├─0ms DEBUG mini_redis::server cmd=Set(Set { key: "key:000000000001", value: b"VXK", expire: None })
    ├─┘
    ├─┐mini_redis::server::Handler::process_frame 
    ├─┘
    ┘
    

    This PR also reduces ConnectionReset 'errors' to warnings, as they do not reflect errors in mini-redis, but rather client misbehavior. (And you get them frequently when using redis-benchmark.)

    opened by jswrenn 4
Owner
Tokio
Rust's asynchronous runtime.
Tokio
📺 Netflix in Rust/ React-TS/ NextJS, Actix-Web, Async Apollo-GraphQl, Cassandra/ ScyllaDB, Async SQLx, Kafka, Redis, Tokio, Actix, Elasticsearch, Influxdb Iox, Tensorflow, AWS

Fullstack Movie Streaming Platform ?? Netflix in RUST/ NextJS, Actix-Web, Async Apollo-GraphQl, Cassandra/ ScyllaDB, Async SQLx, Spark, Kafka, Redis,

null 34 Apr 17, 2023
Simple crate that wraps a tokio::process into a tokio::stream

tokio-process-stream tokio-process-stream is a simple crate that wraps a tokio::process into a tokio::stream Having a stream interface to processes is

Leandro Lisboa Penz 8 Sep 13, 2022
Rewrite Redis in Rust for evaluation and learning.

Drill-Redis This library has been created for the purpose of evaluating Rust functionality and performance. As such, it has not been fully tested. The

Akira Kawahara 3 Oct 18, 2022
An intentionally-limited Rust implementation of the Redis server with no external dependencies.

lil-redis An intentionally-limited Rust implementation of the Redis server. lil redis is an accessible implementation of a very basic Redis server (wi

Miguel Piedrafita 37 Jan 1, 2023
Lightweight async Redis client with connection pooling written in pure Rust and 100% memory safe

redi-rs (or redirs) redi-rs is a Lightweight Redis client with connection pooling written in Rust and 100% memory safe redi-rs is a Redis client writt

Oğuz Türkay 4 May 20, 2023
Redis compatible server framework for Rust

Redis compatible server framework for Rust Features Create a fast custom Redis compatible server in Rust Simple API. Support for pipelining and telnet

Josh Baker 61 Nov 12, 2022
A simplified version of a Redis server supporting SET/GET commands

This is a starting point for Rust solutions to the "Build Your Own Redis" Challenge. In this challenge, you'll build a toy Redis clone that's capable

Patrick Neilson 2 Nov 15, 2022
A cross-platform redis gui client

Redis-Manager A cross-platform redis gui client started developing with Tauri, React and Typescript in Vite. Get Started Prerequisites Install Node.js

Kurisu 11 Mar 31, 2023
Simple and flexible queue implementation for Rust with support for multiple backends (Redis, RabbitMQ, SQS, etc.)

Omniqueue Omniqueue is an abstraction layer over queue backends for Rust. It includes support for RabbitMQ, Redis streams, and SQS out of the box. The

Svix 8 May 26, 2023
A pure Rust database implementation using an append-only B-Tree file format.

nebari nebari - noun - the surface roots that flare out from the base of a bonsai tree Warning: This crate is early in development. The format of the

Khonsu Labs 194 Jan 3, 2023
CRUD system of book-management with ORM and JWT for educational purposes.

Book management English | 中文 Required Rust MySQL 5.7 Usage Execute init.sql to create tables. Set environment variable DATABASE_URL and JWT_SECRET in

null 32 Dec 28, 2022
RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

Qovery 145 Nov 23, 2022
Macros for redis-rs to serialize and deserialize structs automatically

redis-macros Simple macros and wrappers to redis-rs to automatically serialize and deserialize structs with serde. Installation To install it, simply

Daniel Grant 4 Feb 18, 2023
Redis re-implemented in Rust.

rsedis Redis re-implemented in Rust. Why? To learn Rust. Use Cases rsedis does not rely on UNIX-specific features. Windows users can run it as a repla

Sebastian Waisbrot 1.6k Jan 6, 2023
Redis library for rust

redis-rs Redis-rs is a high level redis library for Rust. It provides convenient access to all Redis functionality through a very flexible but low-lev

Armin Ronacher 2.8k Jan 8, 2023
RedisJSON - a JSON data type for Redis

RedisJSON RedisJSON is a Redis module that implements ECMA-404 The JSON Data Interchange Standard as a native data type. It allows storing, updating a

null 3.4k Jan 1, 2023
A rust Key-Value store based on Redis.

Key-Value Store A Key-Value store that uses Redis to store data. Built using an async web framework in Rust with a full Command-Line interface and log

Miguel David Salcedo 0 Jan 14, 2022
[POC] Redis Module for TiKV

RedisTikvPoc [POC] Redis Module for TiKV This is a POC repository. This Redis Module will add branch new Redis commands to operate TiKV data. After bu

Rain Li 6 Jun 25, 2022
Basic Redis Protocol specification in Rust

Basic Redis Protocol specification in Rust

Bruno 1 Jan 20, 2022