RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

Overview

RedisLess

Discord Test and Build


THIS PROJECT IS TESTABLE, BUT NOT PRODUCTION READY YET!!


RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

RedisLess is the concatenation of Redis and Serverless.

Motivation

As a backend developer, I mostly work on providing web API for frontend developers and backend developers. Redis is part of my toolset - especially when I share data and sync states between my backend apps. For those simple use cases, I don't want to have to spawn a Redis server.

So, imagine a world where your Redis instance is nothing more than a lib in your app. Imagine that this lib can sync the data with other neighborhood instances?

That's the idea behind RedisLess, a lightweight, embedded, and scalable in-memory Key/Value store library compatible with the Redis API.

Follow me on Twitter 🙋‍♂️

Read more about RedisLess:

How to use it?

To use RedisLess, you only need to:

  1. Install RedisLess library for your favorite language (see supported clients below).
  2. Connect your favorite Redis client to redis://localhost:16379.
  3. You don't need to change your code - RedisLess is Redis API compatible. (see supported Redis commands)

Under the hood, the RedisLess library starts a local Redis API compatible instance on port 16739 (you can change the port).

NodeJS client

Install

# RedisLess library with Python binding
npm install redisless

# redis client
npm install async-redis

Usage

const {RedisLess} = require('redisless');
const redis = require('async-redis');

const port = 16379;
const redisless = new RedisLess(port);

redisless.start();

const r = redis.createClient({host: 'localhost', port: port});

await r.set('foo', 'bar');
await r.get('foo'); // return 'bar'
await r.del('foo'); // delete key 'foo'

redisless.stop();

Python client

Install

# RedisLess library with Python binding
pip install redisless

# redis client
pip install redis

Usage

from redisless import RedisLess
import redis

redisless = RedisLess()

# start RedisLess embedded instance
redisless.start()

# Connect to RedisLess on localhost:16379
redis = redis.Redis(host='localhost', port=16379)

redis.set('foo', 'bar')
redis.get('foo')  # return bar 
redis.delete('foo')  # return 1 

# stop RedisLess embedded instance
redisless.stop()

What is RedisLess

  • Embedded in-memory. (No Redis server required!).
  • Compatible with the Redis API (RESP).
  • Not intrusive: you don't need to modify you code to use RedisLess!
  • Built with DX and performance in mind.
  • Cloud native friendly.

What RedisLess is not:

  • Production ready yet.
  • 1:1 Redis implementation.

Use cases

The goal of RedisLess is not to remove the need of Redis server for heavy workload. If you consider to store more than 2 GB of data in RedisLess you are better to use Redis. There is no hard limit, but RedisLess is designed to manage and share small dataset between apps.

Good use cases:

  • Distributed lock between your apps.
  • Share data (lower than 2 GB) between your apps.
  • Cache where reads are more important than writes.

Bad use cases:

  • Store more than 2 GB of data.
  • Cache where writes are more important than reads.

Planned features

Planned supported clients

Expected performance

Strong attention to performance and code cleanliness is given when designing RedisLess. It aims to be crash-free, super-fast and put a minimum strain on your server resources.

RedisLess is written in Rust and export functions through FFI (Foreign Function Interface), making it usable from any language. We provide clients for NodeJS, Python, Golang, Java, and Rust. Supporting a new language can be done in 5 minutes. Look at Python and NodeJS clients implementation for inspiration.

Contribution welcome!

It is never too soon to contribute to a great project. If you are interested in contributing, please join us on Discord, then we can discuss. The project is in its early days, but we are serious about building a solid library to help thousands of developers.

Set up MacOSX development environment

Pre-requisites

  1. Install Xcode
  2. Install Brew
  3. Install Rust
  4. Run brew install mingw-w64 to cross build for Windows
  5. Run brew install FiloSottile/musl-cross/musl-cross to cross build for Linux
  6. Run brew tap SergioBenitez/osxct && brew install x86_64-unknown-linux-gnu # to cross build for Linux

Build clients

To build NodeJS, Python, Golang and other libs you need to run:

cd scripts && ./build-for-mac.sh && cd ..

Use clients

Once the libs built, you are ready to use the clients into the clients folder.

Contributors

Thanks to our contributors ❤️

References

Comments
  • draft: add TYPE command

    draft: add TYPE command

    How do you test a new command? I'm not sure if I'm already on the right direction, i.e., if I only need to figure out what to feed send_packed_command() with, and how to receive the connection's response.

    #28

    opened by tbmreza 11
  • Support command

    Support command "TTL"

    Returns the remaining time to live of a key that has a timeout. This introspection capability allows a Redis client to check how many seconds a given key will continue to be part of the dataset.

    In Redis 2.6 or older the command returns -1 if the key does not exist or if the key exist but has no associated expire.

    Starting with Redis 2.8 the return value in case of error changed:

    The command returns -2 if the key does not exist.
    The command returns -1 if the key exists but has no associated expire.
    
    good first issue 
    opened by pjeziorowski 10
  • TestConnection struct

    TestConnection struct

    My follow-up to #55. I ended up implementing a different approach. When writing the example in that issue, I didn't know that you can't annotate the type of vector of closures.

    I believe this draft PR is concrete enough to express my idea. What's left is to write current test cases in new format (as rstest attributes).

    opened by tbmreza 2
  • PEXPIRE command support

    PEXPIRE command support

    PEXPIRE DOC


    Also did some tweaking to Expiry to just hold a timestamp

    We also probably want to change parse to return a Result to avoid indentation hell

    opened by clarity0 2
  • Support TTL

    Support TTL

    Redis keys are expired in two ways: a passive way, and an active way.

    A key is passively expired simply when some client tries to access it, and the key is found to be timed out.

    Of course this is not enough as there are expired keys that will never be accessed again. These keys should be expired anyway, so periodically Redis tests a few keys at random among keys with an expire set. All the keys that are already expired are deleted from the keyspace.

    Specifically this is what Redis does 10 times per second:

    Test 20 random keys from the set of keys with an associated expire.
    Delete all the keys found expired.
    If more than 25% of keys were expired, start again from step 1.
    

    This is a trivial probabilistic algorithm, basically the assumption is that our sample is representative of the whole key space, and we continue to expire until the percentage of keys that are likely to be expired is under 25%

    This means that at any given moment the maximum amount of keys already expired that are using memory is at max equal to max amount of write operations per second divided by 4.

    opened by pjeziorowski 1
  • Proposal for resolving clippy warnings

    Proposal for resolving clippy warnings

    I've decided to start to resolve this first on the present project so you can see if you like it.

    It seems that in redisless case there wasn't much really important, it's mostly suggestions for idiomatic rust

    ~I am not sure why clippy complains about imports on src/cluster/tests.rs file, if there is some sort of a bug or anything I am missing in the compilation process. For now I have just included the ifcfg crate.~ Well, I shouldn't work that late, it was just missing the #[cfg(test)] annotation on the module.

    A few #[allow(dead_code)] here and there since the repo is WIP...

    opened by Dalot 1
  • Refactor server::tests

    Refactor server::tests

    context: https://github.com/Qovery/RedisLess/blob/main/redisless/src/server/tests/mod.rs

    I have been trying rstest crate that I just learned from another project. I managed to get the following test passes:

    #[rstest]
    #[case::incr_by_1(
        3001_u16,
        vec![ 
            // precondition:
            con_set(23, 45),
            // test subject:
            con_incr(23, 1),
        ],
        vec![ 
            assert_con_exists(23, true),
            assert_con_get(23, 46),
        ]
    )]
    fn redis_client(
        #[case] port: u16,
        #[case] commands: Vec<Box<dyn for<'r> Fn(&'r mut Connection)>>,
        #[case] assertions: Vec<Box<dyn for<'r> Fn(&'r mut Connection)>>,
    ){
        let (server, mut con) = get_server_connection(port);
        commands.iter().for_each(|f| f(&mut con));
        assertions.iter().for_each(|f| f(&mut con));
        assert_eq!(server.stop(), Some(ServerState::Stopped));
    }
    
    

    To declare, for example, a delete command test case, insert the following attribute:

    #[case::del_existent_key(
        3002_u16,
        vec![ 
            // precondition:
            con_set(23, 45),
            // test subject:
            con_del(23),
        ],
        vec![ 
            assert_con_exists(23, false),
        ]
    )]
    

    If anyone thinks this interface will get a long way, I'll be happy to line up a pull request.

    opened by tbmreza 0
  • garbage collection

    garbage collection

    still WIP

    1. This PR strives to change the API of Storage to be more user-friendly. Specifically, read+hread to be read-only (taking &self instead of &mut self). This is to clear the ground to compose them with other commands. Specifically, COPY (#25) needs to call both read and write on the Storage and cannot have multiple mutable references.
    2. there are 2 parts for this PR.
    • first, read commands merely marks the values with expired TTL with a "tombstone" in an atomic, immutable way. this relaxes the &mut self requirement
    • second, --- not implemented yet--- a GC worker is spawned. This thread should work the same as original Redis GC algorithm works see here.

    Note about the 2nd stage. One could argue that deleting keys in a lazy fashion is not enough as it could lead to memory bloat and rightly so! However, A GC thread is needed anyway since expired keys need to be evicted even if they are not visited.

    opened by barkanido 2
  • List of Redis commands supported by RedisLess

    List of Redis commands supported by RedisLess

    • [x] append
    • [x] dbsize
    • [x] del
    • [x] exists
    • [x] expire
    • [x] get
    • [x] getset
    • [x] hget
    • [x] hmset
    • [x] hset
    • [x] incr
    • [x] incrby
    • [x] info
    • [x] mget
    • [x] mset
    • [x] msetnx
    • [x] pexpire
    • [x] ping
    • [x] psetex
    • [x] set
    • [x] setex
    • [x] setnx
    • [ ] acl
    • [ ] asking
    • [ ] auth
    • [ ] bgrewriteaof
    • [ ] bgsave
    • [ ] bitcount
    • [ ] bitfield
    • [ ] bitfield_ro
    • [ ] bitop
    • [ ] bitpos
    • [ ] blmove
    • [ ] blpop
    • [ ] brpop
    • [ ] brpoplpush
    • [ ] bzpopmax
    • [ ] bzpopmin
    • [ ] client
    • [ ] cluster
    • [ ] command
    • [ ] config
    • [ ] copy
    • [ ] crc64
    • [ ] debug
    • [ ] decr
    • [ ] decrby
    • [ ] dict
    • [ ] discard
    • [ ] dump
    • [ ] echo
    • [ ] endianconv
    • [ ] eval
    • [ ] eval_ro
    • [ ] evalsha
    • [ ] evalsha_ro
    • [ ] exec
    • [ ] expireat
    • [ ] failover
    • [ ] flushall
    • [ ] flushdb
    • [ ] geoadd
    • [ ] geodist
    • [ ] geohash
    • [ ] geopos
    • [ ] georadius
    • [ ] georadiusbymember
    • [ ] georadiusbymember_ro
    • [ ] georadius_ro
    • [ ] geosearch
    • [ ] geosearchstore
    • [ ] getbit
    • [ ] getdel
    • [ ] getex
    • [ ] getrange
    • [ ] hdel
    • [ ] hello
    • [ ] hexists
    • [ ] hgetall
    • [ ] hincrby
    • [ ] hincrbyfloat
    • [ ] hkeys
    • [ ] hlen
    • [ ] hmget
    • [ ] host:
    • [ ] hrandfield
    • [ ] hscan
    • [ ] hsetnx
    • [ ] hstrlen
    • [ ] hvals
    • [ ] incrbyfloat
    • [ ] intset
    • [ ] keys
    • [ ] lastsave
    • [ ] latency
    • [ ] lindex
    • [ ] linsert
    • [ ] llen
    • [ ] lmove
    • [ ] lolwut
    • [ ] lpop
    • [ ] lpos
    • [ ] lpush
    • [ ] lpushx
    • [ ] lrange
    • [ ] lrem
    • [ ] lset
    • [ ] ltrim
    • [ ] memory
    • [ ] migrate
    • [ ] module
    • [ ] monitor
    • [ ] move
    • [ ] multi
    • [ ] object
    • [ ] persist
    • [ ] pexpireat
    • [ ] pfadd
    • [ ] pfcount
    • [ ] pfdebug
    • [ ] pfmerge
    • [ ] pfselftest
    • [ ] post
    • [ ] psubscribe
    • [ ] psync
    • [ ] pttl
    • [ ] publish
    • [ ] pubsub
    • [ ] punsubscribe
    • [ ] quicklist
    • [ ] randomkey
    • [ ] readonly
    • [ ] readwrite
    • [ ] rename
    • [ ] renamenx
    • [ ] replconf
    • [ ] replicaof
    • [ ] reset
    • [ ] restore
    • [ ] restore-asking
    • [ ] role
    • [ ] rpop
    • [ ] rpoplpush
    • [ ] rpush
    • [ ] rpushx
    • [ ] sadd
    • [ ] save
    • [ ] scan
    • [ ] scard
    • [ ] script
    • [ ] sdiff
    • [ ] sdiffstore
    • [ ] sds
    • [ ] select
    • [ ] setbit
    • [ ] setrange
    • [ ] sha1test
    • [ ] shutdown
    • [ ] sinter
    • [ ] sinterstore
    • [ ] sismember
    • [ ] slaveof
    • [ ] slowlog
    • [ ] smembers
    • [ ] smismember
    • [ ] smove
    • [ ] sort
    • [ ] spop
    • [ ] srandmember
    • [ ] srem
    • [ ] sscan
    • [ ] stralgo
    • [ ] strlen
    • [ ] subscribe
    • [ ] substr
    • [ ] sunion
    • [ ] sunionstore
    • [ ] swapdb
    • [ ] sync
    • [ ] time
    • [ ] touch
    • [ ] ttl
    • [ ] type
    • [ ] unlink
    • [ ] unsubscribe
    • [ ] unwatch
    • [ ] util
    • [ ] wait
    • [ ] watch
    • [ ] xack
    • [ ] xadd
    • [ ] xautoclaim
    • [ ] xclaim
    • [ ] xdel
    • [ ] xgroup
    • [ ] xinfo
    • [ ] xlen
    • [ ] xpending
    • [ ] xrange
    • [ ] xread
    • [ ] xreadgroup
    • [ ] xrevrange
    • [ ] xsetid
    • [ ] xtrim
    • [ ] zadd
    • [ ] zcard
    • [ ] zcount
    • [ ] zdiff
    • [ ] zdiffstore
    • [ ] zincrby
    • [ ] zinter
    • [ ] zinterstore
    • [ ] ziplist
    • [ ] zipmap
    • [ ] zlexcount
    • [ ] zmalloc
    • [ ] zmscore
    • [ ] zpopmax
    • [ ] zpopmin
    • [ ] zrandmember
    • [ ] zrange
    • [ ] zrangebylex
    • [ ] zrangebyscore
    • [ ] zrangestore
    • [ ] zrank
    • [ ] zrem
    • [ ] zremrangebylex
    • [ ] zremrangebyrank
    • [ ] zremrangebyscore
    • [ ] zrevrange
    • [ ] zrevrangebylex
    • [ ] zrevrangebyscore
    • [ ] zrevrank
    • [ ] zscan
    • [ ] zscore
    • [ ] zunion
    • [ ] zunionstore
    opened by clarity0 1
Owner
Qovery
Qovery - The simplest way to deploy your apps in the Cloud
Qovery
A simple embedded key-value store written in rust as a learning project

A simple embedded key-value store written in rust as a learning project

Blobcode 1 Feb 20, 2022
General basic key-value structs for Key-Value based storages

General basic key-value structs for Key-Value based storages

Al Liu 0 Dec 3, 2022
PickleDB-rs is a lightweight and simple key-value store. It is a Rust version for Python's PickleDB

PickleDB PickleDB is a lightweight and simple key-value store written in Rust, heavily inspired by Python's PickleDB PickleDB is fun and easy to use u

null 155 Jan 5, 2023
An embedded, in-memory, immutable, copy-on-write, key-value database engine

An embedded, in-memory, immutable, copy-on-write, key-value database engine. Features In-memory database Multi-version concurrency control Rich transa

SurrealDB 7 May 5, 2024
A fast and simple in-memory database with a key-value data model written in Rust

Segment Segment is a simple & fast in-memory database with a key-value data model written in Rust. Features Dynamic keyspaces Keyspace level control o

Segment 61 Jan 5, 2023
Pure rust embeddable key-value store database.

MHdb is a pure Rust database implementation, based on dbm. See crate documentation. Changelog v1.0.3 Update Cargo.toml v1.0.2 Update Cargo.toml v1.0.1

Magnus Hirth 7 Dec 10, 2022
RefineDB - A strongly-typed document database that runs on any transactional key-value store.

RefineDB - A strongly-typed document database that runs on any transactional key-value store.

Heyang Zhou 375 Jan 4, 2023
Log structured append-only key-value store from Rust In Action with some enhancements.

riakv Log structured, append only, key value store implementation from Rust In Action with some enhancements. Features Persistent key value store with

Arindam Das 5 Oct 29, 2022
A LSM-based Key-Value Store in Rust

CobbleDB A LSM-based Key-Value Store in Rust Motivation There is no open-source LSM-based key-value store in Rust natively. Some crates are either a w

Yizheng Jiao 2 Oct 25, 2021
A sessioned Merkle key/value store

storage A sessioned Merkle key/value store The crate was designed to be the blockchain state database. It provides persistent storage layer for key-va

Findora Foundation 15 Oct 25, 2022
🔥🌲 High-performance Merkle key/value store

merk High-performance Merkle key/value store Merk is a crypto key/value store - more specifically, it's a Merkle AVL tree built on top of RocksDB (Fac

Nomic 189 Dec 13, 2022
Lightweight async Redis client with connection pooling written in pure Rust and 100% memory safe

redi-rs (or redirs) redi-rs is a Lightweight Redis client with connection pooling written in Rust and 100% memory safe redi-rs is a Redis client writt

Oğuz Türkay 4 May 20, 2023
Garage is a lightweight S3-compatible distributed object store

Garage [ Website and documentation | Binary releases | Git repository | Matrix channel ] Garage is a lightweight S3-compatible distributed object stor

Deuxfleurs 156 Dec 30, 2022
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
A Redis module that provides rate limiting in Redis as a single command.

redis-cell A Redis module that provides rate limiting in Redis as a single command. Implements the fairly sophisticated generic cell rate algorithm (G

Brandur Leach 1.1k Jan 6, 2023
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem

Google 31.5k Jan 1, 2023
AgateDB is an embeddable, persistent and fast key-value (KV) database written in pure Rust

AgateDB is an embeddable, persistent and fast key-value (KV) database written in pure Rust. It is designed as an experimental engine for the TiKV project, and will bring aggressive optimizations for TiKV specifically.

TiKV Project 535 Jan 9, 2023
ForestDB - A Fast Key-Value Storage Engine Based on Hierarchical B+-Tree Trie

ForestDB is a key-value storage engine developed by Couchbase Caching and Storage Team, and its main index structure is built from Hierarchic

null 1.2k Dec 26, 2022
A "blazingly" fast key-value pair database without bloat written in rust

A fast key-value pair in memory database. With a very simple and fast API. At Xiler it gets used to store and manage client sessions throughout the pl

Arthur 16 Dec 16, 2022