ConstDB - an in-memory cache store which aims at master-master replications

Related tags

Caching ConstDB
Overview

Introduction

ConstDB is an in-memory cache store which aims at master-master replications. A group of ConstDB instances can be deployed across many regions, and each of them is available for reading and writing. Every instance keeps in touch with all other ones. When a write command has been executed in one instance, it is asynchronously replicated to all the other instances. The data structures that are used for storing the client's input messages implement CRDT(Conflict-Free Replicated Data Types), thus we are sure data in different instances are eventually consistent.
ConstDB is compatible with redis protocol, and I have implemented a portion of redis's data types and commands. I'll keep developing it and take as much features in redis as possible into ConstDB. Contribution is also welcomed.

Features:

  • CRDT: Each write command is associated with an increasing 64-bit uuid and an unique node id, therefore the synchronization is idempotent and loopback is avoided. The data stored in cache are also kept with a timestamp at which time they are generated(actually we use the uuid as a timestamp, because it's first 41-bits form a timestamp in milliseconds).So in this way, if two commands modifying the same key are executed at different nodes, we solve the conflicts through Last-Write-Win and Add-Win strategies when they are merged at one node. We also support other crdt types including PNCounter, LWWRegister, MultiValueRegister and ORSet and so on. We also implemented the gc mechanism.
  • Online scaling: Nodes are free to join or leave a group. When we want to add a node(B) into an already running group, we simply let it to MEET a node(A) that is contained in that group. The new node(B) firstly connects to that node. Secondly they exchange their whole data set through their own snapshot file(this is safe because their data both implement CRDT)——in which all the necessary information of their replicas are also listed. When one node has merged the other's snapshot, it also records it's replicas. Then, each of them connects to the newly found replicas and exchange their data(probably a partial replication instead of a full snapshot dumping is used in this step, because the node(A) has told the new node(B) the replicate positions itself has received from it's replicas). Finnaly the new node(B) keeps pace with all other nodes in this group.
  • Multiple io threads: Input requests and output responses are read or written on io threads, and each command is executed sequentially in the main thread. This way the performance is much efficient.

Test

You can experience the master-master replications by the following test.

  1. Download the project and build.

git clone https://github.com/tancehao/ConstDB && cd ConstDB
cargo build

  1. Run 2 processes listening to different local ports. Make 2 copies of the file constdb.toml.

mkdir server9001 server9002

cp constdb.toml server9001/constdb.toml && cp constdb.toml server9002/constdb.toml

And then edit the constdb.toml config files in each directory(change the id, node_alias and port options). Finally start 2 process with these different configs.

cd server9001 && ../target/debug/constdb-server constdb.toml
cd ..
cd server9002 && ../target/debug/constdb-server constdb.toml

  1. Now a happy moment has come. We modify different keys in each node and make them keep in touch.

redis-cli -p 9001 incr k1 // 1
redis-cli -p 9002 incr k2 // 1
redis-cli -p 9002 meet 127.0.0.1:9001 // 1

wait for some seconds.

redis-cli -p 9001 get k2 // 1
redis-cli -p 9002 get k1 // 1
redis-cli -p 9002 incr k2 // 2
redis-cli -p 9001 get k2 // 2
redis-cli -p 9001 hset h1 k1 v1 // OK
redis-cli -p 9002 hget h1 k1 // v1

we can see that the each node stays abreast of the other.

Commands available

  • get
  • set
  • del
  • incr
  • decr
  • sadd
  • srem
  • spop
  • smembers
  • hset
  • hget
  • hgetall
  • hdel

Contact

Email: [email protected]
Wechat id: xiaotanzi-bien

You might also like...
Non-volatile, distributed file cache backed by content-addressed storage

blobnet A low-latency file server that responds to requests for chunks of file data. This acts as a non-volatile, over-the-network content cache. Inte

Rust type wrapper to cache hash of potentially large structures.

CachedHash For a type T, CachedHashT wraps T and implements Hash in a way that caches T's hash value. This is useful when T is expensive to hash (fo

Key-value cache RESP server with support for key expirations ⌛

BADER-DB (بادِر) Key-value cache RESP server with support for key expirations ⌛ Supported Features • Getting Started • Basic Usage • Cache Eviction •

A generational arena based LRU Cache implementation in 100% safe rust.

generational-lru Crate providing a 100% safe, generational arena based LRU cache implementation. use generational_lru::lrucache::{LRUCache, CacheError

Attribute-Level Caching in Heterogeneous In-Memory DBMS

Alchemy Attribute-Level Caching in Heterogeneous In-Memory DBMS Alchemy is a DRAM-PM hybrid database engine built from scratch to achieve high perform

Key-Value based in-memory cache library which supports Custom Expiration Policies

Endorphin Key-Value based in-memory cache library which supports Custom Expiration Policies with standard HashMap, HashSet interface. use endorphin::H

Nix binary cache implemented in rust using libnix-store

harmonia Build Whole application nix-shell --run cargo b C Library Wrapper around libnixstore nix-shell --run make Note: The makefile is only to pro

This contract is to provide vesting account feature for the both cw20 and native tokens, which is controlled by a master address

Token Vesting This contract is to provide vesting account feature for the both cw20 and native tokens, which is controlled by a master address. Instan

Simple GTK Rust Fuzzer which aims to test all available classes and functions in GTK.

Gtk Rust Fuzzer Simple GTK Rust Fuzzer which aims to test all available classes and functions in GTK. It finds bugs inside GTK functions, GTK exported

An object-relational in-memory cache, supports queries with an SQL-like query language.

qlcache An object-relational in-memory cache, supports queries with an SQL-like query language. Warning This is a rather low-level library, and only p

Stretto is a Rust implementation for ristretto. A high performance memory-bound Rust cache.

Stretto is a Rust implementation for ristretto. A high performance memory-bound Rust cache.

A general-purpose distributed memory cache system compatible with Memcached

memcrsd memcached server implementation in Rust memcrsd is a key value store implementation in Rust. It is compatible with binary protocol of memcache

A read-only, memory-mapped cache.

mmap-cache A low-level API for a memory-mapped cache of a read-only key-value store. Design The [Cache] index is an [fst::Map], which maps from arbitr

osu-link is a program which links osu!stable beatmaps to osu!lazer's new store format, saving you disk space.

osu-link is a program which links osu!stable beatmaps to osu!lazer's new store format, saving you disk space.

A blazingly fast, ShareX uploader coded in Rust (using actix web) which utilizes AES-256-GCM-SIV to securely store uploaded content.

Magnesium Oxide ❔ What is this? Magnesium-Oxide (MGO) is a secure file uploader with support for ShareX. 🌠 Features 🔥 Blazingly fast uploads and enc

RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

A memory efficient immutable string type that can store up to 24* bytes on the stack

compact_str A memory efficient immutable string type that can store up to 24* bytes on the stack. * 12 bytes for 32-bit architectures About A CompactS

Rust lib for a Vec-like structure that can store different types of different sizes contiguous with each other in memory.

hvec In memory of Anna Harren, who coined the term turbofish - which you'll see a lot of if you use this crate. The main purpose of this crate is the

Super-simple, fully Rust powered
Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.

memex Super simple "memory" for LLM projects, semantic search, etc. Running the service Note that if you're running on Apple silicon (M1/M2/etc.), it'

Comments
  • Why  set_reuseport to be true?

    Why set_reuseport to be true?

    https://github.com/tancehao/ConstDB/blob/5e5a1b51773c4f9b4d33062b377b4e37d223baec/src/server.rs#L99

    In daemon mode, if the user accidentally starts it twice, setting reuse to true will run two processes, which should not be the expected behavior.

    opened by fxsjy 0
Releases(0.0.1)
Owner
null
Stretto is a Rust implementation for ristretto. A high performance memory-bound Rust cache.

Stretto is a Rust implementation for ristretto. A high performance memory-bound Rust cache.

Al Liu 310 Dec 29, 2022
A general-purpose distributed memory cache system compatible with Memcached

memcrsd memcached server implementation in Rust memcrsd is a key value store implementation in Rust. It is compatible with binary protocol of memcache

null 274 Dec 14, 2022
A read-only, memory-mapped cache.

mmap-cache A low-level API for a memory-mapped cache of a read-only key-value store. Design The [Cache] index is an [fst::Map], which maps from arbitr

Duncan 3 Jun 28, 2022
Rust cache structures and easy function memoization

cached Caching structures and simplified function memoization cached provides implementations of several caching structures as well as a handy macro f

James Kominick 996 Jan 7, 2023
This is a Rust implementation for HashiCorp's golang-lru. This crate contains three LRU based cache, LRUCache, TwoQueueCache and AdaptiveCache.

This is a Rust implementation for HashiCorp's golang-lru. This crate contains three LRU based cache, LRUCache, TwoQueueCache and AdaptiveCache.

Al Liu 84 Jan 3, 2023
A native stateless cache implementation.

fBNC fBNC, Blockchain Native Cache. A native stateless storage library for block chain. Its value is to improve the stability and security of online s

Findora Foundation 1 Jan 12, 2022
A lightweight key-value cache system developed for experimental purposes

A lightweight key-value cache system developed for experimental purposes. It can also include distributed systems setup if I can.

Burak Selim Senyurt 8 Jul 23, 2022
A set of safe Least Recently Used (LRU) map/cache types for Rust

LruMap A set of safe Least-Recently-Used (LRU) cache types aimed at providing flexible map-like structures that automatically evict the least recently

Khonsu Labs 4 Sep 24, 2022
Read-optimized cache of Cardano on-chain entities

Read-optimized cache of Cardano on-chain entities Intro Scrolls is a tool for building and maintaining read-optimized collections of Cardano's on-chai

TxPipe 58 Dec 2, 2022
Turn your discord cache back to viewable images.

discache ?? Every time you view an Image in Discord, it gets saved in your cache folder as an unviewable file. Discache allows you to convert those fi

sam 2 Dec 14, 2022