Mordern Redis Cluster solution for easy operation.

Last update: Dec 6, 2022

Related tags

Database interfaces kubernetes rust redis cloud scale proxy redis-cluster migration failover k8s slot redis-cloud redis-proxy redis-protocol redis-clusters redis-instances redis-cluster-protocol

Overview

Undermoon

Undermoon is a self-managed Redis clustering system based on Redis Cluster Protocol supporting:

Horizontal scalability and high availability
Cluster management through HTTP API
Automatic failover for both master and replica
Fast scaling

Any storage system implementing redis protocol could also somehow work with undermoon, such as KeyDB.

For more in-depth explanation of Redis Cluster Protocol and how Undermoon implement it, please refer to Redis Cluster Protocol.

Architecture

Metadata Storage

Metadata storage stores all the metadata of the whole undermoon cluster, including existing Redis instances, proxies, and exposed Redis clusters. Now it's an in-memory storage server called Memory Broker. When using undermoon-operator, this Memory Broker will change to use ConfigMap to store the data.

Coordinator

Coordinator will synchronize the metadata between broker and server proxy. It also actively checks the liveness of server proxy and initiates failover.

Storage Cluster

The storage cluster consists of server proxies and Redis instances. It serves just like the official Redis Cluster to the applications. A Redis Cluster Proxy could be added between it and applications so that applications don't need to upgrade their Redis clients to smart clients.

Chunk

Chunk is the smallest building block of every single exposed Redis Cluster. Each chunk consists of 4 Redis instances and 2 server proxies evenly distributed in two different physical machines. So the node number of each Redis cluster will be the multiples of 4 with half masters and half replicas.

The design of chunk makes it very easy to build a cluster with a good topology for workload balancing.

Getting Started

Run Undermoon in Kubernetes

Using undermoon-operator is the easiest way to create Redis clusters if you have Kubernetes.

helm install my-undermoon-operator undermoon-operator-
   
    .tgz

helm install \
    --set 'cluster.clusterName=my-cluster-name' \
    --set 'cluster.chunkNumber=2' \
    --set 'cluster.maxMemory=2048' \
    --set 'cluster.port=5299' \
    my-cluster \
    -n my-namespace \
    undermoon-cluster-
    
     .tgz

See the README.md of undermoon-operator for how to use it.

Run Undermoon Using Docker Compose

See docker compose example.

Setup Undermoon Manually

Or you can set them up without docker following this docs: setting up undermoon manually.

Development

undermoon tries to avoid unsafe and some calls that could crash like unwrap.

Run the following commands before committing your codes:

$ make lint
$ make test

See more in the development guide.

Documentation

API

Comments

Optimization
[x] Change RwLock on server proxy meta to

(1) use mutex lock on write operation and use atomic pointer for read and write

(2) or Use CAS with https://github.com/vorner/arc-swap

[x] Batch sendto syscall.

[x] Optimize memory copy

[x] Let Resp objects just store the index of the Redis packet to reduce memory allocation.
opened by doyoubi 2
Bump regex from 1.5.4 to 1.5.6
Bumps regex from 1.5.4 to 1.5.6.

Changelog

Sourced from regex's changelog.

1.5.6 (2022-05-20)

This release includes a few bug fixes, including a bug that produced incorrect matches when a non-greedy ? operator was used.

[BUG #680](rust-lang/regex#680): Fixes a bug where [[:alnum:][:^ascii:]] dropped [:alnum:] from the class.

[BUG #859](rust-lang/regex#859): Fixes a bug where Hir::is_match_empty returned false for \b.

[BUG #862](rust-lang/regex#862): Fixes a bug where 'ab??' matches 'ab' instead of 'a' in 'ab'.

1.5.5 (2022-03-08)

This releases fixes a security bug in the regex compiler. This bug permits a vector for a denial-of-service attack in cases where the regex being compiled is untrusted. There are no known problems where the regex is itself trusted, including in cases of untrusted haystacks.

SECURITY #GHSA-m5pq-gvj9-9vr8: Fixes a bug in the regex compiler where empty sub-expressions subverted the existing mitigations in place to enforce a size limit on compiled regexes. The Rust Security Response WG published an advisory about this: https://groups.google.com/g/rustlang-security-announcements/c/NcNNL1Jq7Yw

Commits

9aef5b1 1.5.6

2931b07 syntax: bump minimum regex-syntax version to 0.6.26

b41bde0 regex-syntax-0.6.26

d98da65 changelog: 1.5.6

1c19619 syntax: fix literal extraction for 'ab??'

88a2a62 syntax: fix 'is_match_empty' predicate

72f09f1 syntax: fix ascii class union bug

b537286 doc: fix some typos

258bdf7 changelog: 1.5.5

d130381 1.5.5

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies rust
opened by dependabot[bot] 1
Bump tokio from 1.8.1 to 1.8.4
Bumps tokio from 1.8.1 to 1.8.4.

Release notes

Sourced from tokio's releases.

Tokio v1.8.4

1.8.4 (November 15, 2021)

This release backports a bugfix for a data race when sending and receiving on a closed oneshot channel ([RUSTSEC-2021-0124]) from v1.13.1.

Fixed

sync: fix a data race between oneshot::Sender::send and awaiting a oneshot::Receiver when the oneshot has been closed (#4226)

Tokio v1.8.3

1.8.3 (July 22, 2021)

This release backports two fixes from 1.9.0

Fixed

Fix leak if output of future panics on drop (#3967)

Fix leak in LocalSet (#3978)

#3967: tokio-rs/tokio#3967 #3978: tokio-rs/tokio#3978

Tokio 1.8.2

Fixes a missed edge case from 1.8.1.

Fixed

runtime: drop canceled future on next poll (#3965)

Commits

2273eb1 chore: fix CI on master (#4008)

249f05c chore: fix output of macro after new rustc release (#4189)

2bf6132 macros: fix type resolution error in #[tokio::main] (#4176)

c9228bf macros: make tokio-macros attributes more IDE friendly (#4162)

441427c macros: fix wrong error messages (#4067)

cc7d9e1 chore: explicitly relaxed clippy lint for runtime entry macro (#4030)

f49b7fc tokio-macros: compat with clippy::unwrap_used (#3926)

ea87e4e net: fix the uds_datagram tests with the latest nightly stdlib (#3952)

e2e7b5e examples: replace time crate with httpdate (#4169)

9a58f7f tests: update Nix to 0.22.0 (#3951)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies rust
opened by dependabot[bot] 1
Bump crossbeam-deque from 0.7.3 to 0.7.4
Bumps crossbeam-deque from 0.7.3 to 0.7.4.

Changelog

Sourced from crossbeam-deque's changelog.

Version 0.8.1

Support targets that do not have atomic CAS on stable Rust (#698)

Version 0.8.0

Bump the minimum supported Rust version to 1.36.

Bump crossbeam-channel to 0.5.

Bump crossbeam-deque to 0.8.

Bump crossbeam-epoch to 0.9.

Bump crossbeam-queue to 0.3.

Bump crossbeam-utils to 0.8.

Commits

0485137 Prepare for the next release

180b462 Fix deque steal race condition

48b13dc Bump crossbeam-channel to v0.4.4

679dac8 Bump crossbeam-channel to v0.4.3

f652c6a Change crossbeam-channel's license to MIT OR Apache-2.0.

149a3c9 Release crossbeam-queue 0.2.3

18363b5 Merge #476

e32c891 Release new versions

6d6591a Merge #474

528c3ca Undo bump of rand as it bumps MSRV

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies rust
opened by dependabot[bot] 1
Resolve peer host for SLAVEOF each time
Now the replicator module only resolves the hostname at the beginning. This may result in

outdated IP in k8s.

InvalidAddress error in k8s if the first resolve failed.

bug
opened by doyoubi 1
Work around bug in pooled redis client

The pooled Redis client could result in getting the reply of the last request in server proxy. I still have no idea why. But replacing the pooled Redis client with a non-pooled one could fix the problem. This will not have any performance impact on server proxy because the server proxy has already used some caching strategy.

opened by doyoubi 1
Support moving slots around

The current migration protocol only supports double scale which is not flexible for large clusters. We need to support moving the slot around just like native Redis Cluster.

issue: https://github.com/doyoubi/undermoon/issues/54

This PR implements the migration protocol described in https://github.com/doyoubi/undermoon/issues/54#issuecomment-569629973

opened by doyoubi 1
New Migration Process through SCAN command
Recent Migration Limitations

There are two drawbacks for the current migration process based on the current approach - trigger replication between the source Redis and destination Redis:

We can only perform a double scale because multiple FULL SYNC to a Redis node can't preserve all the data - the latter one will remove some of the right data transferred by the formers.

We have to wait for a fixed long time to HOPE that the replication between two Redis nodes is done because there's no way to know it. (Well, actually we can perform a block and check method).

Solutions

There are two solutions:

(1) Implement the Redis Replication Protocol which consists of two parts:

RDB parsing (hard and time-consuming to maintain)

Replication Network Protocol (not that hard but feasible)

(2) Use SCAN, DUMP, RESTORE to mimic the replication.

I don't want to use the first solution since there are just too many works for the RDB parsing. And we need to keep updating the codes as the RDB format changes.

The second one should be able to be compatible with the future versions of Redis.

The SCAN command has a great property that it can guarantee that all the keys set before the first SCAN command will finally be returned, for multiple times sometimes though. We can perform a 3 stage migration to mimic the replication.

Wait for all the commands to be finished by Redis.

Start the scanning and forward the data to peer Redis.

Redirect all the write operation after the first SCAN to peer Redis.

But it also has some problems:

It will have some impacts on the latency while scanning the data. But that could be tunable.

If a large collection type key gets updated frequently, we need to store the large data from DUMP again and again, which could result in a large amount of memory.

Detailed Steps for Scanning

Redirect all the later commands to Queue_block1.

Wait for all the existing commands to be finished. We can maintain a counter for running commands to achieve that.

Start to send SCAN, DUMP to get the data and add RESTORE command to Queue1. When SCAN is done, mark the Queue1 as SENT_FINISHED.

When the first SCAN command gets the reply, release all the commands in Queue_block1.

Let all the commands in Queue_block1 and the commands after the first SCAN to another send function which for all the write requests, do the following with Lua script to support atomic:

apply the write operation

get the latest version of the key by DUMP and forward the new data to Queue2

Forward all the RESTORE commands in Queue1. Once the Queue1 is set SENT_FINISHED and is empty, start to forward the RESTORE commands in Queue2.

When Queue1 is set SENT_FINISHED and is empty, start to block the commands in Queue_block2, wait for all the commands in Queue2 to be sent.

Migration source proxy commits the process with destination proxy. Destination starts to handle new slots. Source proxy starts to redirect the migrated slots to destination proxy.

Release the commands in Queue_block2. Redirect all the keys inside migrated slots to destination proxy.

INFOMGR starts to return success.

Wait for the Coordinator to commit the migration.

Delete the migrated out data in source proxy.

enhancement
opened by doyoubi 1
AUTH not cleaned up when recreating new cluster.

When a server_proxy deleted an old cluster and create a new one, the existing connections are still tagged the old cluster name, which results in db not found error.
enhancement

opened by doyoubi 1
Rename Host to Proxy

At the first time, I think we will only deploy one proxy per host. So host and proxy are the same and are used interchangeably in the codes and API. Now to support some clients and redis cluster proxies which do not support AUTH command for the backend clusters, we need to deploy multiple proxies in the same machine to support multiple tenants. I have changed the API in https://github.com/doyoubi/undermoon/pull/21. Later we need to change the host in code to proxy.

opened by doyoubi 1
Bump version to 0.6.2
Release v0.6.2

v0.6.2 is a minor release that starts to support client-side timeout.

Features

Amend node id format in the output of CLUSTER NODES

Add session timeout

Others

Add migration comparison test scripts
opened by doyoubi 0
v0.6 Road map
Undermoon v0.6 will come with some breaking changes and further improvements on migration during scaling.

Data Migration

[x] Optimize UMSYNC command

[ ] Amend retry for locked key in importing proxy during migration.

[ ] ~~Use multiple futures to scan the keys for migrating proxy based on slots to fasten the migration and reduce the latency of UMSYNC command.~~

[ ] Support new key lock that can be acquired by multiple non-del commands or del commands (which can be implemented by changing to slot lock instead of key lock and using reference counting).

[x] Add statistics for the performance key points in migration like lock contention count.

[ ] Add "migrated status" for hot key in the importing proxy during migration.

[x] Use reference counting to optimize WaitableTask

[x] Add client timeout

Reliability

[ ] Implement a new checker in Golang in place of the checker in undermoon-operator

Related issue: https://github.com/doyoubi/undermoon-operator/issues/33
opened by doyoubi 0
Persistency?

Hi Do you have plan to add persistency data to your implementation? (or already implemented?) If I turn off all nodes and run them again, will there be a redis data loss?

opened by sanarena 2
v0.5 Road Map
v0.5 Road Map

While v0.4 was focusing on working with undermoon-operator to support undermoon in kubernetes, v0.5 will focus on accessibility for users and contributors from the open-source community.

The key part here is to be compatible with some popular Redis Cluster clients.

Command Compatibility

Support more commands

[x] Add support for BZPOPMIN and BZPOPMAX thanks to @cfeitong

Redis Clients Campatibility Tests and Fixes

[x] redis-benchmark

[x] Python

[x] Java

[x] Jedis Add Jedis test

[x] Lettuce Add HELLO command

[x] Redission Add Reddision simple tests

[x] Golang Add go-redis tests. Fix BRPOPLPUSH

General Improvements

Performance Optimization

[x] Use state machine implementation in IO

[x] Support batching syscall for backend IO

Support Large Clusters

[x] Metadata Compression

Others

[x] Disallow changing cluster config during migration

[x] Support atomicity for some multi-key commands during migration

[x] Add timeout for backend connection

Kubernetes Improvements

[ ] ~~Change to use alpine image~~

[x] Implement 2-phase build in one dockerfile

[x] Support customized kubernetes scheduler

Community Friendly

[ ] Amend code style

[ ] Amend imports

[x] Remove unused features in Cargo.toml

[x] ~~Upgrade tokio~~ done in v0.6

[ ] Add template for issues and pull requests.

[ ] Write posts to introduce undermoon to the community

[x] hacker news

[ ] Detailed explanation on migration protocol with pictures
opened by doyoubi 0