Currently, ChiselStore performs reads using the Raft consensus protocol -- just like writes -- which provides strong consistency. However, strong consistency also makes reads pretty expensive. One way to optimize reads is to provide a "local read" option, which performs reads on any node. The "local read" option relaxes the consistency model by allowing stale reads in exchange for higher performance.

If we have registered a pending transition on the leader, but there's a leader change, the the client request has to either be rejected or restarted.

To fix this, we need a new Abandoned transition state in Little Raft:

https://github.com/andreev-io/little-raft/issues/13

Implements connection pooling so that we don't have to open a new connectio with every request. Benchmark won't even complete without this for very high QPS scenarios for strong consistency.

Add connection pooling for queries to avoid every thread hammering the same connection. Note that we should switch to the "memdb" VFS of sqlite. Unfortunately the driver we're currently using does not support that so let's use a file-backed database instead.

Relaxed reads don't require leader to be present because we're reading from the local replica and accept stale reads. Therefore, wait for the leader only when strong consistency is needed.

Fix a race condition where Little Raft applies a transition before we register a command completion.

Co-authored-by: Pekka Enberg [email protected] Co-authored-by: Glauber Costa [email protected]

@glommer suggests:

Pekka, please add a variation of #![warn(missing_docs, missing_debug_implementations, rust_2018_idioms)] to the top-level lib file so we make sure that all public interfaces have Debug, documentation, etc (likely you will want to enforce 2021 idioms)

As suggested by @glommer, let's move gRPC to core to make it easier for users to embed ChiselStore out-of-the-box. Users can, of course, also implement the necessary traits themselves if they don't want to use gRPC.

The apply_transition() function is executed on all state machines, but clients connect only to the leader. We can, therefore, only insert results for consumption on a leader. On followers, there is nobody consuming them, which results in a slow but sure resource leak.

When attempting to start the example server, I trip over:

penberg@vonneumann chiselstore % cargo run --example gouged -- --id 1 --peers 2 3
    Updating crates.io index
   Compiling event-listener v2.5.3
   Compiling chiselstore v0.1.0 (/Users/penberg/src/chiselstrike/chiselstore/core)
   Compiling async-mutex v1.4.0
    Finished dev [unoptimized + debuginfo] target(s) in 6.63s
     Running `target/debug/examples/gouged --id 1 --peers 2 3`
RPC listening to 127.0.0.1:50001 ...
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))', core/src/rpc.rs:55:52
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))', core/src/rpc.rs:55:52

I bisected the problem to 75ec0c75950499a91ef5dffcdc368cdc67924e38

We currently let SQL transaction commands to replicate, which can result in a dead-lock.

For example, if a writer issues a BEGIN TRANSACTION command, but dies after it has replicated, the cluster is in a situation where all nodes have an active transaction, but there's now writer to commit or roll back.

If we have an I/O error, we must not resume completion, but instead crash the process so that node with partial results does not continue in the same Raft cluster.

With relaxed reads, there's currently no guarantee on read-your-write consistency. This is because a write will be acknowledged when the write is applied to the state machine of the leader, but not on the local replica.

Example:

• A follower node F receives a write request, which is delegated to node L, which is the leader.
• The write is replicated to the logs of all nodes (but not necessarily applied).
• The write is applied to the state machine of L.
• L acknowledges the write to F and F acknowledges the write to the client.
• A relaxed read request arrives on node F, which does not yet have the write applied to its state machine, violating read-your-writes consistency.