Hello there,

I am new to this project and have one question about the consensus crate.

I feel the 'Consensus:order_leaders' is redundant. The traverse of the dag is repeated twice(or more if some leader has no link to the later blocks) in both order_leader and order_dag. Is there any purpose for doing that?

What this PR does

• [x] Cleanup worker but keeping the same overall structure
• [x] Simplify the structure but keeping the same number of tasks (avoid code repetition)
• [x] Write good unit tests
• [x] Facilities to allow the primary to cleanup the worker (we didn't have that before)
• [x] Fallback workers sync in case the target recipient is dishonest (we didn't have that before)
• [x] Pipe the new worker with the existing primary (untouched)
• [x] Adapt and improve the benchmark scripts (make them print errors if anything is fishy)
• [x] Add meaningful comments
• [x] Run a few measurements (on AWS) to ensure we do not loose performance

Tips for review

1. Start with worker.rs. The worker has three main jobs, handle the primary messages, process client transactions, and listen to others' workers messages.
2. Check the function handle_primary_messages, then handle_clients_transactions, then handle_workers_messages
3. Dive into each component use by these functions: Synchronizer, BatchMaker, QuorumWaiter, Processor, and Helper.
4. Each we call the function spawn of a component, it means we are starting a new tokio task.
5. Check the network crate.
6. Pay special attention at ReliableSender (it is delicate and easy to get wrong): https://github.com/facebookresearch/narwhal/blob/cleanup-worker/rust/network/src/reliable_sender.rs
7. Check the config crate (nothing special here).

Hello! I am working on a secondary development of this project.

Reading the code, I see that the client (from benchmark_client.rs) sends data to the replicas here. A replica accepts it while makes and commit a new block. During this call, the client does not wait for a reply from the replica and thus it receives no data from the replica.

Consider a scenario where the client needs to fetch the data stored on the state machine replication. Could you please give me some hints on how I can modify your code so that replicas could respond to client requests?

Thanks a lot!

Hello,     We conducted a 100-node test on the WAN, and the test configuration and results are as follows:

+ CONFIG:
Faults: 33 node(s)
Committee size: 100 node(s)
Worker(s) per node: 1 worker(s)
Collocate primary and workers: True
Input rate: 234,500 tx/s
Transaction size: 200 B
Execution time: 41 s
Header size: 1,000 B
Max header delay: 200 ms
GC depth: 50 round(s)
Sync retry delay: 10,000 ms
Sync retry nodes: 33 node(s)
batch size: 200,000 B
Max batch delay: 200 ms
+ RESULTS:
Consensus TPS: 213,986 tx/s
Consensus BPS: 42,797,208 B/s
Consensus latency: 4,771 ms
End-to-end TPS: 207,890 tx/s
End-to-end BPS: 41,577,920 B/s
End-to-end latency: 7,852 ms

    We have some questions about the test log and configuration.

1. First, according to the configuration, the sending rate of each client is 3500 tx/s, and our test time is 30s. But according to client's log, each client sends about 800 tx in this period of time.
2. According to the worker's log, every 4 tx make up a batch, but batch contains 14000B is displayed, which doesn't seem to match the configured tx_size = 200 B.

Batch jOiahFVevxMc4+RQEIlZfEjFHha/oBesYqcEHBKSZiU= contains sample tx 786
Batch jOiahFVevxMc4+RQEIlZfEjFHha/oBesYqcEHBKSZiU= contains sample tx 787
Batch jOiahFVevxMc4+RQEIlZfEjFHha/oBesYqcEHBKSZiU= contains sample tx 788
Batch jOiahFVevxMc4+RQEIlZfEjFHha/oBesYqcEHBKSZiU= contains sample tx 789
Batch jOiahFVevxMc4+RQEIlZfEjFHha/oBesYqcEHBKSZiU= contains 140000 B

3. I would also like to ask about the meaning of the same number after each "Committed B" in the primary log.

Committed B97(mZFTSr1a8XJoClO4) -> WK0oFGTH44pm3PYAtejZ05EDysdIuDJ1MZuphZQe3m4=
Committed B97(mZFTSr1a8XJoClO4) -> isS9EtiKzZ2qm3DfL37mt8o02TPS519+/aEDggnzTTE=
Committed B97(mZFTSr1a8XJoClO4) -> vI69CmxG2PlTMDc5GYZreMZVUIFIhS8zSIQCQmBxAlk=

• [x] Clean up the primary (without changing its structure)
• [x] Clean up consensus
• [x] Write meaningful tests
• [x] Run a few measurements to ensure we don't loose performance

Could you pay special attention to the consensus?

Hello.

I think there is a bug in reliable sender's connection's pending replies queue ordering, but please let me know if it is my understanding that is lacking.

If we send multiple messages to a peer, there is no reason that the acknowledgements will be received in the same order. Yet the code seems to assume that because it is sending the received ACK message to the first handler in the queue (pop_front).

It seems that a peer connection is needed to establish between any two regions to achieve high bandwidth in aws ec2, while i didn't find in benchmark scripts.

What is the best way to:

• Update the committee (change of authorities)
• Change authorities network info (not the authorities per se, but for instance their ip addresses)

The Primary serializes and deserializes data many times for different reasons (storage, net, replying to sync requests). It would be nice to use the same trick as the worker to avoid/limit that.