This PR describes a Rust training course called "Practical Networked Applications in Rust", and lays out a draft lesson plan.

Rendered README.

Rendered lesson plan.

The flow of this course is curious, with topics derived from the needs of the hands-on projects and the hands-on projects designed to teach useful and interesting topics, leaving some important topics not covered (at least yet). I believe the lesson plan proceeds quite differently from typical Rust intro/beginner material. I hope that the documentation conveys the rationale sufficiently.

It is not quite finished, but I think there's enough here to understand what it's about, enough to review and critique.

The structure of this course is inspired by the MIT 5.824 distributed systems course, which we are also adapting to Rust as a follow-on course to this one. I intend that the two courses ultimately share a common structure.

The lesson plan is intended to be an index, with links to the full lessons and projects. The descriptions within the lesson plan of the goals, topics etc are intended as a guide to writing the full lessons and project descriptions. Once those are written we might reduce the descriptive content in the lesson plan so that it's easier for the reader to take in the full lesson/project progression by skimming.

I very much want feedback. Are the goals and non-goals reasonable? Is the project interesting? Does the lesson plan make sense? Are the topics interesting? Within the given lesson plan, and considering the projects being implemented, what other topics should be included? As a new Rust programmer, does the content here excite you to take the course? What changes would make the course more exciting?

I'm also looking for people interested in helping build this course, since there's quite a lot of material to cover. If this looks like something you would like to participate in, please speak up.

Note that there is a file here called notes.md that contains notes I've taken about this course. It includes a list of potential subjects and links to other courses and draft training material. I intend the material to take inspiration, if not content, from https://github.com/ferrous-systems/rust-three-days-course, https://github.com/nrc/talks, and other existing training material. We'll do a survey of existing training material soon, with goals of understanding how this course can provide unique value, and how other courses progress from topic to topic.

I was hoping that this PR would fill out some of the material's structure, like how the slides and text are organized, an example full project description, how the projects' tests work; but I haven't made it that far yet.

cc @siddontang PTAL. I hope that the projects are aligned with your expectations. I suspect the volume of lesson material here is more ambitious than you expected, and is very probably more than can be completed in Q1. We can deprioritize certain lessons for an MVP. Give me all the feedback you've got. We haven't discussed how this course is licensed or who owns it, but I've preemptively put a CC-BY notice at the bottom of the readme so that there's some intent conveyed as to the license.

cc @Hoverbear @breeswish @overvenus @rleungx @Hijiao @huachaohuang @nrc if you have the time I'd be grateful for your feedback.

cc @skade I'd be grateful for any review you can offer. Hoverbear and I have been inspired by your rust-three-days-course and you've got lots of experience in Rust training.

• Practical Networked Applications in Rust
  • Update: BufReaderWithPos ~~and BufWriterWithPos~~.
  • Fix: Temp_dir is dropped during the test.
• Distributed Systems in Rust
  • Update: README.md
  • Fix: wrong code in the comment.

This pr is based on #7 .

The structopt extension is not included yet.

I don't know how to test the cli in integration tests, so I only write tests for KvStore.

The clap get_matches function uses os_args, which I cannot easily hack.

Another way is to use get_matches_from_safe and pass args in, but that will make the signature of our main function horrible to beginners.

assert_cli or assert_cmd should work too. But then I cannot think of the advantage of placing the user interface into the library.

It's been a long time since I've submitted anything, so here's my current branch.

The main things to note here is that project 1 has more text now, and that I've removed gRPC and tokio from the plan (from discussion with Liu we agreed they were too ambitious for these 5 projects and could come in a later project).

Rendered

Tomorrow I will create an issue outlining the next steps so that others might help.

PTAL @overvenus @nolouch

To accomplish #118

Specify which tests should be pass in each parts.

Before modifying the code in subsequent projects, I want to hear from @sticnarf @brson . If you think it's ok, I will add commits to change the code in Project these project.

I'm not clear in the problems below:

• In project2, we need to implement set/rm in part2, but some tests for set/rm use get, which should be implemented in part3.
• In project3, I only specify the tests in part4 and part6. I wonder if there are some more proper way to place these tests.

Fixes https://github.com/pingcap/talent-plan/issues/38

I put this together quickly after our discussion @sticnarf. I do have a few concerns:

• there is apparently no production-quality concurrent map type we can incorporate :-/
• like in the previous project, this abstracts a thing then compares to implementations. I don't know if it is valuable to do the same type of exercise again. I'd prefer if we can find a new angle on this, so I've suggested we do a compile-time abstraction using cargo features, but I'm open to any ideas.

I found in my computer, sort.Slice is about 15s, merge sort in single thread need about 9s. Then I write a base quick sort in hand with go, it is only about 7s in benchmark(sort.Slice slower than hand quick sort maybe because reflect?). Anybody see the same phenomenon？

Implementions is as follow(go version: 1.12.2):

merge sort:

// MergeSort performs the merge sort algorithm.
// Please supplement this function to accomplish the home work.
func MergeSort(src []int64) {
	dst := make([]int64, len(src))
	copy(dst, src)
	mSort(dst, src, 0, len(src))
}

//从src归并到dest
func mSort(src []int64, dest []int64, low int, high int) {
	if high-low <= 1 {
		return
	}

	mid := (low + high) >> 1
	mSort(dest, src, low, mid)
	mSort(dest, src, mid, high)

	for i, p, q := low, low, mid; i < high; i++ {
		if q >= high || (p < mid && src[p] < src[q]) {
			dest[i] = src[p]
			p++
		} else {
			dest[i] = src[q]
			q++
		}
	}
}

my hand quick sort:

func quickSort(arr []int64) {
	qSort(arr, 0, len(arr) - 1)
}

func qSort(arr []int64, low int, high int) {
	if low < high{
		pivot := partition(arr, low, high)
		qSort(arr, low, pivot - 1)
		qSort(arr, pivot + 1, high)
	}
}

func partition(arr []int64, low int, high int) int {
	pivotKey := arr[low]
	for low < high{
		for low < high && arr[high] >= pivotKey{
			high--
		}
		arr[low] = arr[high]
		for low < high && arr[low] <= pivotKey{
			low++
		}
		arr[high] = arr[low]
	}
	arr[low] = pivotKey

	return low
}

benchmark:

func BenchmarkMergeSort(b *testing.B) {
	numElements := 16 << 20
	src := make([]int64, numElements)
	original := make([]int64, numElements)
	prepare(original)

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		b.StopTimer()
		copy(src, original)
		b.StartTimer()
		MergeSort(src)
	}
}


func BenchmarkNormalSort(b *testing.B) {
	// 16M个整数
	numElements := 16 << 20
	src := make([]int64, numElements)
	original := make([]int64, numElements)
	prepare(original)

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		b.StopTimer()
		copy(src, original)
		b.StartTimer()
		sort.Slice(src, func(i, j int) bool { return src[i] < src[j] })
	}
}

func BenchmarkQuickSort(b *testing.B) {
	// 16M个整数
	numElements := 16 << 20
	src := make([]int64, numElements)
	original := make([]int64, numElements)
	prepare(original)

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		b.StopTimer()
		copy(src, original)
		b.StartTimer()
		quickSort(src)
	}
}

Benchmark results as follows:

goos: linux
goarch: amd64
pkg: pingcap/talentplan/tidb/mergesort
BenchmarkMergeSort-12                 1         9503371329 ns/op
BenchmarkNormalSort-12                1       15261800118 ns/op
BenchmarkQuickSort-12                  1         7552893791 ns/op
PASS
ok      pingcap/talentplan/tidb/mergesort       37.860s

We can see that merge sort is faster than sort.Slice，hand quick sort is faster than merge sort.

I also benchmark sort.Sort, found it is slower than sort.Slice

If so, Is it more meaningful to compare parallel merge sort to quick sort in hand, instead of sort.Slice?

This builds on https://github.com/pingcap/talent-plan/pull/36.

This project has been in flux, I made a bunch of calls here on what it is about, and I think it has become pretty substantial. Anybody who completes this course is going to be on their way to writing a real database.

Closes #37.

Recently, I find there are some students who turn to the example solution directly before they finish their own project. So I think we need to remove the example solution from the repo.

Note that it does not mean we don't trust students. Probably, they just don't read the lesson plan carefully and click into the solution by mistake.

There is another important reason why we'd better remove it. The distributed system course in this repo adapts the way of filling the framework. It is better to keep consistent among courses. (There is a student who took dss course first. Then, he thought the rust course is the same way so he cloned the repo and modified the existsing code to finish the course.)

Hi, I am working on the kvraft lab, and I find my code pass or not pass some tests randomly, and I had spent long time to debug with it (use println! 🙂), it leads me to the tests, and the Rpc framework, and I not sure is it just bugs of my implement or the tests, but here is what I found: within the generic_test function, if the crash arg set to true, the tester will isolating all server's network, so servers can't not reach neither other servers nor the clients, and the tester also take a clone of all server's persister. And there come the problem: the Rpc framework only check network connection and set up network environment(to simulate worse network environment) when request comes in and pull it to the poller to simulate network, but the Rpc framework will not check network connection when simulate finished and process the reply base on the network environment seted up when request come in. And come back to the test, when the tester call shutdown_server on a server, the situation mentioned on the defination of shutdown_server may still happen:

    // disable client connections to the server.
    // it's important to do this before creating
    // the new Persister in saved[i], to avoid
    // the possibility of the server returning a
    // positive reply to an Append but persisting
    // the result in the superseded Persister.
    self.net.delete_server(&format!("{}", i));

specifiy when a client send a PutAppand request to the leader, and the leader send the append_entries request to majority of the followers, and then tester call shutdown_server on all server and create new Persister base on the Persister of a server. But as Rpc framework do not check network on reply, the majority of follower can still response to leader, and leader can still response a positive reply to client, and leader and follower persisting the result in the superseded Persister. This happen only when tester call shutdown_server between

1. leader send the append_entries request to majority of the followers
2. Rpc framework finish simulate of the network transfer and dispatch request to followers for process

usually this interval will be very small but when the unreliable arg to generic_test set to true there will be a short_delay in this interval which increasing the possibility, and also when implementing lease read, all log entries will be PutAppend command, and also increasing the possibility, and in my implement of kvraft which happend 10s times pass and 1 time not pass mainly on tests: test_persist_concurrent_unreliable_3a and test_snapshot_unreliable_recover_3b like: panicked at '1 missing element "x 1 31 y" in Append result ... And that is what I find, I not really sure is it jut a bug of my implement, as I am new to Rust, may be there are just some native misunderstanding of the tests and the Rpc framework, and I very need help on it, thanks advance .

Hi, I come across the problem when implementing KVRaft 3A.

I could pass test_basic_3a, where there is only one client.

But in test_concurrent_3a, a weird blocking problem appears, I have simplified the code to make the problem more obvious, here is what I did:

Two clients call put_append RPC to one same KVServer(leader or follower) at the same time;

KVServer's put_append handler does one thing and only one thing: thread::sleep(5 Seconds);

So KVserver is sleeping, but in my opinion, its Raft Module should work as normal (i.e. sends heartbeat to maintain its authority). But, after a Raft Election Timeout (from the time when the KVServer receives these two RPCs), all other raft module becomes candidate and request votes from others, and as I observed, no one could receive any RPC from anyone.

Above is a simplified version, in my real implementation(which passed one client test_basic_3a), this problem still exists, this is how it comes:

Like before, two clients call put_append RPC to KVServer(leader) at the same time;

Node is locked, so I can focus on dealing with one RPC at a time.

KVServer's put_append handler calls Raft.start(&message) once, which would send AppendEntries RPC to peers;

If what I'm suspecting is correct: the two client RPC calls is blocking the whole RPC network, thus no AppendEntries RPC will be received by peers, such that the KVServer(leader server) Polls apply_ch but cannot get a response, and waits forever.

Then after a Raft Election Timeout this leader loses its leadership.

Loops.

I hope someone could help me with this problem, is this a common problem or has anyone got the KVRaft task done without encountering the blocking problem?

Below is part of my simplified code, if someone interested in trying himself.

In client.rs

impl Clerk {
    fn put_append(&self, op: Op) {
        let reply = self.servers[leader_i].put_append(&Op_args)
                         .map_err(Error::Rpc).wait();
    }
}

In server.rs

impl Node {
    pub fn new(kv: KvServer) -> Node {
        Node {
            server: Arc::new(Mutex::new(kv)),
        }
    }
}

impl KvService for Node {
    fn get(&self, arg: GetRequest) -> RpcFuture<GetReply> {
        unimplemented!()
    }

    fn put_append(&self, arg: PutAppendRequest) -> RpcFuture<PutAppendReply> {
        thread::sleep(5000 * MILLIS);
        unimplemented!()
    }
}

I think I can work around this problem by letting KVServer refuse the second RPC from Client, such that there could be only one RPC not responsed at one time, but this shouldn't be the correct way right?

What problem does this PR solve?

• Brief description of the problem: Current links for first 4 chapter's learning materials are invalid; 404 error.

What is changed and how it works?

Update links to the new url of the repositories under GitHub Skills

Check List

Tests

• No code

generally maintained the same format

What problem does this PR solve?

• Issue number: close #xxx
• Breif description of the problem:

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Possible performance regression
• Increased code complexity

Related changes

• Need to update the documentation

What problem does this PR solve?

• Issue number: close #434
• Breif description of the problem: the tutorial of CLI has been out of date, and not work well under current clap ecosystem.

What is changed and how it works?

I append some help documentations for bb1's exercise and bump up project 1's rustc version. actually I also consider the exercise doesn't match current rust ecosystem, but I have no idea which tutorial could work it out instead. maybe this is not a bad choice: https://medium.com/@ukpaiugochi0/building-a-cli-from-scratch-with-clapv3-fb9dc5938c82

Bumps thread_local from 1.0.1 to 1.1.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.