Blobnet v0.2 — Design

Resolves: MOD-306, MOD-542

Blobnet: An embedded caching, content-addressed blob storage system with configurable sources and proxies.

Issues with Blobnet v0.1

• We have to download the entire file even if we only read a range, for caching reasons.
• There is no way of implementing local caching on the worker.
• The system isn’t flexible enough to allow for a gradual migration to S3 (or even just trying it out as another source of truth).

Proposal for Blobnet v0.2

The blobnet system is configured with an ordered set of sources, plus a cache.

Having two sources allows us to gradually transition from NFS to S3 without downtime.

let blobnet = Blobnet::builder()
    .source(provider::S3::new("modal-blobnet"))
    .source(provider::NFS::new("/efs/blobnet"))
    .cache("/var/tmp/.blobnet-cache", 1 << 21)
    .build();

Each provider has the following API:

#[async_trait]
trait Provider {
    async fn read(hash: &str, range: (u64, u64)) -> Result<Option<Bytes>>; // Return the data
    async fn write(data: Bytes) -> Result<String>; // Returns the hash of the data
}

All functions are fallible. The blobnet server has a Blobnet struct, and the worker client also has a Blobnet struct, which allows them to share SSD (local instance storage) caching logic.

The difference is that the worker client just uses its Blobnet struct to handle imagefs requests, while the blobnet server uses its struct to serve requests over HTTP.

S3 Provider

Takes the S3 bucket as an argument, also probably an instance of aws_sdk_s3's Client object to interact with them. Writes complete files to chunks in S3 by SHA-256 hash, such as /aa/bb/cc/dddddddddddddddddddddddddddddddddddddddddddddddddddddddddd.

NFS Provider

Takes a local directory as input, checks that it is a network file system on creation, and writes complete files atomically to that directory. Similar to S3-provider.

“Client Proxy” Provider

This acts as a client to a running blobnet server. This can be used on the worker itself to connect to the blobnet instance over HTTP, but then also include a cache to reuse the exact same caching logic on its local file system. Specifically:

let blobnet = Blobnet::builder()
    .source(provider::ClientProxy::new("http://blobnet.modal.internal"))
    .cache("/var/tmp/.blobnet-cache", 1 << 21)
    .build();

Cache

Saves chunks of files within a certain page size in the cache whenever something is read. All file reads go through the cache first before hitting the provider if it is missing. Note that this page size is configurable. In the examples above the page size is set to 2 MB (1 << 21) but this is configurable, since it only affects the cache.

The cache is a local file system directory. But the files saved are in the format /aa/bb/cc/ddd...dd/<chunk-num> because we don’t want to download the entire file if our goal is to execute a small read.

Update (Oct. 26): I think I’ll first benchmark Sled as a cache instead of EBS. It has the potential to be a lot faster, especially given its built in page cache and file defragmentation. Let’s see what happens. It doesn’t seem to be mature enough yet though, maybe RocksDB is better?

Update 2: Okay, I did some quick measurements, and with reasonable confidence I think using the file system directly is better. However there is still quite some latency here and we might want to also include an in-memory LRU cache too — this is the difference between 200-400 us for a 2 MB read with warm page cache, and 700-2800 ns for the same 2 MB read from memory.

I organized the benchmarks into groups and added two new benchmarks:

• image_delayed/cold — load an "image" by sequentially fully reading all files in a list with simulated random delay of mean 400 µs, where the files are:
  • 128 files of size 0-1KB
  • 10 files of size 1-10 MB
  • 1 file of size 1 GB
• image_delayed/warm — same as above, but a local memory + disk cache (non-delayed) has already been populated

400 µs is around the order of magnitude, or actually maybe twice as large as, the delay we were getting from blobnet over the network. The hope is that this benchmark can replicate the difference between a cold and warm cache, and we can use it to measure and tune prefetching strategies.

Note: I should also add a variable component to the delay that is proportional to number of bytes transferred.

I want to start take a look at system calls like read_at(), and to know whether changes like this actually affect performance, we need a benchmarking library. This PR adds criterion and some basic benchmarks.

Here's what the output looks like for removing a stat operation in #11 for example (when running on my MacBook Pro, I need to run this on Linux machines to actually get relevant results, will do after checking out this branch):

The improvement in the memory case is not important; just my computer not being a perfect deterministic environment.

Planning on storing mount sizes in the server. Returning file length in the GET response header, so we can add up the final mount size in the server MountBuild method.

This change was motivated by noticing that the read buffer was implicitly configured to be 16 KiB by tokio::fs::File and could not be changed, which made large file operations very inefficient.

I ended up doing a bit more though, since this is a pretty important part of the system. A quick mini-benchmark showed me that allocating 2 MiB of memory (Vec::with_capacity(1 << 21)) takes nanoseconds, while allocating and initializing 2 MiB of memory (vec![0; 1 << 21]) takes about 60 µs, which is really significant. It showed up in my measurements. So I took some time to remove that overhead as well.

1 KiB File Mini-Benchmarks

But even on small files, this change improves a lot due to avoiding uninitialized memory (with MaybeUninit and unsafe code) and minimizing seek(2) system calls using the low-level libc::pread function instead.

(I fixed a bug in our current benchmarks that made it not actually wait for the read response.) Now, on benchmarks that involve inserting then reading small 1 KiB files 1000 times, this shows a 3x speedup for reads from LocalDir, on an Ubuntu machine on AWS EBS.

I expect that speedup to carry through to other operations like ones on NFS.

64 MiB File Mini-Benchmarks

The main improvement should be in large files, though, due to the buffer size increase from 16 KiB to 2 MiB. What if we changed the benchmark to insert a single, 64 MiB file and read it 10 times?

Still twice as fast, great. It's on par with the cached-chunks implementation now, which makes sense, since both the source and the cache live on local file system.

Real-world impact

After this is merged, I can update #15 and we'll be able to see simulated latencies on NFS better.

This changes the semantics of blobnet.get(hash, range) consistently so that if the range starts after the end of the file, an empty result is returned instead of Error::BadRange. With these new semantics we can remove a stat() system call from the LocalDir provider and also remove the hacky secondary fallback request we had for the S3 provider.

This also makes end-of-file responses cacheable, resolving a part of MOD-306 that I mentioned in a comment.

This changes the ReadStream type definition from

pub type ReadStream = Pin<Box<dyn AsyncRead + Send>>;

to

pub type ReadStream<'a> = Pin<Box<dyn AsyncRead + Send + 'a>>;

By adding this lifetime, the code does become a little bit more complicated since lifetimes need to be propagated, but we can simplify some use of blobnet.put() as a result. Now to put a data: &[u8] we can just do:

blobnet.put(Box::pin(data)).await?;

Instead of the previous:

use std::io::Cursor;
blobnet.put(Box::pin(Cursor::new(data.to_owned()))).await?;

This avoids an unnecessary memory copy. The reason why this was necessary in the past is because ReadStream previously just always took a 'static lifetime implicitly, but for these arguments it's sometimes useful to have a stream that's borrowed rather than owned.

cc @freider who pointed this out

In my measurements, adding a 2 MiB buffer to the std::io::copy call has increased write throughput of a 128 MiB file to EFS from:

• Without buffering (before): 1 MiB / s
• With buffering (after): 30 MiB / s

This change adds the buffer. Note that std::io::write() has a specialization specifically for BufWriter that optimizes out its second internal stack buffer, so this code is equivalent or as fast as directly writing a for-loop over buffered chunks.

Note: There are some messages on Slack. It's late, I'm going to discuss this tomorrow with a fresh perspective.

This makes two changes to the API:

• A request to an empty range such as 0-0 will check if the file exists and the index is within the file's length.
  • Previously this just immediately returned OK.
• If the file has n bytes, GET requests with a range that starts at byte n will succeed and return an empty string.
  • Previously they returned a BadRange response, mimicking S3.

This makes the blobnet compatible with v0.1 Client's way of checking for existence, using a range 0-0.

I found a memory leak in the existing code and wrote a test that reproduced it.

But at this point I'm just going to remove all of this cache-advisor / bimap code right now because it's prone to memory leaks if not implemented perfectly, and I'll replace it with our own, very simple LRU cache implementation. No two-phase stuff, just extremely basic.

This is important because blobnet would require a second network round trip to check the file length, which isn't good for workloads like blobnet where we shouldn't have range issues regardless because we store it in the image manifest.

To do this, the blobnet API is slightly modified so that ranges dangling past the end of the file are now valid and truncated automatically. This is consistent with S3's API. I also corrected for some of the error handling logic.

I intend to release 0.2 after merging this and then testing the S3 provider locally.