Jocky - For controlling your horses

Current Idea

The idea is essentially to have a custom directory that writes immutable segments, this means only 2 file descriptors (one for writing and one for reading) are required per-index, which reduces the amount used in multi-tenant situations.

The writers have more stable performance since they depend less on the file cache to buffer writes and reads, the AIO implementation also leverages asynchronous background writing which makes CPU usage a bit more stable:

• 8 Threads, 2GB RAM, 2GB Swap, 5GB Produced index compressed.

• Green: AIO Writer

• Blue: MMapDirectory

Writes are messured in negative MiB/s for some reason, but just ignore the -

These unfinished segments as such can then be exported to a new file as the finished 'segment' as such, with each file re-organised to be contiguous.

Testing Screenshots

Mmap No Advise

Mmap MADV_SEQUENTIAL

Mmap MADV_RANDOM

Mmap No Advise - Constrained File Cache (6GB MAX)

Mmap MADV_SEQUENTIAL - Constrained File Cache (6GB MAX)

Mmap MADV_RANDOM - Constrained File Cache (6GB MAX)