When working on an actual multithreading unit test, I discovered an edge case when the archiving process kicks off as the threads are finishing. The wal is dropped, but the archiving thread continues the archiving process. The wal is reopened while the cleanup process is happening and yields a partial recovery of archived entries.

By having an explicit shutdown function this test can reliably know when its safe to open the log again.

Additionally, if we use file locking, we can protect against this behavior across processes too -- although only one process will ever have access to the WAL at any time.

Now that I've begun integration in Sediment, I realized that I would prefer chunk data to be written as:

• Chunk command byte
• Length (4 bytes)
• Data (Length bytes)
• CRC (4 bytes)

By moving the CRC to the end, we can compute it on the fly, which would allow for a ChunkWriter type that could implement Write and allow buffered streaming chunk writing.

We need to ensure that a crash during the checkpoint operation can't leave the data store or okwal in a confused state.

To ensure EntryId is always increasing, we can't risk losing track of the last entry id. In the current code, if a checkpoint occurs and no entries are left uncheckpointed, closing and reopening the log would lose track of the last entry id.

I think the best approach is to add one more file that tracks checkpointing progress. This file should be able to track what EntryId was last checkpointed. This value should be updated after the checkpoint process finishes but before the files are recycled. This ensures that if a crash happens during any stage of the process, the ideal behavior can be taken:

• Crash during checkpoint: After recovery, checkpointing will be attempted again. Primary data store must be able to apply WAL entries idempotently (or identify which have already been applied)
• Crash during recycle: During recovery, any files containing a range of EntryIds that is less than the current checkpointed ID can be recycled upon load.