A Write Ahead Log (WAL) implementation in Rust

Related tags

Command-line okaywal
Overview

Okay WAL

A write-ahead log (WAL) implementation for Rust.

There's The Great Wall, and then there's this: an okay WAL.

This crate exposes a WAL that supports:

  • ACID-compliant writes from multiple threads.
  • Random access for previously written data.
  • Supports automatic archiving/checkpointing to allow reusing disk space and preventing the WAL from growing too large.
  • Interactive recovery process with basic data versioning support.

Multi-threaded characteristics

This WAL implementation does not attempt to guarantee that recovery of WAL entries is performed in the same order in which the writes were written to the log. For example, imagine two threads writing to the log simultaneously. Each write to the log requires writing the actual data and then calling commit, which issues a file synchronization call to ensure all bytes are fully flushed to disk.

What is the order in which the two entries should be recovered in? If the WAL is being written to while modifying an in-memory data structure, the actual order of the changes is being determined outside of the order of the bytes being stored within the WAL. This means that the WAL has no way to properly know the exact order of the data.

During recovery, okaywal will recover entries in the order in which they were started. This may require you to either store additional data in the log or handle out-of-order operations manually.

For single-threaded writes, the order is guaranteed to be the order the data is written in.

You might also like...
Simple macros to write colored and formatted text to a terminal. Based on `termcolor`, thus also cross-platform.
Simple macros to write colored and formatted text to a terminal. Based on `termcolor`, thus also cross-platform.

Bunt: simple macro-based terminal colors and styles bunt offers macros to easily print colored and formatted text to a terminal. It is just a convenie

Lukas Kalbertodt 202 Dec 22, 2022
That program use on platform windows. And if you write any text on uncorrect keyboard layout, that program for that.
That program use on platform windows. And if you write any text on uncorrect keyboard layout, that program for that.

📌 This program is designed to translate text into the correct layout when typing is incorrect. 📌 Example ghbdtn - привет Just (by default) pressing

Gest Se 5 Jan 26, 2023
Contains challenges, write-ups, and deployment configurations from b01lersCTF 2023.

CTF Name A template repository for a CTF competition. This is a description of the CTF event. CTFTime Link Structure Challenges are organized by categ

null 7 Mar 29, 2023
Write simple proc-macros inline with other source code.

script-macro An experimental way to write simple proc-macros inline with other source code. Did you ever end up getting frustrated at the boilerplate

Markus Unterwaditzer 17 Jun 10, 2023
ask.sh: AI terminal assistant that can read and write your terminal directly!
ask.sh: AI terminal assistant that can read and write your terminal directly!

ask.sh: AI terminal assistant that read from & write to your terminal ask.sh is an AI terminal assistant based on OpenAI APIs such as GPT-3.5/4! What'

hmirin 5 Jun 20, 2023
bevy_scriptum is a a plugin for Bevy that allows you to write some of your game logic in a scripting language

bevy_scriptum is a a plugin for Bevy that allows you to write some of your game logic in a scripting language. Currently, only Rhai is supported, but more languages may be added in the future.

Jarosław Konik 7 Jun 24, 2023
Write your own exploit for $CASH

Cashio Exploit Workshop The Cashio hack was one of the biggest hacks occurred in Solana ecosystem which allowed the attacker to print infinite amount

Narya.ai 21 Mar 22, 2023
Write CI/CD pipelines using TypeScript

Katoa Katoa is a community fork of Cidada, a tool created by Fig which was sunset in late 2023 following acquisition by AWS. This fork and the underly

Katoa 47 Oct 6, 2023
Scriptable tool to read and write UEFI variables from EFI shell. View, save, edit and restore hidden UEFI (BIOS) Setup settings faster than with the OEM menu forms.
Scriptable tool to read and write UEFI variables from EFI shell. View, save, edit and restore hidden UEFI (BIOS) Setup settings faster than with the OEM menu forms.

UEFI Variable Tool (UVT) UEFI Variable Tool (UVT) is a command-line application that runs from the UEFI shell. It can be launched in seconds from any

null 4 Dec 11, 2023
Comments
  • Add shutdown function and file locking

    Add shutdown function and file locking

    When working on an actual multithreading unit test, I discovered an edge case when the archiving process kicks off as the threads are finishing. The wal is dropped, but the archiving thread continues the archiving process. The wal is reopened while the cleanup process is happening and yields a partial recovery of archived entries.

    By having an explicit shutdown function this test can reliably know when its safe to open the log again.

    Additionally, if we use file locking, we can protect against this behavior across processes too -- although only one process will ever have access to the WAL at any time.

    opened by ecton 0
  • Refactor format to allow streaming chunk writing

    Refactor format to allow streaming chunk writing

    Now that I've begun integration in Sediment, I realized that I would prefer chunk data to be written as:

    • Chunk command byte
    • Length (4 bytes)
    • Data (Length bytes)
    • CRC (4 bytes)

    By moving the CRC to the end, we can compute it on the fly, which would allow for a ChunkWriter type that could implement Write and allow buffered streaming chunk writing.

    opened by ecton 0
  • Make checkpointing more resilliant

    Make checkpointing more resilliant

    We need to ensure that a crash during the checkpoint operation can't leave the data store or okwal in a confused state.

    To ensure EntryId is always increasing, we can't risk losing track of the last entry id. In the current code, if a checkpoint occurs and no entries are left uncheckpointed, closing and reopening the log would lose track of the last entry id.

    I think the best approach is to add one more file that tracks checkpointing progress. This file should be able to track what EntryId was last checkpointed. This value should be updated after the checkpoint process finishes but before the files are recycled. This ensures that if a crash happens during any stage of the process, the ideal behavior can be taken:

    • Crash during checkpoint: After recovery, checkpointing will be attempted again. Primary data store must be able to apply WAL entries idempotently (or identify which have already been applied)
    • Crash during recycle: During recovery, any files containing a range of EntryIds that is less than the current checkpointed ID can be recycled upon load.
    opened by ecton 0
Owner
Khonsu Labs
Khonsu Labs
GitHub
Provides a cross platform way to shut down, reboot or log out operations

@napi-rs/system-shutdown This package provides a cross platform way to shut down, reboot or log out operations. Supported platforms: Linux, Windows an

LongYinan 7 Nov 2, 2023
Log-structured, transactional virtual block device backed by S3

mvps Log-structured, transactional virtual block device compatible with the NBD protocol. mvps stands for "multi-versioned page store". MVPS can store

Heyang Zhou 3 Dec 3, 2023
Extract patterns from unstructured log messages

logu logu is for extracting patterns from (streaming) unstructured log messages. For parsing unstructured logs, it uses the parser from Drain. In simp

null 78 Oct 21, 2024
colorStyle is a library of styles for command-line text write in Rust.

Colorstyle colorStyle is a library of styles for command-line text. Inspired by flylog/colorstyle (golang) Example let text = colorstyle::green("gre

Code Translation 6 Nov 12, 2022
Library to write x64 Assembly code from Rust, more properly. Designed for the nasm assembler

x64asm Library to write x64 Assembly code from Rust, more properly. Designed for the nasm assembler How to use let mut f = Formatter::new(false); // f

Anтo 7 Dec 30, 2022
A Rust program/library to write a Hello World message to the standard output.

hello-world Description hello-world is a Rust program and library that prints the line Hello, world! to the console. Why was this created? This progra

null 0 May 11, 2022
Write Cross-platform application with React-like decralative UI framework and scalable ECS architecture all in Rust.

bevy_dioxus Dioxus Plugin for Bevy Write Cross-platform application with React-like decralative UI framework and scalable ECS architecture all in Rust

Junichi Sugiura 269 Dec 29, 2022
Sets of libraries and tools to write applications and libraries mixing OCaml and Rust

Sets of libraries and tools to write applications and libraries mixing OCaml and Rust. These libraries will help keeping your types and data structures synchronized, and enable seamless exchange between OCaml and Rust

Meta 36 Jan 28, 2023
A tool for adding new lines to files, skipping duplicates and write in Rust!

anew A tool for adding new lines to files written in Rust. The tool aids in appending lines from stdin to a file, but only if they don't already appea

z3r0yu 4 Nov 16, 2023
Write a simple CLI script, that when given a 64-byte encoded string

Write a simple CLI script, that when given a 64-byte encoded string, it finds a suitable 4-byte prefix so that, a SHA256 hash of the prefix combined with the original string of bytes, has two last bytes as 0xca, 0xfe. Script should expect the original content of the string to be passed in hexadecimal format and should return two lines, first being the SHA256 string found and second 4-byte prefix used (in hexadecimal format).

Andy Bell 0 Feb 4, 2022