disk backed wal queue

Related tags

Data structures qwal
Overview

Repository Template  Discord Build Status Quality Checks License Checks Security Checks Code Coverage

Queue like disk backed WAL


Pronouced Quál - from the german wordrd for agony - because it is.

Operations

The basic concept is simple, qual supports exactly 4 opperations:

push

Appends a new entry to the end of the queue, returns it's index.

pop

Reads the next index from the queue, returns the entry and it's index or none if the queue is fully consumed.

ack

Acknowledges the processing of a entry, this means that in the entry can be removed.

revert

Reverts back to the last acknowledged entry - will anything since then.

Performance Characteristics

push

Writes the data to disk and performs a fsync. Also a seek might be executed if a pop was performed since the last push.

pop

Reads the data from disk. Also a seek is performed if a push since the alst reads.

ack

No disk operations are performed, ack's are persisted either during a push operation or as part of the shutdown.

Operations

qwal provides all limits as soft limits, meaning they are considered reached once an operation exceeded them not before.

chunk_size

To allow easier reclemation of space the WAL is stored in multiple chunks. The chunk size defines the limit. A chunk is considered full once a push made it exceed the chunk_size

max_chunks

The soft limit of of chunks that can be open at the same time. The WAL is considered full when max_chunks + 1 would need to be created

You might also like...
Beanstalk is a simple, fast work queue.

beanstalkd Simple and fast general purpose work queue.

A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping

Swap Queue A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping. This is meant to be used [thread_local]

A dynamically prioritizable priority queue.

bheap A generic binary max heap implementation for implementing a dynamically prioritizable priority queue. This implementation uses a vector as the u

A lock-free multi-producer multi-consumer unbounded queue.

lf-queue A lock-free multi-producer multi-consumer unbounded queue. Examples [dependencies] lf-queue = "0.1" Single Producer - Single Consumer: use lf

A better message queue built by rust

bettermq A better message queue built by rust I start this project to study Rust

A single-producer single-consumer Rust queue with smart batching

Batching Queue A library that implements smart batching between a producer and a consumer. In other words, a single-producer single-consumer queue tha

Message/job queue based on bonsaidb, similar to sqlxmq.

Bonsaimq Simple database message queue based on bonsaidb. The project is highly influenced by sqlxmq. Warning: This project is in early alpha and shou

A scalable message queue powered by a segmented, partitioned, replicated and immutable log.
A scalable message queue powered by a segmented, partitioned, replicated and immutable log.

A scalable message queue powered by a segmented, partitioned, replicated and immutable log. This is currently a work in progress. laminarmq is intende

A rocksdb.rs wrapper bringing stack and queue functionalities

RocksDB_sq (Stack & Queue) A Rust crate that adds stack and queue functionality to RocksDB. This crate provide a wrapper around a RocksDB database and

Simple and flexible queue implementation for Rust with support for multiple backends (Redis, RabbitMQ, SQS, etc.)

Omniqueue Omniqueue is an abstraction layer over queue backends for Rust. It includes support for RabbitMQ, Redis streams, and SQS out of the box. The

A naive buffered/sync channel implementation in Rust, using the queue data structure

buffered-queue-rs Introduction This is my attempt at a simple and very naive buffered/synced queue implementation in Rust. The base thread-safe queue

A lightweight distributed message queue. Like AWS SQS and RSMQ but on Postgres.

Postgres Message Queue (PGMQ) A lightweight distributed message queue. Like AWS SQS and RSMQ but on Postgres. Features Lightweight - Built with Rust a

RocksDB-based queue with python bindings

RocksQ An inproc RocksDB-based queue with Python bindings. It is implemented in Rust. Features: max capacity limit in number of elements; size calcula

Terminal disk space navigator 🔭
Terminal disk space navigator 🔭

diskonaut How does it work? Given a path on your hard-drive (which could also be the root path, eg. /). diskonaut scans it and indexes its metadata to

Save disk space by cleaning non-essential files from software projects.
Save disk space by cleaning non-essential files from software projects.

Kondo 🧹 Cleans unneeded directories and files from your system. It will identify the disk space savings you would get from deleting temporary/unneces

CoreOS Disk Image Rehydrator

In CoreOS we ship a lot of disk images, one for each platform. These are almost entirely the same thing, mostly just with an ignition.platform.id stamp in each disk image, wrapped in a hypervisor/platform specific container.

Zenith - sort of like top or htop but with zoom-able charts, CPU, GPU, network, and disk usage
Zenith - sort of like top or htop but with zoom-able charts, CPU, GPU, network, and disk usage

Zenith - sort of like top or htop but with zoom-able charts, CPU, GPU, network, and disk usage

Terminal disk space navigator 🔭
Terminal disk space navigator 🔭

Given a path on your hard-drive (which could also be the root path, eg. /). diskonaut scans it and indexes its metadata to memory so that you could explore its contents (even while still scanning!).

osu-link is a program which links osu!stable beatmaps to osu!lazer's new store format, saving you disk space.

osu-link is a program which links osu!stable beatmaps to osu!lazer's new store format, saving you disk space.

Comments
  • Double pop issue

    Double pop issue

    This fixes an issue when a new WAL file is created, then an item is read, and immediately this file is then used to read from.

    What happened is:

    as part of initializing a new wall file, we preserve the last ACK id, however, this action did only advance the read, not the write pointer, leading to the new entry being written at the start of the file overwriting toe ack, then the read pointer starting inside the middle of the new entry.

    opened by Licenser 1
Owner
The Tremor Project
The Tremor Project
The Tremor Project
wait-free spsc linked-list queue with individually reusable nodes

A wait-free single-producer single-consumer linked-list queue with individually reusable nodes.

glowcoil 22 Dec 26, 2022
A lock-free multi-producer multi-consumer unbounded queue.

lf-queue A lock-free multi-producer multi-consumer unbounded queue. Examples [dependencies] lf-queue = "0.1" Single Producer - Single Consumer: use lf

Pierre Brouca 2 Sep 11, 2022
Message/job queue based on bonsaidb, similar to sqlxmq.

Bonsaimq Simple database message queue based on bonsaidb. The project is highly influenced by sqlxmq. Warning: This project is in early alpha and shou

Flix 6 Nov 8, 2022
A scalable message queue powered by a segmented, partitioned, replicated and immutable log.

A scalable message queue powered by a segmented, partitioned, replicated and immutable log. This is currently a work in progress. laminarmq is intende

Arindam Das 20 Dec 16, 2022
A doubly-linked list for Rust backed by generational-arena.

generational_token_list A doubly-linked list backed by generational-arena. Inspired by indexlist. Instead of returning pointers or numerical indices t

null 9 Dec 10, 2022
dua (-> Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory

dua (-> Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory. It's parallel by default and will max

Sebastian Thiel 1.8k Jan 2, 2023
A Write Ahead Log (WAL) implementation in Rust

Okay WAL A write-ahead log (WAL) implementation for Rust. There's The Great Wall, and then there's this: an okay WAL. This crate exposes a WAL that su

Khonsu Labs 5 Oct 28, 2022
A priority queue for Rust with efficient change function.

PriorityQueue This crate implements a Priority Queue with a function to change the priority of an object. Priority and items are stored in an IndexMap

null 139 Dec 30, 2022
A job queue built on sqlx and PostgreSQL.

sqlxmq A job queue built on sqlx and PostgreSQL. This library allows a CRUD application to run background jobs without complicating its deployment. Th

Diggory Blake 85 Jan 5, 2023
wait-free spsc linked-list queue with individually reusable nodes

A wait-free single-producer single-consumer linked-list queue with individually reusable nodes.

glowcoil 22 Dec 26, 2022