A scalable message queue powered by a segmented, partitioned, replicated and immutable log.

Related tags

Data structures message-queue io-uring segmented-log
Overview

derefdb

A scalable message queue powered by a segmented, partitioned, replicated and immutable log.

This is currently a work in progress.

laminarmq is intended to be a re-implementation of the original Kafka paper with some enhancements.

The following main features are intended:

  • True async APIs based on io_uring using glommio.
  • Segmented Log for storing messages in a partition.
  • SWIM gossip protocol based service discovery.
  • Raft consensus powered replication of partitions in a topic.

Architecture

We discuss the architecture of laminarmq in docs/architecture.md.

License

Dual licensed under MIT License and Apache 2.0 License to be compatible with the Rust project.

You might also like...
Algorithms and Data Structures of all kinds written in Rust.

Classic Algorithms in Rust This repo contains the implementation of various classic algorithms for educational purposes in Rust. Right now, it is in i

Alexander González 49 Dec 14, 2022
Obake is a procedural macro for declaring and maintaining versioned data-structures.

Obake is a procedural macro for declaring and maintaining versioned data-structures. The name 'obake' is taken from the Japanese 'お化け (おばけ)', a class of supernatural beings in Japanese folklore that shapeshift.

Nathan Corbyn 174 Dec 26, 2022
A simple rust library to help create octrees and quadtrees for chunked level of detail

LodTree LodTree, a simple tree data structure for doing chunk-based level of detail. Goals The aim of this crate is to provide a generic, easy to use

Dimev 14 Dec 29, 2022
Serialize/DeSerialize for Rust built-in types and user defined types (complex struct types)

Serialize/DeSerialize for Rust built-in types and user defined types (complex struct types)

null 2 May 3, 2022
Broot - A new way to see and navigate directory trees
Broot - A new way to see and navigate directory trees

Broot A better way to navigate directories Installation Instructions Get an overview of a directory, even a big one br -s Notice the unlisted? That's

Canop 8k Jan 8, 2023
Rust-algorithm-club - Learn algorithms and data structures with Rust

Rust Algorithm Club 🚧 🚧 This repo is under construction. Most materials are written in Chinese. Check it out here if you are able to read Chinese. W

Weihang Lo 360 Dec 28, 2022
Common data structures and algorithms in Rust

Contest Algorithms in Rust A collection of classic data structures and algorithms, emphasizing usability, beauty and clarity over full generality. As

Aram Ebtekar 3.3k Dec 27, 2022
A hashmap implementation, which uses hashset, and keys are contained within values.

A hashmap implementation, which uses hashset, and keys are contained within values.

Piotr Mikulski 2 Nov 29, 2022
Rust data structures and client for the PubChem REST API

pubchem.rs Rust data structures and client for the PubChem REST API. 🔌 Usage 💊 Compound Create a Compound to query the PubChem API for a single comp

Martin Larralde 2 Jan 18, 2022
Comments
  • Improve code coverage and remove unused modules

    Improve code coverage and remove unused modules

    • Added tests for single_node router
    • Added tests for PartitionConsumer failure paths
    • Exclude compat modules and Display impls from code coverage
    • Move from BytesCow to Bytes in InMemPartition
    opened by arindas 1
  • Stabilize RPC server components.

    Stabilize RPC server components.

    • Migrated copy-on-write pointers to generic byte containers to avoid hidden clones.
    • Formalized the server RPC API for single node deployments
    • Designed and implemented the execution model for processing requests in our RPC server:
      • Formalized abstract generic interfaces that can be implemented for different async runtimes
      • Provided a concrete implementation for glommio runtime.
    • Added crucial documentation on the internals of laminarmq
    opened by arindas 1
  • Added necessary pieces of RPC Server implementation

    Added necessary pieces of RPC Server implementation

    • Migrated copy-on-write pointers to generic byte containers to avoid hidden clones.
    • Formalized the server RPC API for single node deployments
    • Designed and implemented the execution model for processing requests in our RPC server:
      • Formalized abstract generic interfaces that can be implemented for different async runtimes
      • Provided a concrete implementation for glommio runtime.
    • Added crucial documentation on the internals of laminarmq
    opened by arindas 0
  • Overhaul documentation in different submodules

    Overhaul documentation in different submodules

    • Added missing documentation for different items under crate::server
    • Added diagrams to explain the execution model and the segmented log data structure
    • Make Testing much more accessible with proper description in README
    opened by arindas 0
Releases(0.0.4)

  • 0.0.4(Dec 28, 2022)

    What's Changed

    • Improve code coverage and remove unused modules by @arindas in https://github.com/arindas/laminarmq/pull/6

    Full Changelog: https://github.com/arindas/laminarmq/compare/0.0.3...0.0.4

    Source code(tar.gz)
    Source code(zip)

  • 0.0.3(Dec 23, 2022)

    What's Changed

    • Refactor/record api by @arindas in https://github.com/arindas/laminarmq/pull/1
    • Merge development updates for refactors. by @arindas in https://github.com/arindas/laminarmq/pull/2
    • Overhaul documentation in different submodules by @arindas in https://github.com/arindas/laminarmq/pull/3
    • Added necessary pieces of RPC Server implementation by @arindas in https://github.com/arindas/laminarmq/pull/4
    • Stabilize RPC server components. by @arindas in https://github.com/arindas/laminarmq/pull/5

    Full Changelog: https://github.com/arindas/laminarmq/compare/0.0.2...0.0.3

    Source code(tar.gz)
    Source code(zip)

  • 0.0.2(Sep 23, 2022)

    Release notes:

    • Update {store, segment, log}::read() API to return the offset or position (where applicable) for the next record. This makes client code more ergonomic as they always know which offset to read from if they know the first offset.
    • Removed leaky abstractions
      • Removed header_padded_record_length() from commit_log::store::common
      • Removed advance_to_offset() and offset_of_record_after() from commit_log::CommitLog. They were leaking implementation details.
    Source code(tar.gz)
    Source code(zip)

  • 0.0.1(Sep 21, 2022)

Owner
Arindam Das
Specializes in distributed systems, deep learning inference and AI SaaS at scale.
Arindam Das
GitHub https://arindas.github.io/laminarmq/laminarmq/index.html
Blazing fast immutable collection datatypes for Rust.

imbl Blazing fast immutable collection datatypes for Rust. This is a fork of the im crate, which appears to be unmaintained. (This fork is not current

null 38 Jul 11, 2022
A priority queue for Rust with efficient change function.

PriorityQueue This crate implements a Priority Queue with a function to change the priority of an object. Priority and items are stored in an IndexMap

null 139 Dec 30, 2022
wait-free spsc linked-list queue with individually reusable nodes

A wait-free single-producer single-consumer linked-list queue with individually reusable nodes.

glowcoil 22 Dec 26, 2022
disk backed wal queue

Repository Template  Queue like disk backed WAL Pronouced Quál - from the german wordrd for agony - because it is. Operations The basic concept is si

The Tremor Project 8 Jun 4, 2022
A lock-free multi-producer multi-consumer unbounded queue.

lf-queue A lock-free multi-producer multi-consumer unbounded queue. Examples [dependencies] lf-queue = "0.1" Single Producer - Single Consumer: use lf

Pierre Brouca 2 Sep 11, 2022
Serde is a framework for serializing and deserializing Rust data structures efficiently and generically.

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically.

null 6.5k Dec 31, 2022
Coding-challenge - Algorithms and Data-structures, problems and solutions in Rust language using cargo-workspaces

Coding Challenge LeetCode/Hackerrank e.t.c Using this as an opportunity to improve my knowledge of rust lang If you found this repo useful to you, add

Tolumide Shopein 17 Apr 24, 2022
Extra iterator adaptors, iterator methods, free functions, and macros.

Itertools Extra iterator adaptors, functions and macros. Please read the API documentation here How to use with cargo: [dependencies] itertools = "0.1

null 1.9k Dec 30, 2022
Array helpers for Rust's Vector and String types

array_tool Array helpers for Rust. Some of the most common methods you would use on Arrays made available on Vectors. Polymorphic implementations for

Daniel P. Clark 69 Dec 9, 2022
K-dimensional tree in Rust for fast geospatial indexing and lookup

kdtree K-dimensional tree in Rust for fast geospatial indexing and nearest neighbors lookup Crate Documentation Usage Benchmark License Usage Add kdtr

Rui Hu 152 Jan 2, 2023