Background task processing for Rust applications with Tokio, Diesel, and PostgreSQL.

Overview

Backie

Async persistent background task processing for Rust applications with Tokio. Queue asynchronous tasks to be processed by workers. It's designed to be easy to use and horizontally scalable. It uses Postgres as a storage backend and can also be extended to support other types of storage.

High-level overview of how Backie works:

  • Client puts tasks on a queue
  • Server starts a multiple workers per queue
  • Worker pulls tasks off the queue and starts processing them
  • Tasks are processed concurrently by multiple workers

Backie started as a fork of fang crate, but quickly diverged significantly in its implementation.

Key features

Here are some of the Backie's key features:

  • Guaranteed execution: at least one execution of a task
  • Async workers: Workers are started as Tokio tasks
  • Application context: Tasks can access an shared user-provided application context
  • Single-purpose workers: Tasks are stored together but workers are configured to execute only tasks of a specific queue
  • Retries: Tasks are retried with a custom backoff mode
  • Graceful shutdown: provide a future to gracefully shutdown the workers, on-the-fly tasks are not interrupted
  • Recovery of unfinished tasks: Tasks that were not finished are retried on the next worker start
  • Unique tasks: Tasks are not duplicated in the queue if they provide a unique hash
  • Execution timeout: Tasks are retried if they are not completed in time

Other planned features

  • Scheduling of tasks: Tasks can be scheduled to be executed at a specific time

Task execution protocol

The following diagram shows the protocol used to execute tasks:

stateDiagram-v2
    [*] --> Ready
    Ready --> Running: Task is picked up by a worker
    Running --> Done: Task is finished
    Running --> Failed: Task failed
    Failed --> Ready: Task is retried
    Failed --> [*]: Max retries reached
    Done --> [*]

When a task goes from Running to Failed it is retried. The number of retries is controlled by the [BackgroundTask::MAX_RETRIES] attribute. The default implementation uses 3 retries.

Safety

This crate uses #![forbid(unsafe_code)] to ensure everything is implemented in 100% safe Rust.

Minimum supported Rust version

Backie's MSRV is 1.68.

Installation

  1. Add this to your Cargo.toml
[dependencies]
backie = "0.1"

If you are not already using, you will also want to include the following dependencies for defining your tasks:

[dependencies]
async-trait = "0.1"
serde = { version = "1.0", features = ["derive"] }
diesel = { version = "2.0", features = ["postgres", "serde_json", "chrono", "uuid"] }
diesel-async = { version = "0.2", features = ["postgres", "bb8"] }

Those dependencies are required to use the #[async_trait] and #[derive(Serialize, Deserialize)] attributes in your task definitions and to connect to the Postgres database.

  1. Create the backie_tasks table in the Postgres database. The migration can be found in the migrations directory.

Usage

The [BackgroundTask] trait is used to define a task. You must implement this trait for all tasks you want to execute.

One important thing to note is the use of the attribute [BackgroundTask::TASK_NAME] which must be unique for the whole application. This attribute is critical for reconstructing the task back from the database.

The [BackgroundTask::AppData] can be used to argument the task with your application specific contextual information. This is useful for example to pass a database connection pool to the task or other application configuration.

The [BackgroundTask::Error] is the error type that will be returned by the [BackgroundTask::run] method. You can use this to define your own error type for your tasks.

The [BackgroundTask::run] method is where you define the behaviour of your background task execution. This method will be called by the task queue workers.

use async_trait::async_trait;
use backie::{BackgroundTask, CurrentTask};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
pub struct MyTask {
    info: String,
}

#[async_trait]
impl BackgroundTask for MyTask {
    const TASK_NAME: &'static str = "my_task_unique_name";
    type AppData = ();
    type Error = ();

    async fn run(&self, task: CurrentTask, context: Self::AppData) -> Result<(), Self::Error> {
        // Do something
        Ok(())
    }
}

Starting workers

First, we need to create a [TaskStore] trait instance. This is the object responsible for storing and retrieving tasks from a database. Backie currently only supports Postgres as a storage backend via the provided [PgTaskStore]. You can implement other storage backends by implementing the [TaskStore] trait.

Then, we can use the task_store to start a worker pool using the [WorkerPool]. The [WorkerPool] is responsible for starting the workers and managing their lifecycle.

A full example of starting a worker pool can be found in the examples directory.

Queueing tasks

We can schedule tasks at any point using the [PgTask] trait. This will enqueue the task and whenever a worker is available it will start processing. Workers don't need to be started before enqueuing tasks. Workers don't need to be in the same process as the queue as long as the workers have access to the same underlying storage system. This enables horizontal scaling of the workers.

License

This project is licensed under the [MIT license][license].

Contributing

  1. Fork it!
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Acknowledgements

I would like to thank the authors of the Fang and background_job crates which were the main inspiration for this project.

You might also like...
Allows processing of iterators of Result types

try-continue try-continue provides one method, try_continue, which allows you to work with iterators of type ResultT, _, as if they were simply iter

convert images to ansi or irc, with a bunch of post-processing filters
convert images to ansi or irc, with a bunch of post-processing filters

img2irc (0.2.0) img2irc is a utility which converts images to halfblock irc/ansi art, with a lot of post-processing filters halfblock means that each

Services Info Register/KeepAlive/Publish/Subscribe. Based on etcd-rs, tokio

Services Info Register/KeepAlive/Publish/Subscribe. Based on etcd-rs, tokio

Sets of libraries and tools to write applications and libraries mixing OCaml and Rust

Sets of libraries and tools to write applications and libraries mixing OCaml and Rust. These libraries will help keeping your types and data structures synchronized, and enable seamless exchange between OCaml and Rust

 create and test the style and formatting of text in your terminal applications
create and test the style and formatting of text in your terminal applications

description: create and test the style and formatting of text in your terminal applications docs: https://docs.rs/termstyle termstyle is a library tha

General Rust Actix Applications and AWS Programming Utilities

RUST Actix-Web Microservice Our Rust Beginners Kit for Application Development A collection of sample code using the actix rust framework to A) Develo

CarLI is a framework for creating single-command and multi-command CLI applications in Rust

CarLI is a framework for creating single-command and multi-command CLI applications in Rust. The framework provides error and IO types better suited for the command line environment, especially in cases where unit testing is needed.

Upkeep your websites and web applications with ease from the comfort of the command line.
Upkeep your websites and web applications with ease from the comfort of the command line.

Upkeep Upkeep your websites and web applications with ease from the comfort of the command line. Explore the docs » View Demo · Report Bug · Request F

Start and stop system for applications to save your budget on hourly billing VPS.

Start and stop system (STT) Start and stop system for applications to save your budget on hourly billing VPS. Service A service consists of start/stop

Releases(0.6.0)
Owner
Rafael Carício
I like writing code.
Rafael Carício
Cornucopia is a small CLI utility resting on tokio-postgres and designed to facilitate PostgreSQL workflows in Rust

Cornucopia Generate type checked Rust from your SQL Install | Example Cornucopia is a small CLI utility resting on tokio-postgres and designed to faci

Louis Gariépy 1 Dec 20, 2022
Concurrent and multi-stage data ingestion and data processing with Rust+Tokio

TokioSky Build concurrent and multi-stage data ingestion and data processing pipelines with Rust+Tokio. TokioSky allows developers to consume data eff

DanyalMh 29 Dec 11, 2022
ddi is a wrapper for dd. It takes all the same arguments, and all it really does is call dd in the background

ddi A safer dd Introduction If you ever used dd, the GNU coreutil that lets you copy data from one file to another, then you may have encountered a ty

Tomás Ralph 80 Sep 8, 2022
TimeKnight is a neat little TUI-based timer app I use in conjunction with a task tracker

TimeKnight is a neat little TUI-based timer app I use in conjunction with a task tracker. It's kind of a secret sauce for productivity (particularly if you have ADHD or have a ridiculously overactive brain).

Monomadic 1 Feb 8, 2022
This crate provides a set of functions to generate SQL statements for various PostgreSQL schema objects

This crate provides a set of functions to generate SQL statements for various PostgreSQL schema objects, such as tables, views, materialized views, functions, triggers, and indexes. The generated SQL statements can be useful for schema introspection, documentation, or migration purposes.

Tyr Chen 11 Apr 4, 2023
Experimental extension that brings OpenAI API to your PostgreSQL to run queries in human language.

Postgres <> ChatGPT Experimental PostgreSQL extension that enables the use of OpenAI GPT API inside PostgreSQL, allowing for queries to be written usi

CloudQuery 315 Apr 10, 2023
A utility for exporting administrative/moderation statistics from your Lemmy instance's PostgreSQL database to InfluxDB!

Lemmy (Stats) Data Exporter About This Project This project aims to act as a bridge between Lemmy's PostgreSQL database and InfluxDB, primarily to tra

Russell 3 Jul 5, 2023
Rust Imaging Library's Python binding: A performant and high-level image processing library for Python written in Rust

ril-py Rust Imaging Library for Python: Python bindings for ril, a performant and high-level image processing library written in Rust. What's this? Th

Cryptex 13 Dec 6, 2022
Shaping, Processing, and Transforming Data with the Power of Sulfur with Rust

Sulfur WIP https://www.youtube.com/watch?v=PAAvNmoqDq0 "Shaping, Processing, and Transforming Data with the Power of Sulfur" Welcome to the Sulfur pro

Emre 6 Aug 22, 2023
Russh - Async (tokio) SSH2 client and server rimplementation

Russh Async (tokio) SSH2 client and server rimplementation. This is a fork of Thrussh by Pierre-Étienne Meunier which adds: More safety guarantees AES

Warptech Industries 113 Dec 28, 2022