Simple, extensible multithreaded background job processing library for Rust.

Related tags

Command-line apalis
Overview

Apalis Build Status

Apalis is a simple, extensible multithreaded background job processing library for Rust.

Features

  • Simple and predictable job handling model.
  • Jobs handlers with a macro free API.
  • Take full advantage of the tower ecosystem of middleware, services, and utilities.
  • Workers take full of the actor model.
  • Fully Tokio compatible.
  • Optional Web interface to help you manage your jobs.

Apalis job processing is powered by tower::Service which means you have access to the tower and tower-http middleware.

Apalis has support for

  • Redis
  • SQlite
  • PostgresSQL
  • MySQL
  • Bring Your Own Job Source eg Cron or Twitter streams

Getting Started

To get started, just add to Cargo.toml

[dependencies]
apalis = { version = "0.3.1", features = ["redis"] }

Usage

use apalis::{redis::RedisStorage, JobError, JobRequest, JobResult, WorkerBuilder, Storage, Monitor, JobContext};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Serialize)]
struct Email {
    to: String,
}

async fn email_service(job: Email, _ctx: JobContext) -> Result<JobResult, JobError> {
    Ok(JobResult::Success)
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    std::env::set_var("RUST_LOG", "debug");
    env_logger::init();
    let redis = std::env::var("REDIS_URL").expect("Missing env variable REDIS_URL");
    let storage = RedisStorage::new(redis).await.unwrap();
    Monitor::new()
        .register_with_count(2, move || {
            WorkerBuilder::new(storage.clone())
                .build_fn(email_service)
        })
        .run()
        .await
}

Then

//This can be in another part of the program or another application
async fn produce_route_jobs(storage: &RedisStorage<Email>) {
    let mut storage = storage.clone();
    storage
        .push(Email {
            to: "[email protected]".to_string(),
        })
        .await
        .unwrap();
}

Web UI

If you are running Apalis Board, you can easily manage your jobs. See a working Rest API here

UI

Feature flags

  • tracing (enabled by default) — Support Tracing 👀
  • redis — Include redis storage
  • postgres — Include Postgres storage
  • sqlite — Include SQlite storage
  • mysql — Include MySql storage
  • sentry — Support for Sentry exception and performance monitoring
  • prometheus — Support Prometheus metrics
  • retry — Support direct retrying jobs
  • timeout — Support timeouts on jobs
  • limit 💪 Limit the amount of jobs
  • filter — Support filtering jobs based on a predicate
  • extensions — Add a global extensions to jobs

Storage Comparison

Since we provide a few storage solutions, here is a table comparing them:

Feature Redis Sqlite Postgres Sled Mysql Mongo
Scheduled jobs x x
Retryable jobs x x
Persistence x x
Rerun Dead jobs x * x

Thanks to

  • tower - Tower is a library of modular and reusable components for building robust networking clients and servers.
  • redis-rs - Redis library for rust
  • sqlx - The Rust SQL Toolkit

Roadmap

v 0.4

  • Improve monitoring
  • Improve Apalis Board
  • Add job progress
  • Add more sources

v 0.3

  • Standardize API (Storage, Worker, Data, Middleware, Context )
  • Introduce SQL
  • Implement layers for Sentry and Tracing.
  • Improve documentation
  • Organized modules and features.
  • Basic Web API Interface
  • Sql Examples
  • Sqlx migrations

v 0.2

  • Redis Example
  • Actix Web Example

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

See also the list of contributors who participated in this project.

It was formerly actix-redis-jobs and if you want to use the crate name please contact me.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Acknowledgments

  • Inspiration: The redis part of this project is heavily inspired by Curlyq which is written in GoLang
Comments
  • Rerunning dead jobs in Mysql is not implemented

    Rerunning dead jobs in Mysql is not implemented

    The storage api for mysql doesnt support this feature.

        // Worker not seen in 5 minutes yet has running jobs
        StorageWorkerPulse::RenqueueOrpharned { count: _ } => {
        //...
        Ok(true)
    

    The goal is to rewrite the query at mysql.rs#L202 to be Mysql compatible and uncomment it.

    bug good first issue 
    opened by geofmureithi 5
  • s/Kill/Killed/ in storage.kill

    s/Kill/Killed/ in storage.kill

    This PR fixes a bug with killing a job.

    Bug details

    In MySQL/Postgres/Sqlite3 backends, Storage::kill(self, worker_id, job_id) sets the status to 'Kill', which is incompatible to the definition of JobStatus::Killed. https://github.com/geofmureithi/apalis/blob/3cfc8df9a9bcb288ea24c3892e2446c45247b947/packages/apalis-core/src/request.rs#L12-L26

    As a result, deserialization of job context will panic while converting Kill to JobState. https://github.com/geofmureithi/apalis/blob/3cfc8df9a9bcb288ea24c3892e2446c45247b947/packages/apalis-sql/src/from_row.rs#L53

    bug 
    opened by autotaker 1
  • s/Job/jobs/ in mysql.rs

    s/Job/jobs/ in mysql.rs

    I found a bug with table name in mysql.rs.

    In default configuration (lower_case_table_names=0) of MySQL on linux, table name are case-sensitive. The table name managing jobs should be jobs, but in some SQL it is referred as Jobs.

    As a result, this library does not work with such MySQL servers.

    With this PR, I modified every occurrence of Jobs in SQL statements with jobs.

    opened by autotaker 1
  • Mysql rerun dead jobs

    Mysql rerun dead jobs

    Closes #5

    This PR implements missing feature to re-run dead jobs for MySQL backends.

    Tests

    I added unit tests for this feature. The scenario is the following steps.

    Scenario 1

    1. Push a job
    2. Register a worker with last_seen = (6 minutes ago).
    3. Pull the job by the worker.
    4. Call heartbeat with ReenqueueOrphaned.
    5. Assert that the job status is reset to Pending.

    Scenario 2

    1. Push a job
    2. Register a worker with last_seen = (4 minutes ago).
    3. Pull the job by the worker.
    4. Call heartbeat with ReenqueueOrphaned.
    5. Assert that the job status is not changed.

    Other changes

    • Extracted method keep_alive_at(worker_id, last_seen) from keep_alive(worker_id) to improve testability
    • Modified the condition run_at < NOW() in stream_jobs to run_at <= NOW(), because jobs cannot be pulled immediately after pushed.
    • Added Test Suite with MySQL action to ci.yaml to execute unit tests with MySQL.
    opened by autotaker 0
  • V0.3

    V0.3

    • Standardize API (Storage, Worker, Data, Middleware, Context )
    • Introduce SQL
    • Implement layers for sentry and tracing.
    • Improve documentation
    • Organized modules and features.
    • Basic Web API Interface
    • Sql Examples
    opened by geofmureithi 0
  • Add Integration Tests for Postgres backend

    Add Integration Tests for Postgres backend

    Part of #9

    This PR adds Test Suite with Postgres action and implements 5 tests for Postgres backend.

    Tests

    • test_consume_last_pushed_job
    • test_acknowledge_job
    • test_kill_job
    • test_heartbeat_renqueueorphaned_pulse_last_seen_6min
    • test_heartbeat_renqueueorphaned_pulse_last_seen_4min

    Other changes

    Refactoring

    I extracted keep_alive_at(self, worker_id, last_seen) from keep_alive(self, worker_id) to improve testability.

    Changed SQL statement to upsert worker in keep_alive:

    When keep_alive_at(self, worker_id, last_seen) is called for worker_id already registered, the last_seen column of the worker row is updated with NOW() (system clock on the Postgres server). Although this behavior was trivial things, the value of last_seen can be different from the current time due to the refactoring.

    Therefore, I changed the updated value to EXCLUDED.last_seen, which is the same value as it is inserted if the worker_id is registered at first time.

     ON CONFLICT (id) DO
    -  UPDATE SET last_seen = NOW()
    +  UPDATE SET last_seen = EXCLUDED.last_seen
    
    opened by autotaker 3
  • Integration tests for each backend

    Integration tests for each backend

    Currently integrations tests are running on CI only for MySQL backend.

    Let's add integration tests for the following backends

    • [x] Mysql by @autotaker (added by #6)
    • [ ] Redis
    • [ ] Sqlite
    • [ ] Postgres

    Let's strive to have different pull requests for each storage. This will help in collaboration. Also before starting on any backend, please add a comment and assigned.

    enhancement 
    opened by autotaker 5
  • Add documentation and examples of how workers process jobs

    Add documentation and examples of how workers process jobs

    Currently it only shows one worker send_email. Would be good to showcase more types of jobs being consumed.

        Monitor::new()
            .register_with_count(5, move |_| {
                WorkerBuilder::new(sqlite.clone())
                    .layer(TraceLayer::new())
                    .build_fn(send_email)
            })
            .run()
            .await
    
    documentation 
    opened by prabirshrestha 3
Releases(v0.3.4)
  • v0.3.4(Sep 20, 2022)

    What's Changed

    • s/Job/jobs/ in mysql.rs by @autotaker in https://github.com/geofmureithi/apalis/pull/4
    • Mysql rerun dead jobs by @autotaker in https://github.com/geofmureithi/apalis/pull/6
    • Bump up to 0.3.4 by @geofmureithi in https://github.com/geofmureithi/apalis/pull/7

    New Contributors

    • @autotaker made their first contribution in https://github.com/geofmureithi/apalis/pull/4

    Full Changelog: https://github.com/geofmureithi/apalis/compare/0.3.3...v0.3.4

    Source code(tar.gz)
    Source code(zip)
  • 0.3.3(Aug 13, 2022)

  • v0.3.1(Jul 9, 2022)

    • Introduced Postgres helpers for adding jobs in sql.
    • Improved documentation and Api

    What's Changed

    • Chore/clippy n improvements by @geofmureithi in https://github.com/geofmureithi/apalis/pull/2

    Full Changelog: https://github.com/geofmureithi/apalis/compare/v0.3.0...v0.3.1

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jun 5, 2022)

Owner
For my old account see @geofmureithi-zz
null
A CI inspired approach for local job automation.

nauman A CI inspired approach for local job automation. Features • Installation • Usage • FAQ • Examples • Job Syntax About nauman is an easy-to-use j

Egor Dmitriev 6 May 8, 2022
A simple, lightweight and extensible command line argument parser for rust codebases

A simple, lightweight and extensible command line argument parser for rust codebases. This crate aims to provide you with an easy-to-use and extensibl

Victor Ndaba 19 Jul 1, 2022
Rust Imaging Library's Python binding: A performant and high-level image processing library for Python written in Rust

ril-py Rust Imaging Library for Python: Python bindings for ril, a performant and high-level image processing library written in Rust. What's this? Th

Cryptex 5 Sep 12, 2022
skyWM is an extensible tiling window manager written in Rust. skyWM has a clear and distinct focus adhering to the KISS and Unix philosophy.

Please note: skyWM is currently in heavy development and is not usable as of yet. Documentation and versions will change quickly. skyWM skyWM is an ex

MrBeeBenson 62 Sep 19, 2022
Configurable, extensible, interactive line reader

linefeed linefeed is a configurable, concurrent, extensible, interactive input reader for Unix terminals and Windows console. API Documentation linefe

Murarth 175 Aug 10, 2022
Concurrent and multi-stage data ingestion and data processing with Rust+Tokio

TokioSky Build concurrent and multi-stage data ingestion and data processing pipelines with Rust+Tokio. TokioSky allows developers to consume data eff

DanyalMh 23 Sep 9, 2022
Allows processing of iterators of Result types

try-continue try-continue provides one method, try_continue, which allows you to work with iterators of type Result<T, _>, as if they were simply iter

Nick Krichevsky 3 Dec 26, 2021
a simple program that you can scrap, is shit and really simple but is cool.

if you want to run it you need to have installed curl by default scrap youtube, but you can change it, also change the number of threads and run: carg

pai 7 Oct 15, 2021
A dead simple ANSI terminal color painting library for Rust.

yansi A dead simple ANSI terminal color painting library for Rust. use yansi::Paint; print!("{} light, {} light!", Paint::green("Green"), Paint::red(

Sergio Benitez 155 Sep 24, 2022
Simple console input macros with the goal of being implemented in the standard library.

Simple console input macros with the goal of being implemented in the standard library.

undersquire 2 Feb 10, 2022
(Rust) Coloring terminal so simple you already know how to do it !

Colored Coloring terminal so simple, you already know how to do it! "this is blue".blue(); "this is red".red(); "this is red on blue".red(

Thomas Wickham 1.1k Sep 22, 2022
A simple CLI pomodoro timer written in Rust.

Pomodoro A simple CLI pomodoro timer written in Rust. Based on the Pomodoro Technique. Works on any platform that supports desktop notifications. Exam

null 6 Nov 26, 2021
Simple Interactive Terminal Todo App in Rust

todo-rs Simple Interactive Terminal Todo App in Rust Quick Start $ cargo run TODO Controls Keys Description k, j Move cursor up and down Shift+K, Shif

Tsoding 47 Sep 19, 2022
Simple test app based on rust-psp

PSP Test App Simple test app based on rust-psp. Demonstrating the usage of C libs. Build Download and unzip the prebuilt PSPSDK (built from clang-psp)

Yifeng Wang 3 Nov 17, 2021
Simple cli clipboard manager written in rust

Simple cli clipboard manager written in rust

null 5 May 31, 2022
Simple system monitoring app that runs on terminal. Made purely with Rust.

What is it? RCTOP is a simple WIP system monitoring app that runs purely on terminal and doesn't feature GUI. One can compare it to htop, but more str

Niko Huuskonen 6 Nov 12, 2021
My solutions for the 2021 edition of the Advent of Code, using Rust and SOM (Simple Object Machine)

Advent of Code 2021 These are my solutions for the 2021 edition of the Advent of Code. The solutions are all implemented using both Rust and SOM (Simp

Nicolas Polomack 1 Dec 23, 2021
Simple command line flag parser for rust.

easy_flag Simple command line flag parser for rust. use easy_flag::FlagSet; fn main() -> Result<(), String>{ let mut help = false; let mut my

BillyfBrain 3 Oct 20, 2021
A simple, fast and interruptable download accelerator, written in Rust

snatch A simple, fast and interruptable download accelerator, written in Rust WARNING This project is no longer maintained by @k0pernicus and @jean-se

Dernier Cri ® 650 Aug 29, 2022