Fang
Background job processing library for Rust.
Currently, it uses Postgres to store state. But in the future, more backends will be supported.
Installation
- Add this to your Cargo.toml
[dependencies]
fang = "0.3"
typetag = "0.1"
serde = { version = "1.0", features = ["derive"] }
- Create
fang_taskstable in the Postgres database. The migration can be found in the migrations directory.
Usage
Defining a job
Every job should implement fang::Runnable trait which is used by fang to execute it.
use fang::Error;
use fang::Runnable;
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
struct Job {
pub number: u16,
}
#[typetag::serde]
impl Runnable for Job {
fn run(&self) -> Result<(), Error> {
println!("the number is {}", self.number);
Ok(())
}
}
As you can see from the example above, the trait implementation has #[typetag::serde] attribute which is used to deserialize the job.
Enqueuing a job
To enqueue a job use Postgres::enqueue_task
use fang::Postgres;
...
Postgres::enqueue_task(&Job { number: 10 }).unwrap();
The example above creates a new postgres connection on every call. If you want to reuse the same postgres connection to enqueue several jobs use Postgres struct instance:
let postgres = Postgres::new();
for id in &unsynced_feed_ids {
postgres.push_task(&SyncFeedJob { feed_id: *id }).unwrap();
}
Starting workers
Every worker runs in a separate thread. In case of panic, they are always restarted.
Use WorkerPool to start workers. WorkerPool::new accepts one parameter - the number of workers.
use fang::WorkerPool;
WorkerPool::new(10).start();
Configuration
To configure workers, instead of WorkerPool::new which uses default values, use WorkerPool.new_with_params. It accepts two parameters - the number of workers and WorkerParams struct.
Configuring the type of workers
You can start workers for a specific types of tasks. These workers will be executing only tasks of the specified type.
Add task_type method to the Runnable trait implementation:
...
#[typetag::serde]
impl Runnable for Job {
fn run(&self) -> Result<(), Error> {
println!("the number is {}", self.number);
Ok(())
}
fn task_type(&self) -> String {
"number".to_string()
}
}
Set task_type to the WorkerParamas:
let mut worker_params = WorkerParams::new();
worker_params.set_task_type("number".to_string());
WorkerPool::new_with_params(10, worker_params).start();
Without setting task_type workers will be executing any type of task.
Configuring retention mode
By default, all successfully finished tasks are removed from the DB, failed tasks aren't.
There are three retention modes you can use:
pub enum RetentionMode {
KeepAll, \\ doesn't remove tasks
RemoveAll, \\ removes all tasks
RemoveFinished, \\ default value
}
Set retention mode with set_retention_mode:
let mut worker_params = WorkerParams::new();
worker_params.set_retention_mode(RetentionMode::RemoveAll);
WorkerPool::new_with_params(10, worker_params).start();
Configuring sleep values
You can use use SleepParams to confugure sleep values:
pub struct SleepParams {
pub sleep_period: u64, \\ default value is 5
pub max_sleep_period: u64, \\ default value is 15
pub min_sleep_period: u64, \\ default value is 5
pub sleep_step: u64, \\ default value is 5
}p
If there are no tasks in the DB, a worker sleeps for sleep_period and each time this value increases by sleep_step until it reaches max_sleep_period. min_sleep_period is the initial value for sleep_period. All values are in seconds.
Use set_sleep_params to set it:
let sleep_params = SleepParams {
sleep_period: 2,
max_sleep_period: 6,
min_sleep_period: 2,
sleep_step: 1,
};
let mut worker_params = WorkerParams::new();
worker_params.set_sleep_params(sleep_params);
WorkerPool::new_with_params(10, worker_params).start();
Potential/future features
- Retries
- Scheduled tasks
- Extendable/new backends
Contributing
- Fork it!
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
Author
Ayrat Badykov (@ayrat555)
100 Dec 26, 2022
24 Dec 26, 2022
529 Jan 9, 2023
13 May 30, 2022
50 Jan 2, 2023
23 Nov 25, 2023
294 Dec 23, 2022
22 Mar 27, 2023
3 Nov 30, 2023
80 Sep 8, 2022
3 Mar 25, 2024
85 Jan 5, 2023
47 Dec 6, 2022
8 Oct 24, 2022
0 Feb 4, 2022
6 Nov 8, 2022
3 Aug 31, 2022
20 Oct 29, 2022
15 Apr 23, 2023
13 Dec 6, 2022