Factotum
A dag running tool designed for efficiently running complex jobs with non-trivial dependency trees.
The zen of Factotum
- A Turing-complete job is not a job, it's a program
- A job must be composable from other jobs
- A job exists independently of any job schedule
User quickstart
Assuming you're running 64 bit Linux:
wget https://github.com/snowplow/factotum/releases/download/0.6.0/factotum_0.6.0_linux_x86_64.zip
unzip factotum_0.6.0_linux_x86_64.zip
./factotum --version
Factotum requires one argument, which is a factotum factfile that describes the job to run. For example, to run the sample sleep.factfile:
wget https://raw.githubusercontent.com/snowplow/factotum/master/samples/sleep.factfile
./factotum run sleep.factfile
Specifying variables in the job file can be done using --env JSON (or -e JSON). The JSON here is free-form and needs to correspond to the placeholders you've set in your job.
For example, the following will print "hello world!":
wget https://raw.githubusercontent.com/snowplow/factotum/master/samples/variables.factfile
./factotum run variables.factfile --env '{ "message": "hello world!" }'
Starting from an arbitrary task can be done using the --start TASK or -s TASK arguments, where TASK is the name of the task you'd like to start at.
For example, to start at the "echo beta" task in this job, you can run the following:
wget https://raw.githubusercontent.com/snowplow/factotum/master/samples/echo.factfile
./factotum run echo.factfile --start "echo beta"
To get a quick overview of the options provided, you can use the --help or -h argument:
./factotum --help
For more information on this file format and how to write your own jobs, see the Factfile format section below.
Factfile format
Factfiles are self-describing JSON which declare a series of tasks and their dependencies. For example:
{
"schema": "iglu:com.snowplowanalytics.factotum/factfile/jsonschema/1-0-0",
"data": {
"name": "Factotum demo",
"tasks": [
{
"name": "echo alpha",
"executor": "shell",
"command": "echo",
"arguments": [ "alpha" ],
"dependsOn": [],
"onResult": {
"terminateJobWithSuccess": [],
"continueJob": [ 0 ]
}
},
{
"name": "echo beta",
"executor": "shell",
"command": "echo",
"arguments": [ "beta" ],
"dependsOn": [ "echo alpha" ],
"onResult": {
"terminateJobWithSuccess": [],
"continueJob": [ 0 ]
}
},
{
"name": "echo omega",
"executor": "shell",
"command": "echo",
"arguments": [ "and omega!" ],
"dependsOn": [ "echo beta" ],
"onResult": {
"terminateJobWithSuccess": [],
"continueJob": [ 0 ]
}
}
]
}
}
This example defines three tasks that run shell commands - echo alpha, echo beta and echo omega. echo alpha has no dependencies - it will run immediately. echo beta depends on the completion of the echo alpha task, and so will wait for echo alpha to complete. echo omega depends on the echo beta task, and so will wait for echo beta to be complete before executing.
Given the above, the tasks will be executed in the following sequence: echo alpha, echo beta and finally, echo omega. Tasks can have multiple dependencies in factotum, and tasks that are parallelizable will be run concurrently. Check out the samples for more sample factfiles or the wiki for a more complete description of the factfile format.
Developer quickstart
Factotum is written in Rust.
Using Vagrant
- Clone this repository -
git clone [email protected]:snowplow/factotum.git cd factotum- Set up a Vagrant box and ssh into it -
vagrant up && vagrant ssh- This will take a few minutes
cd /vagrant- Compile and run a demo -
cargo run -- run samples/echo.factfile
Using stable Rust without Vagrant
- Install Rust
- on Linux/Mac -
curl -sSf https://static.rust-lang.org/rustup.sh | sh
- on Linux/Mac -
- Clone this repository -
git clone [email protected]:snowplow/factotum.git cd factotum- Compile and run a demo -
cargo run -- run samples/echo.factfile
Copyright and license
Factotum is copyright 2016-2021 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
739 Jan 7, 2023
0 Oct 29, 2021
6 Dec 15, 2022
375 Jan 2, 2023
1.3k Jan 8, 2023
41 Nov 10, 2022
705 Dec 27, 2022
8 Sep 19, 2022
43 Jan 5, 2023
18 Aug 3, 2023
47 Oct 6, 2023
39 May 7, 2024
3 Sep 30, 2022
75 Dec 1, 2022
2 Apr 5, 2022
88 Nov 6, 2022
10 Dec 30, 2022
3 Jan 10, 2022
9 Apr 10, 2022
1 Apr 15, 2022