Containerize your development and continuous integration environments. 🥂

Overview

Toast 🥂

Build status

Toast is a tool for doing work in containers. You define tasks in a YAML file called a toastfile, and Toast runs them in a containerized environment based on a Docker image of your choosing. What constitutes a "task" is up to you: tasks can install system packages, build an application, run a test suite, or even serve web pages. Tasks can depend on other tasks, so Toast can be understood as a high-level containerized build system.

Welcome to Toast.

Here's the toastfile for the example shown above:

image: ubuntu
tasks:
  install_gcc:
    command: |
      apt-get update
      apt-get install --yes gcc

  build:
    dependencies:
      - install_gcc
    input_paths:
      - main.c
    command: gcc main.c

  run:
    dependencies:
      - build
    command: ./a.out

Toast caches each task by committing the container to an image. The image is tagged with a cryptographic hash of the shell command for the task, the contents of the files copied into the container, and all the other task inputs. This hash allows Toast to skip tasks that haven't changed since the last run.

In addition to local caching, Toast can use a Docker registry as a remote cache. You, your teammates, and your continuous integration (CI) system can all share the same remote cache. Used in this way, your CI system can do all the heavy lifting like building and installing dependencies so you and your team can focus on development.

Related tools:

  • Docker Compose: Docker Compose is a convenient Docker-based development environment which shares many features with Toast. However, it doesn't support defining tasks (like lint, test, run, etc.) or remote caching.
  • Nix: Nix achieves reproducible builds by leveraging ideas from functional programming rather than containerization. We're big fans of Nix. However, Nix requires a larger commitment compared to Toast because you have to use the Nix package manager or write your own Nix derivations. For better or worse, Toast allows you to use familiar idioms like apt-get install ....

To prevent Docker images from accumulating on your machine when using Docker-related tools such as Toast or Docker Compose, we recommend using Docuum to perform least recently used (LRU) image eviction.

Tutorial

Defining a simple task

Let's create a toastfile. Create a file named toast.yml with the following contents:

image: ubuntu
tasks:
  greet:
    command: echo 'Hello, World!' # Toast will run this in a container.

Now run toast. You should see the following:

Defining a simple task.

If you run it again, Toast will find that nothing has changed and skip the task:

Caching.

Toast caches tasks to save you time. For example, you don't want to reinstall your dependencies every time you run your tests. However, caching may not be appropriate for some tasks, like running a development server. You can disable caching for a specific task and all tasks that depend on it with the cache option:

image: ubuntu
tasks:
  greet:
    cache: false # Don't cache this task.
    command: echo 'Hello, World!'

Adding a dependency

Let's make the greeting more fun with a program called figlet. We'll add a task to install figlet, and we'll change the greet task to depend on it:

image: ubuntu
tasks:
  install_figlet:
    command: |
      apt-get update
      apt-get install --yes figlet

  greet:
    dependencies:
      - install_figlet # Toast will run this task first.
    command: figlet 'Hello, World!'

Run toast to see a marvelous greeting:

Adding a dependency.

Importing files from the host

Here's a more realistic example. Suppose you want to compile and run a simple C program. Create a file called main.c:

#include <stdio.h>

int main(void) {
  printf("Hello, World!\n");
}

Update toast.yml to compile and run the program:

image: ubuntu
tasks:
  install_gcc:
    command: |
      apt-get update
      apt-get install --yes gcc

  build:
    dependencies:
      - install_gcc
    input_paths:
      - main.c # Toast will copy this file into the container before running the command.
    command: gcc main.c

  run:
    dependencies:
      - build
    command: ./a.out

Notice the input_paths array in the build task. Here we're copying a single file into the container, but we could instead import the entire directory containing the toastfile with .. By default, the files will be copied into a directory called /scratch in the container. The commands will be run in that directory as well.

Now if you run toast, you'll see this:

Importing files from the host.

For subsequent runs, Toast will skip the task if nothing has changed. But if you update the greeting in main.c, Toast will detect the change and rerun the build and run tasks on the next invocation.

Exporting files from the container

A common use case for Toast is to build a project. Naturally, you might wonder how to access the build artifacts produced inside the container from the host machine. It's easy to do with output_paths:

image: ubuntu
tasks:
  install_gcc:
    command: |
      apt-get update
      apt-get install --yes gcc

  build:
    dependencies:
      - install_gcc
    input_paths:
      - main.c
    output_paths:
      - a.out # Toast will copy this file onto the host after running the command.
    command: gcc main.c

When Toast runs the build task, it will copy the a.out file to the host.

Exporting files from the container.

Passing arguments to a task

Sometimes it's useful for tasks to take arguments. For example, a deploy task might want to know whether you want to deploy to the staging or production cluster. To do this, add an environment section to your task:

image: ubuntu
tasks:
  deploy:
    cache: false
    environment:
      CLUSTER: staging # Deploy to staging by default.
    command: echo "Deploying to $CLUSTER..."

When you run this task, Toast will read the value from the environment:

Passing arguments to a task.

If the variable doesn't exist in the environment, Toast will use the default value:

Using argument defaults.

If you don't want to have a default, set it to null:

image: ubuntu
tasks:
  deploy:
    cache: false
    environment:
      CLUSTER: null # No default; this variable must be provided at runtime.
    command: echo "Deploying to $CLUSTER..."

Now if you run toast deploy without specifying a CLUSTER, Toast will complain about the missing variable and refuse to run the task.

Environment variables listed in a task are also set for any tasks that run after it.

Running a server and mounting paths into the container

Toast can be used for more than just building a project. Suppose you're developing a website. You can define a Toast task to run your web server! Create a file called index.html with the following contents:

<!DOCTYPE html>
<html>
  <head>
    <title>Welcome to Toast!</title>
  </head>
  <body>
    <p>Hello, World!</p>
  </body>
</html>

We can use a web server like nginx. The official nginx Docker image will do, but you could also use a more general image and define a Toast task to install nginx.

In our toast.yml file, we'll use the ports field to make the website accessible outside the container. We'll also use mount_paths rather than input_paths to synchronize files between the host and the container while the server is running.

image: nginx
tasks:
  serve:
    cache: false # It doesn't make sense to cache this task.
    mount_paths:
      - index.html # Synchronize this file between the host and the container.
    ports:
      - 3000:80 # Expose port 80 in the container as port 3000 on the host.
    location: /usr/share/nginx/html/ # Nginx will serve the files in here.
    command: nginx -g 'daemon off;' # Run in foreground mode.

Now you can use Toast to run the server:

Running a server.

Configuring the shell

It's often desirable to configure the shell in some way before running any commands. Shells are typically configured with so-called "startup files" (e.g., ~/.bashrc). However, many shells skip loading such configuration files when running in non-interactive, non-login mode, which is how the shell is invoked by Toast. Toast provides an alternative mechanism to configure the shell that doesn't require creating any special files or invoking the shell in a particular way.

Consider the following toastfile which uses Bash as the shell, since that's the default preferred login shell in Ubuntu:

image: ubuntu
tasks:
  install_figlet:
    command: |
      apt-get update
      apt-get install --yes figlet

What happens if apt-get update fails? Due to the way Bash works, the failure would be ignored and execution would continue to the subsequent line. You can fix this with set -e as follows:

image: ubuntu
tasks:
  install_figlet:
    command: |
      set -e # Make Bash fail fast.
      apt-get update
      apt-get install --yes figlet

However, it's tedious and error-prone to add that to each task separately. Instead, you can add it to every task at once by setting command_prefix as follows:

image: ubuntu
command_prefix: set -e # Make Bash fail fast.
tasks:
  install_figlet:
    command: |
      apt-get update
      apt-get install --yes figlet

For Bash in particular, we recommend going even further and setting set -euo pipefail instead of just set -e.

Dropping into an interactive shell

If you run Toast with --shell, Toast will drop you into an interactive shell inside the container when the requested tasks are finished, or if any of them fails. This feature is useful for debugging tasks or exploring what's in the container. Suppose you have the following toastfile:

image: ubuntu
tasks:
  install_figlet:
    command: |
      apt-get update
      apt-get install --yes figlet

You can run toast --shell to play with the figlet program:

Dropping into a shell.

When you're done, the container is deleted automatically.

How Toast works

Given a set of tasks to run, Toast computes a topological sort of the dependency DAG to determine in what order to run the tasks. Toast then builds a Docker image for each task based on the image from the previous task in the topological sort, or the base image in the case of the first task.

The topological sort of an arbitrary DAG isn't necessarily unique. Toast uses an algorithm based on depth-first search, traversing children in lexicographical order. The algorithm is deterministic and invariant to the order in which tasks and dependencies are listed, so reordering tasks in a toastfile won't invalidate the cache. Furthermore, toast foo bar and toast bar foo are guaranteed to produce identical schedules to maximize cache utilization.

For each task in the schedule, Toast first computes a cache key based on a hash of the shell command, the contents of the input_paths, the cache key of the previous task in the schedule, etc. Toast will then look for a Docker image tagged with that cache key. If the image is found, Toast will skip the task. Otherwise, Toast will create a container, copy any input_paths into it, run the shell command, copy any output_paths from the container to the host, commit the container to an image, and delete the container. The image is tagged with the cache key so the task can be skipped for subsequent runs.

Toast aims to make as few assumptions about the container environment as possible. Toast only assumes there is a program at /bin/su which can be invoked as su -c COMMAND USER. This program is used to run commands for tasks in the container as the appropriate user with their preferred shell. Every popular Linux distribution has a su utility that supports this usage. Toast has integration tests to ensure it works with popular base images such as debian, alpine, busybox, etc.

Toastfiles

A toastfile is a YAML file (typically named toast.yml) that defines tasks and their dependencies. The schema contains the following top-level keys and defaults:

image: <required>   # Docker image name with optional tag or digest
default: null       # Name of default task to run or `null` to run all tasks by default
location: /scratch  # Path in the container for running tasks
user: root          # Name of the user in the container for running tasks
command_prefix: ''  # A string to be prepended to all commands by default
tasks: {}           # Map from task name to task

Tasks have the following schema and defaults:

description: null           # A description of the task for the `--list` option
dependencies: []            # Names of dependencies
cache: true                 # Whether a task can be cached
environment: {}             # Map from environment variable to optional default
input_paths: []             # Paths to copy into the container
excluded_input_paths: []    # A denylist for `input_paths`
output_paths: []            # Paths to copy out of the container if the task succeeds
output_paths_on_failure: [] # Paths to copy out of the container if the task fails
mount_paths: []             # Paths to mount into the container
mount_readonly: false       # Whether to mount the `mount_paths` as readonly
ports: []                   # Port mappings to publish
location: null              # Overrides the corresponding top-level value
user: null                  # Overrides the corresponding top-level value
command: ''                 # Shell command to run in the container
command_prefix: null        # Overrides the corresponding top-level value
extra_docker_arguments: []  # Additional arguments for `docker container create`

The toastfile for Toast itself is a comprehensive real-world example.

Configuration

Toast can be customized with a YAML configuration file. The default location of the configuration file depends on the operating system:

  • For macOS, the default location is $HOME/Library/Application Support/toast/toast.yml.
  • For other Unix platforms, Toast follows the XDG Base Directory Specification. The default location is $XDG_CONFIG_HOME/toast/toast.yml or $HOME/.config/toast/toast.yml if XDG_CONFIG_HOME isn't set to an absolute path.
  • For Windows, the default location is {FOLDERID_RoamingAppData}\toast\toast.yml.

The schema of the configuration file is described in the subsections below.

Cache configuration

Toast supports local and remote caching. By default, only local caching is enabled. Remote caching requires that the Docker Engine is logged into a Docker registry (e.g., via docker login).

The cache-related fields and their default values are as follows:

docker_repo: toast        # Docker repository
read_local_cache: true    # Whether Toast should read from local cache
write_local_cache: true   # Whether Toast should write to local cache
read_remote_cache: false  # Whether Toast should read from remote cache
write_remote_cache: false # Whether Toast should write to remote cache

Each of these options can be overridden via command-line options (see below).

A typical configuration for a CI environment will enable all forms of caching, whereas for local development you may want to set write_remote_cache: false to avoid waiting for remote cache writes.

Docker CLI

You can configure the Docker CLI binary used by Toast. Toast uses the PATH environment variable to search for the specified binary. You can use this mechanism to switch to a drop-in replacement for the Docker CLI, such as Podman.

The relevant field and its default value are as follows:

docker_cli: docker

Command-line options

By default, Toast looks for a toastfile called toast.yml in the working directory, then in the parent directory, and so on. Any paths in the toastfile are relative to where the toastfile lives, not the working directory. This means you can run Toast from anywhere in your project and get the same results.

Run toast with no arguments to execute the default task, or all the tasks if the toastfile doesn't define a default. You can also execute specific tasks and their dependencies:

toast task1 task2 task3…

Here are all the supported command-line options:

USAGE:
    toast [OPTIONS] [--] [TASKS]...

OPTIONS:
    -c, --config-file <PATH>
            Sets the path of the config file

        --docker-cli <CLI>
            Sets the Docker CLI binary

    -r, --docker-repo <REPO>
            Sets the Docker repository

    -f, --file <PATH>
            Sets the path to the toastfile

        --force <TASK>...
            Runs a task unconditionally, even if it’s cached

    -h, --help
            Prints help information

    -l, --list
            Lists the tasks that have a description

        --read-local-cache <BOOL>
            Sets whether local cache reading is enabled

        --read-remote-cache <BOOL>
            Sets whether remote cache reading is enabled

    -s, --shell
            Drops you into a shell after the tasks are finished

    -v, --version
            Prints version information

        --write-local-cache <BOOL>
            Sets whether local cache writing is enabled

        --write-remote-cache <BOOL>
            Sets whether remote cache writing is enabled


ARGS:
    <TASKS>...
            Sets the tasks to run

Installation instructions

Installation on macOS or Linux (x86-64)

If you're running macOS or Linux on an x86-64 CPU, you can install Toast with this command:

curl https://raw.githubusercontent.com/stepchowfun/toast/main/install.sh -LSfs | sh

The same command can be used again to update to the latest version.

The installation script supports the following optional environment variables:

  • VERSION=x.y.z (defaults to the latest version)
  • PREFIX=/path/to/install (defaults to /usr/local/bin)

For example, the following will install Toast into the working directory:

curl https://raw.githubusercontent.com/stepchowfun/toast/main/install.sh -LSfs | PREFIX=. sh

If you prefer not to use this installation method, you can download the binary from the releases page, make it executable (e.g., with chmod), and place it in some directory in your PATH (e.g., /usr/local/bin).

Installation on Windows (x86-64)

If you're running Windows on an x86-64 CPU, download the latest binary from the releases page and rename it to toast (or toast.exe if you have file extensions visible). Create a directory called Toast in your %PROGRAMFILES% directory (e.g., C:\Program Files\Toast), and place the renamed binary in there. Then, in the "Advanced" tab of the "System Properties" section of Control Panel, click on "Environment Variables..." and add the full path to the new Toast directory to the PATH variable under "System variables". Note that the Program Files directory might have a different name if Windows is configured for a language other than English.

To update to an existing installation, simply replace the existing binary.

Installation with Homebrew

If you have Homebrew, you can install Toast as follows:

brew install toast

You can update an existing installation with brew upgrade toast.

Installation with Cargo

If you have Cargo, you can install Toast as follows:

cargo install toast

You can run that command with --force to update an existing installation.

Running Toast in CI

The easiest way to run Toast in CI is to use GitHub Actions. Toast provides a convenient GitHub action that you can use in your workflows. Here's a simple workflow that runs Toast with no arguments:

# .github/workflows/ci.yml
name: Continuous integration
on: [push, pull_request]
jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/[email protected]
    - uses: stepchowfun/toast/.github/actions/[email protected]

Here's a more customized workflow that showcases all the options:

# .github/workflows/ci.yml
name: Continuous integration
on: [push, pull_request]
jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/[email protected]
    - uses: azure/[email protected]
      with:
        username: DOCKER_USERNAME
        password: ${{ secrets.DOCKER_PASSWORD }}
      if: github.event_name == 'push'
    - uses: stepchowfun/toast/.github/actions/[email protected]
      with:
        file: toastfiles/toast.yml
        tasks: build lint test
        docker_repo: DOCKER_USERNAME/DOCKER_REPO
        write_remote_cache: ${{ github.event_name == 'push' }}

Requirements

  • Toast requires Docker Engine 17.06.0 or later.
  • Toast only works with Linux containers; Windows containers aren't currently supported. However, in addition to Linux hosts, Toast also supports macOS and Windows hosts with the appropriate virtualization capabilities thanks to Docker Desktop.

Acknowledgements

Toast was inspired by an in-house tool used at Airbnb for CI jobs. The design was heavily influenced by the lessons I learned working on that tool and building out Airbnb's CI system with the fabulous CI Infrastructure Team.

Special thanks to Julia Wang (@juliahw) for valuable early feedback. Thanks to her and Mark Tai (@marktai) for coming up with the name Toast.

The terminal animations were produced with asciinema and svg-term-cli.

Comments
  • FEATURE REQUEST Windows support

    FEATURE REQUEST Windows support

    Description A clear description of what you want to happen. One of the benefits of containerization of dev envs is being able to run them in a platform agnostic way, so it'd be cool if toast could run on Windows

    Alternatives considered A clear description of any alternative solutions or features you've considered.

    Additional context Add any other context about the feature request here. I'd love to help make this happen.

    enhancement 
    opened by alex-kattathra-johnson-zz 13
  • Add generic docker extra arguments option

    Add generic docker extra arguments option

    Description Add an option to toastfile so that we can add arbitrary extra docker arguments

    Additional context I find that I cannot add --devices and --privileged etc. with current toastfile definition. But they are needed for example when we want to use GPU and other devices within the container. Adding an option to support any docker arguments can be a general solution to solve such problems.

    enhancement 
    opened by NOBLES5E 11
  • Hidden Tasks

    Hidden Tasks

    Description A way to hide a task from --list output. I'm proposing a task option hidden: bool which hides said task from the output of --list

    Alternatives considered

    Hard coding a convention such as a leading underscore (i.e. _foo:) being hidden. This has more downsides as it's less clear than hidden: bool.

    Additional context

    We often had out Toast files to people not familiar with the project they need to build, or even familiar with Toast. For the most part we can get by with just using default tasks, however there are some Toast files where we need to run different target tasks. In these cases users know to use --list to see what options are available. However, many of the tasks shouldn't be used as output targets, and are used as glue tasks. Currently we solve this through convention of a leading underscore (_) means ignore/don't use a task target. We'd prefer being able to hide such tasks from the output of --list entirely.

    enhancement 
    opened by kbknapp 10
  • Remote execution via DOCKER_HOST incurs 50x ping latency.

    Remote execution via DOCKER_HOST incurs 50x ping latency.

    First, let me say that Toast is really well done! I haven't done anything with Docker/containers before, but the Toast README is succinct and clear, and I'm impressed with how you've kept the scope focused. Nice job, and thanks for sharing it with us!

    I'm investigating using Toast from a Mac to run tasks on a remote backend via the DOCKER_HOST env var. However, doing so introduces substantial latency beyond what I'd expect from network overhead. It could be that this is entirely in the Docker engine and out of scope for Toast, but I figured it'd be worth raising here in case anyone has a similar use case and possible suggestion for how to improve the situation.

    I'm testing by running this minimal toast.yml:

    image: ubuntu:[email protected]:c01a10535d9ea3e4065dee4abdb288821b2918929f1e1584287a03e731b9176c
    command_prefix: set -euo pipefail
    tasks:
      hello:
        cache: false
        command: echo "hello world" > my_output
        output_paths:
          - my_output
    

    and all of the numbers below refer to Docker version 20.10.13, build a224086 on an EC2 ARM machine with Toast 0.45.5. The only difference is whether I invoke toast from a terminal on the EC2 machine or from my Mac. Ping to the EC2 machine is 20--30 ms.

    SSH has a 2s login overhead, so it's out of the question. I'm using TCP instead, here's the command I'm using from my Mac:

    LOG_LEVEL=debug DOCKER_HOST=tcp://containertron:2375 toast hello 2>&1 | ts "[%.S]"
    

    (The ts adds a microsecond timestamp to each printed log line.)

    Here's a comparison between execution from the Mac and from a terminal on the server. Note that the remote execution via DOCKER_HOST takes about 1s longer than a local execution on the EC2 server:

    Screen Shot 2022-07-25 at 10 42 59 PM

    Again, I have no experience with docker, but can field a few guesses:

    • TCP connection overhead
    • a bunch of 25 ms round-trips between the Docker CLI and the engine
    • protocol overhead (if Docker chats something different over TCP than UNIX socket?)
    • DNS resolution overhead across Toast's multiple invocations of docker

    Do you have any suggestions for things to investigate to improve the latency here? Thanks!

    bug 
    opened by lynaghk 7
  • Invalidate Cache from task on

    Invalidate Cache from task on

    Description

    It would be great to be able to invalidate the local cache starting from a specific task, and all tasks that depend on it. i.e. I have several build toast files and would like to be able to invalidate from cloning the repository on, but don't really want to invalidate things like installing deps, setup, etc. cloning tasks naturally don't invalidate themselves because the command is the exact same, even if commits have been happening on the remote repo.

    Alternatives considered

    A YAML key like,

    local_cache_valid_if: |
       cmds_that_return 0
       for_cache_is_valid
       or_return_1 if_invalid_cache
    

    Also, the current using --read-local-cache false which invalidates everything but is less than desirable.

    Additional context

    None

    enhancement 
    opened by kbknapp 7
  • Windows host initial support

    Windows host initial support

    MVP for Windows host support. Works with Docker Toolbox (via VirtualBox) on Windows 10 Home. Untested with modern Docker for Windows (requires Windows 10 Pro and/or Insider Preview). All tests are passing, both in Windows host (cargo test) and inside Linux guest (cargo run -- test), but otherwise this PR is a draft, not suitable for production. It works for my workflow, but didn't touch anything related to mounting (but tests are passing, so idk) and remote caching.

    XR6pJVW81x

    Status: Sanity check required

    Fixes: #262

    opened by qthree 6
  • Support for excluded input paths

    Support for excluded input paths

    Description I'm working on project where I need to import the entire content of the folder except for some files that must not exist in the container. I think it might be useful to support something like excluded_input_paths to exclude some folders/files.

    Alternatives considered The only alternatives that come to mind are:

    • declare any necessary path in input_paths
    • add a task to remove unnecessary paths
    enhancement 
    opened by lorenzomar 5
  • Support for per task images

    Support for per task images

    Description While looking into alternative CI solutions that could use a single config to run task locally as well as in the CI pipeline I found toast, great work!

    What I do find myself missing to replace the current CI setup for a couple of client projects is support for using a specific image per task. This is possible in GitLab CI and is quite a nice feature. I guess some sort of "artifact" sharing would be nice in this scenario but as this is based on docker I'm thinking that sharing a volume between the tasks would work.

    Alternatives considered The only alternative I can think of is using multiple toastfiles. However, with that approach, running tasks locally wouldn't be as easy. I guess it's a DX consideration.

    enhancement 
    opened by zimme 5
  • Allow `input_paths` to be absolute

    Allow `input_paths` to be absolute

    Hi there. I'm just starting to play with this and it's a super interesting project, so thank you for this!!!

    I was trying to run the aws cli in a task and would like to copy in my ~/.aws/credentials.

    Any reason not to allow input_paths to be absolute? similar to #268

    enhancement 
    opened by jritsema 5
  • add a way to skip container commit

    add a way to skip container commit

    Description We currently use toast to run some codegen inside a container.

    The codegen reads in a lot of files as input, but spits out only one or two files as output.

    When the codegen is done, copying the output paths out is fast. Then toast spends an order of magnitude more time on the "Committing container..." step

    Is there a way we could add an option skip that step?

    Alternatives considered I also wonder if Toast could automatically skip the container commit if it can figure out that it won't be used. But an option would also be fine.

    Additional context None, thanks for a great tool!

    enhancement 
    opened by nicks 5
  • [ERROR] Error appending data to tar archive. Reason: paths in archives must not have `..` when setting path for scratch

    [ERROR] Error appending data to tar archive. Reason: paths in archives must not have `..` when setting path for scratch

    Description Trying to set the input_paths to use a directory relative to above where toast.yml lives.

    Instructions to reproduce the bug A clear explanation of how to reproduce the bug.

    System information:

    • OS: macOS Mojave 10.14.5
    • Docker version: 18.09.2
    • Toast version: latest

    Additional context I'm trying to have multiple toast files for multiple containers, 1 for the server, 1 for the db. I figured I'd organize the toast files into their own directory to keep things simple and for the server need to copy over the source files.

    bug 
    opened by Akkuma 5
  • PATH is overridden when using su

    PATH is overridden when using su

    Description It seems that PATH is overridden when using su. This is not the case when using docker directly.

    Instructions to reproduce the bug

    $ cat toast.yml 
    image: eclipse-temurin:11-jdk-focal
    tasks:
      java_version:
        command: java -version
    
    $ toast
    [INFO] Ready to run 1 task: java_version.
    [INFO] Running task java_version…
    bash: java: command not found
    

    Example correct output using docker directly:

    $ docker run -t eclipse-temurin:11-jdk-focal java -version
    openjdk version "11.0.17" 2022-10-18
    OpenJDK Runtime Environment Temurin-11.0.17+8 (build 11.0.17+8)
    OpenJDK 64-Bit Server VM Temurin-11.0.17+8 (build 11.0.17+8, mixed mode, sharing)
    

    I suspect this is because su is invoked, and that the PATH environment variable is then overridden.

    Environment information:

    • Toast version: 0.45.5
    • OS: Linux 5.15.76-1-MANJARO #1 SMP PREEMPT Sat Oct 29 14:22:16 UTC 2022 x86_64 GNU/Linux

    Is there any way to fix this?

    Thanks for an interesting and good project!

    bug 
    opened by ivarref 3
  • 'toast'-bin collides with 'gsm'

    'toast'-bin collides with 'gsm'

    Hi,

    I stumbled upon this problem while packaging toast, toast produces a 'toast' bin, sadly this collides with the widely spread 'toast' bin of https://www.quut.com/gsm/ which is a dependency of ffmpeg and is thus very common to be already taken.

    ( see https://linux.die.net/man/1/toast )

    On ArchLinux installing ffmpeg is enough for this, on current Ubuntu it gets taken through libgsm-tools.

    This is not necessarily a bug, but I found no prior issue about this and wanted to raise this, I'll package this as toastci for now.

    Thanks for consideration.

    bug 
    opened by woodgiraffe 1
  • Shared container/host download cache

    Shared container/host download cache

    My toast tasks spent a lot of time repeatedly downloading language dependencies into, e.g., ~/.cargo/registry (Rust) and ~/.m2/ (Java / Clojure). I've run into two problems trying to share these host dependency caches with containers.

    Consider a task definition like:

    image: ubuntu:22.04
    tasks:
      compile-rust:
        dependencies:
          - install-rust
        mount_paths:
          - ~/.cargo/registry:/root/.cargo/registry
        command: |
          cargo build
    

    Problem 1: Containers with mount_paths are not cachable.

    Running toast compile-rust will fail with:

    [ERROR] Unable to parse file /Users/dev/software/toast/scratch/toast.yml. Reason: Task compile-rust has mount_paths but does not disable caching. To fix this, set cache: false for this task.
    

    I understand the intent here --- the contents of the mount_paths are untracked by toast and so in general caching build output could lead to reproducibility problems. However, for the specific need of sharing the host's already-downloaded dependencies, it feels a bit different. We're trusting language tooling (cargo, mvn, whatever) to handle downloading/validating libraries in a sensible way. Since toast is already fine caching containers that download arbitrary stuff from the Internet (i.e., isn't aiming for bitwise-reproducible dependency tracking), it feels reasonable to ignore some of the "inputs" for the purposes of caching.

    Perhaps we should break out this need out as a separate concept, say cache_paths. Then we'd have:

    • input_paths: tracked by toast, for anything where content changes should re-run command (source code, dependency lockfiles, etc.)
    • mount_paths: untracked by toast, for transient data used by commands that never "finish" (e.g., database server)
    • cache_paths: untracked by toast, for data that a command will read/write and handle its own dependencies for.

    Alternatively, we could allow specifying a "don't cache this" option for specific input paths or a "please allow caching" for containers that use mount_paths, but I think it'll be clearer to have an explicitly named concept.

    Problem 2: Intended host paths vary.

    I pulled a fast one with:

    mount_paths:
      - ~/.cargo/registry:/root/.cargo/registry
    

    It actually needs to be a path relative to the toast file (../../.cargo/registry) or an absolute one (/Users/kevin/.cargo/registry). Unfortunately, both make it difficult to make toast-driven projects portable, since it'll force you to checkout the project to a specific folder or run as a specific user.

    The obvious fix in this case is to write a bit of code to expand ~ into the executing user's home directory, though there be more general ways to solve this need (e.g., allowing for interpretation of environment variables in toast config values beyond just the command string).

    If you agree that the need is in-scope for toast, I'm happy to submit PRs in the next few days.

    enhancement 
    opened by lynaghk 3
  • Parallel task execution

    Parallel task execution

    Would you be open to a PR implementing parallel task execution?

    My use case is developing a web app. The backend and frontend are implemented in different languages with separate live-reloading infrastructure, so it makes sense to keep those in separate containers / dependency chains.

    It'd be nice if a single command could run both tasks in parallel (in the example below, toast dev). Currently one must open separate terminals and manually run two separate toast tasks. This makes it harder to hand-off to teammates, since I can't give them a single command to start the dev environment.

    One could write a little shell script to spawn two toast invocations, but then responsibilities are muddled ("I thought I was supposed to use toast to run commands, so why do we have some shell scripts?")

    For my case, it'd be fine to interleave the output. Alternatively we could also have an option to prefix each line of the output with the task name.

    image: ubuntu:[email protected]:0f744430d9643a0ec647a4addcac14b1fbb11424be434165c15e2cc7269f70f8
    command_prefix: set -euo pipefail
    tasks:
    
      dev:
        description: Start the entire development system
        dependencies:
          - frontend
          - backend
    
      frontend:
        description: Start the frontend builder / live-reloader
        command: |
          while true; do
            echo 'frontend...'
            sleep 1
          done
    
      backend:
        description: Start the backend server
        command: |
          while true; do
            echo 'server...'
            sleep 1
          done
    
    enhancement 
    opened by lynaghk 2
  • License isn't recognized as MIT

    License isn't recognized as MIT

    Description

    Github's scanner doesn't recognize your LICENSE.md. The right sidebar just has License [ View license](https://github.com/stepchowfun/toast/blob/main/LICENSE.md). For a moment I feared someone had written yet another license, but it seems LICENSE.md looks like it's MIT.

    You might want to copy the file more byte-identically, maybe Github is fooled by something like word-wrapping.

    bug 
    opened by tv42 1
Releases(v0.45.5)
Owner
Stephan Boyer
Engineering @Google, previously @Airbnb, @Dropbox, @MIT. Trying to turn abstract nonsense into useful technology.
Stephan Boyer
Loads environment variables into your structs in one shot.

econf Loads environment variables into your structs in one shot. econf allows to override struct fields with environment variables easily. This is use

Yushi OMOTE 6 Jul 14, 2022
Docker images for compiling static Rust binaries using musl-libc and musl-gcc, with static versions of useful C libraries. Supports openssl and diesel crates.

rust-musl-builder: Docker container for easily building static Rust binaries Source on GitHub Changelog UPDATED: Major updates in this release which m

Eric Kidd 1.3k Jan 1, 2023
Habitat is open source software that creates platform-independent build artifacts and provides built-in deployment and management capabilities.

Habitat is open source software that creates platform-independent build artifacts and provides built-in deployment and management capabilities. The go

Habitat 2.4k Dec 27, 2022
A buildpack for Rust applications on Heroku, with full support for Rustup, cargo and build caching.

Heroku buildpack for Rust This is a Heroku buildpack for Rust with support for cargo and rustup. Features include: Caching of builds between deploymen

Eric Kidd 502 Nov 7, 2022
Valheim Docker powered by Odin. The Valheim dedicated gameserver manager which is designed with resiliency in mind by providing automatic updates, world backup support, and a user friendly cli interface.

Valheim Docker If you are looking for a guide on how to get started click here Mod Support! It is supported to launch the server with BepInEx but!!!!!

Michael 657 Dec 30, 2022
oci-image and oci-runtime spec in rust.

oci-lib Oci-Spec for your container runtime or container registry. Oci-lib is a rust port for original oci spec written in go. Following crate contain

flouthoc 12 Mar 10, 2022
An infrastructure-as-code and deployment tool for Roblox.

Rocat ?? An infrastructure-as-code and deployment tool for Roblox. ⚠ Please note that this is an early release and the API is unstable. Releases follo

Blake Mealey 45 Dec 29, 2022
An infrastructure-as-code and deployment tool for Roblox.

Mantle ?? An infrastructure-as-code and deployment tool for Roblox. ⚠ Please note that this is an early release and the API is unstable. Releases foll

Blake Mealey 44 Dec 22, 2022
Desktop launcher to install and use Holochain apps locally

Holochain Launcher A cross-platform executable that launches a local Holochain conductor, and installs and opens apps. Feedback is immensely welcome i

Holochain 58 Dec 30, 2022
Tool to monitor the statistics and the energy consumption of docker containers

Docker Activity Docker activity is a tool to monitor the statistics of your containers and output their energy consumption. Warning It's still in earl

Jérémie Drouet 39 Dec 6, 2022
Rust Kubernetes client and controller runtime

kube-rs Rust client for Kubernetes in the style of a more generic client-go, a runtime abstraction inspired by controller-runtime, and a derive macro

kube-rs 1.8k Jan 8, 2023
Qovery Engine is an open-source abstraction layer library that turns easy apps deployment on AWS, GCP, Azure, and other Cloud providers in just a few minutes.

Qovery Engine is an open-source abstraction layer library that turns easy apps deployment on AWS, GCP, Azure, and other Cloud providers in just a few minutes.

Qovery 1.9k Jan 4, 2023
Runc - CLI tool for spawning and running containers according to the OCI specification

runc Introduction runc is a CLI tool for spawning and running containers on Linux according to the OCI specification. Releases You can find official r

Open Container Initiative 9.9k Jan 5, 2023
Inspect and dump OCI images.

reinlinsen ?? rl is a tool to inspect and dump OCI images or single image layers. Installation From source If you have cargo installed you can just ru

Tobias Brumhard 5 May 11, 2023
Manage light-weight sandbox environments for development

Cubicle development container manager Cubicle is a program to manage containers or sandbox environments. It is intended for isolating development envi

Diego Ongaro 8 Nov 29, 2022
Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications

Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications Explore PipeCD docs » Overview PipeCD provides a unified co

PipeCD 650 Dec 29, 2022
🛳 Continuous deployment agent

Subilo ?? Deployment automation agent Subilo is a tool to setup continuous deployments for applications running on machines with no external integrati

null 42 Jun 5, 2022
Rust-lang Continuous Wavelet Transform(CWT) library inspired by fCWT.

fastcwt Rust-lang Continuous Wavelet Transform(CWT) library inspired by fCWT. This crate is a direct translation of fCWT Library written in C++ by Art

null 3 May 2, 2023
2d collision test for game-development in rust (with optional integration and example for bevy)

impacted 2d collision test for game-development in rust (with optional integration and example for bevy) This provides a low-level "narrow-phase" coll

Jonathan Cornaz 17 Nov 5, 2022
BSV stdlib written in Rust and runs in WASM environments

BSV.WASM A Rust/WASM Library to interact with Bitcoin SV Installation NodeJS: npm i bsv-wasm --save Web: npm i bsv-wasm-web --save Rust: https://crate

null 56 Dec 15, 2022