Manage light-weight sandbox environments for development

Last update: Nov 29, 2022

Related tags

Overview

Cubicle development container manager

Cubicle is a program to manage containers or sandbox environments. It is intended for isolating development environments from the host system and from each other.

Cubicle can run on top of any of these isolation mechanisms, called runners:

Docker, which is a popular yet heavy-weight container mechanism. Docker runs Linux containers, but it runs on Mac and Windows as well by running a Linux VM. Under Docker, the environments may use sudo to modify their root partitions. See the Docker-specific docs for details, including security implications and installation instructions.
Bubblewrap, which is a light-weight mechanism that runs on Linux only. Under Bubblewrap, the host's root partition is shared read-only with the environments. See the Bubblewrap-specific docs for details, including security implications and installation instructions.
System user accounts, created and switched to via sudo. With system user accounts, the operating system prevents (or not) the environments from reading/writing the root partition and other user's files with classical file permissions. See the User accounts-specific docs for details, including security implications and installation instructions.

Cubicle is in early stages of development and is likely to change frequently in incompatible ways. Users should review the Git commits to see what's changed before upgrading.

Motivation

Cubicle protects your host system, protects development projects from interfering with each other, and lets you take advantage of modern developer tools.

Modern development requires running a lot of other people's code. First, there's tooling, including editors/IDEs, compilers, code formatters, linters, test runners, bundlers, and package managers, and these often differ from one language/ecosystem to another. These tools and your code's increasing number of dependencies may be subject to software supply chain attacks, where a regular software update suddenly gives an attacker control. It's impractical to audit all the code you run to develop software, and for many projects, it's even impractical to audit all your software's own dependencies in depth.

I found that I was avoiding lots of tools just because I didn't want to add risk to my host system. Do I want to install that janky VS Code extension that has 3 users, just for syntax highlighting? When it suggests installing a linter, do I want to do that too? (Yes, there's some irony in that Cubicle itself is a janky program that has fewer than 3 users -- for now.)

Modern development also moves fast. VS Code gets updates every month. Rust nightly builds are updated, well, nightly. It's hard for distributions to keep up, so you probably end up downloading and maintaining a lot of key tools in userspace.

With Cubicle, you can have a pristine, isolated development environment with modern tooling every single time. If you learn about some new open source project through the bird app or the orange website or whatever, Cubicle lets you spin up a new environment in seconds to try things out. When you find something you like, you can define a package so that it's always ready to go and up to date in your environments.

What does this provide over Docker?

Docker is a popular container manager. It's commonly used to run long-running network services (daemons) in an isolated an reproducible environment. It's also used to build (compile and package) software in a reproducible environment. Docker is very flexible. It's less commonly used this way, but you can develop software inside of a Docker container, or share your source files between your host and a container, running your editor on the host and the other tools inside the container.

Docker containers are usually immutable and built in sequential layers. They're used for the purpose of running a build or a single version of a service, then they are thrown away and replaced.

Cubicle packages are built independently and then mixed together to populate a container environment. Cubicle promotes developer experimentation, so containers can be longer lived and modified in place, if desired. It's easy and common to replace the guts of a Cubicle container while maintaining the user's work.

Feedback

This project is still in early stages. It feels like I've stumbled on a better way of doing things, but I don't know what this project should be when it grows up. I've shared this for the purpose of gathering feedback from early users. Please leave feedback in the GitHub Discussions.

Security

The goal of any sandbox is to isolate software so that software running inside the sandbox cannot "break out", meaning it cannot access or affect the system outside the sandbox. Cubicle may not meet this goal, at least depending on the environment and the enforcement mechanism used. Cubicle does offer a meaningful layer of security when compared to running untrusted software directly on your host.

Cubicle Packages

Since Cubicle environments are created and recreated often, it's helpful to inject configuration and program files into them. This allows you to use a new container right away and not grow attached to them.

The current package format is pretty simple. A package definition is named after its directory. It must contain one or more of these files:

update.sh: An executable that is run in a package builder environment to produce the package files. The script may download software, unpack it, compile it, set up configuration files, etc. It should create an archive at ~/provides.tar that Cubicle will later unpack in the target environments' home directories. Note that update.sh runs when the package builder environment is first created and also when the package builder environment is updated. The package builder environments are kept around as a form of caching.
build-depends.txt: A newline-separated list of package names on which this package depends when it is built. The package builder environment will be seeded with the listed packages, but other environments that depend on this package will not.
depends.txt: A newline-separated list of package names on which this package or its output depends. Both the package builder environment and the new environments that depend on this package will be seeded with the listed packages.
test.sh: An executable that is run in a clean environment to sanity check the package output files. The test environment is seeded with the package's dependencies, the package output files, and the package source directory (except update.sh).

These files and any other files in the package directory are injected into the work directory of the package builder environment.

If the package provides any executable files within ~/.dev-init/, these will be run upon creating and resetting target environments.

Packages are built automatically when they are first used, and they are updated when they are used if 12 hours have elapsed, their package definitions have changed, or one of their dependencies or build-dependencies has been updated more recently.

Cubicle looks for package definitions in the following locations:

Local packages in ${XDG_DATA_HOME:-~/.local/share}/cubicle/packages/*/.
Built-in packages in the Cubicle source code's packages/ directory.

If a package with the same name appears in multiple locations, the first one is used and the others are ignored. The sort order of the names of the containing directories is significant for local packages, so you may want to create 00local to come first.

The package named "default" is used for new environments when a package list is not otherwise specified.

The package named "auto" and its dependencies are automatically included in every environment except those that "auto" itself transitively depends upon. This is useful for configuration files, for example.

Related Projects

Bubblewrap is a low-level tool from the Flatpak developers to run lightweight containers. Julia Evans wrote a recent blog post exploring Bubblewrap.
Development Containers appears to be a proposed specification by Microsoft for configuring full-featured development environments using Docker.
Docker is a popular container manager.
Firejail defines and enforces per-application profiles to sandbox programs and GUI applications.
Flatpak runs packaged GUI applications in a sandbox. Bubblewrap came out of the Flatpak project and is used by Flatpak.
LXC is a low-level tool for running large, long-lived Linux containers, like VMs but with less overhead.
LXD is a command-line tool to manage LXC containers (and VMs).
Nix is a package manager that installs software in isolated directories. This isolates software installations from each other and allows multiple versions of packages to coexist. It does not appear to prevent packages from interacting through files in the user's home directory.
Podman is simlar to Docker but daemonless. Its CLI is mostly compatible with Docker's.
Vagrant is a tool for automatically configuring virtual machines and Docker containers. It uses so-called "Provisioners" like shell scripts, Puppet, or Chef to configure the environments.

Docker Containers on the Desktop by Jessie Frazelle (2015). Discusses using Docker to isolate desktop apps, including 10 examples.

Comments

doesn't work on Mac
A few known issues:

[X] ~Python 3.8 doesn't know about the type syntax~ -- Cubicle was since rewritten in Rust

[X] ~du is different~ -- no longer used on host

[X] ~tar is different~ -- no longer used on host

[X] ~Docker mounts are different. The $HOME path on the host and on the container are different.~ -- #9 adds Docker volume support and fixes $HOME path assumptions.

[ ] X11 surely won't just work.

[ ] Some hard-coding of amd64/x86-64.

There are probably more issues after that. And then there's a big performance question.

cc @darrengarvey
opened by ongardie 4
Chromium, VS Codium crash often under Docker

The window has crashed (reason: 'crashed', code: '133')

I'm not sure, but this might have started when I switched from running it in Bubblewrap to using Docker.

This seems to be reproducible when I take certain actions, like previewing README.md in this repo.

opened by ongardie 3

package.toml doesn't support quoted or dotted key names.

e.g.

[depends.debian]
"libstdc++-arm-none-eabi-newlib" = {}

gives

simon@Starbuck cubicle % cub package update qmk
Error: error scanning built-in packages in "/Users/simon/Github/cubicle/packages"

Caused by:
    0: could not read manifest for package "qmk": "/Users/simon/Github/cubicle/packages/qmk/package.toml"
    1: data did not match any variant of untagged enum DependencyOrTable for key `depends.debian` at line 1 column 1

same for

[depends.debian]
libusb-1.0-0={}

this running #38 if that makes a difference.

opened by superfell 2

error when exiting a cub

This is using #38 on arm/osx I cub enter bob3 do some things, then use exit to leave bob3. At which point i get an error

simon@bob3:~/w$ exit
Error: failed to run command in environment "bob3"

Caused by:
    Non-zero exit status (1) from docker exec

Stack backtrace:
   0: anyhow::backtrace::capture::Backtrace::capture
   1: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   2: cubicle::docker::Docker::run_
   3: <cubicle::runner::CheckedRunner as cubicle::runner::Runner>::run
   4: cub::cli::run
   5: cub::main
   6: std::sys_common::backtrace::__rust_begin_short_backtrace
   7: std::rt::lang_start::{{closure}}
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:280:13
      std::panicking::try::do_call
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/rt.rs:128:48
      std::panicking::try::do_call
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/rt.rs:128:20
   9: _main

opened by superfell 2

docker: Fix running as `root`

I ran into some trouble trying to run Cubicle Docker environments as root:

11: Pulling from library/debian
10cff8997b4d: Already exists 
Digest: sha256:3e82b1af33607aebaeb3641b75d6e80fd28d36e17993ef13708e9493e30e8ff9
Status: Downloaded newer image for debian:11
 ---> 59ed139d178b
Step 2/8 : RUN echo Etc/UTC > /etc/timezone &&     ln -fs '/usr/share/zoneinfo/'Etc/UTC /etc/localtime
 ---> Running in b3346793084e
Removing intermediate container b3346793084e
 ---> e980d48c6ccd
Step 3/8 : RUN addgroup --gid 0 root || addgroup root &&     adduser --disabled-password --gecos '' --uid 0 --ingroup root root &&     adduser root sudo &&     mkdir /home/root/w &&    
 chown root:root /home/root/w
 ---> Running in 490bcc5504a4
addgroup: The group `root' already exists.
addgroup: The group `root' already exists.
The command '/bin/sh -c addgroup --gid 0 root || addgroup root &&     adduser --disabled-password --gecos '' --uid 0 --ingroup root root &&     adduser root sudo &&     mkdir /home/root
/w &&     chown root:root /home/root/w' returned a non-zero code: 1
Error: failed to update package: "configs-core"

Caused by:
    0: error building package "configs-core"
    1: failed to create environment "package-configs-core"
    2: failed to build "cub-cubicle-base" Docker image
    3: `docker build` exited with exit status: 1
root@debian:~/opt/cubicle#

Seems like this should probably just pick another username, like "cubicle".

opened by ongardie 1

`script_path` busted on Mac

From https://doc.rust-lang.org/std/env/fn.current_exe.html#platform-specific-behavior:

If the executable was invoked through a symbolic link, some platforms will return the path of the symbolic link and other platforms will return the path of the symbolic link’s target.

Better make this more obvious, more resilient, and configurable.

cc @darrengarvey

opened by ongardie 1
Stop setting hostname

Fix Issue #44: "docker fails with overly long hostnames"

I don't know if setting the hostname was useful or important anywhere other than the shell prompt. If so, it's easy to revert this.

opened by ongardie 0
Rewrite some packages
The core configs are now more extensible.

asdf now manages Node and Go installs, and it's likely to be useful for other packages later.

Hopefully these now work on aarch64/arm64 (#36).

Due to some renames and rewrites, users might need to clear out package environments and caches upon upgrade:

cub purge 'package-*' rm ~/.cache/cubicle/packages/*.tar
opened by ongardie 0
bubblewrap: Copy/extract seeds with stream

Before this would write the seed tarball out, then read it. This is a little nicer because it's (likely) faster, and it gets rid of some shell script.

opened by ongardie 0
VM runner

It'd be nice to have a runner that runs on virtual machines (for when you're doing something particularly icky or something that itself needs Docker). libvirt and crosvm are two relevant projects to consider.

opened by ongardie 0
streamline setup
New users should be able to download a binary, make it executable, and run it. Without reading any other setup instructions, something useful and reasonable should happen.

[ ] Distribute static binaries (for {mac,linux}-{amd64,aarch64})

[ ] Manage package definitions at runtime

[ ] Set up a default configuration or have a configuration "wizard" when no config found

cc @darrengarvey
opened by ongardie 0
[discussion] packaging qmk

Having recently worked with qmk for keyboard firmware, I thought I'd take a stab at writing a package for it. You can see the attempt here https://github.com/superfell/cubicle/commit/2e0adeba98fe91ed705f617b8670cf33102c9e2a

qmk has a lot of dependencies. There are some base dependencies (git, python3) that are needed to get the qmk cli tool, and then that tool is used to set everything up including a long list of deb packages and more python packages.

when working with qmk, you work directly in the qmk_firmware tree, its not a separate library. qmk setup will clone the repo to the relevant location. I Used ~/w/qmk_firmware. but this seems wrong. But if its somewhere else and added to the provides, then that also seems wrong.

perhaps qmk isn't a tool and so isn't a good fit for cubicle? Or perhaps there some secret way of using qmk without your work being in the qmk_firmware tree.

opened by superfell 2
too many shell scripts

The package definitions currently have a lot of shell scripts, which is bad for my sanity, as well as correctness and portability (across distros or versions of tools, or even operating systems). They're likely to get more complicated, too, thanks to #30, #31, #34, and #36.

It should be possible to use build_depends and write many of these scripts in a more suitable language. Rust is probably my top pick. It'd have challenges of convenience and compilation times, but I think we can find ways to overcome those.

opened by ongardie 0

go package (and others) fail to build due to cpu arch assumption (M1 Mac OSX)

using latest, attempting to build the go package target/debug/cubicle package update go results in

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Copying/extracting seed tarball
15.5KiB 0:00:00 [4.12MiB/s] [================================>] 100%            
Checking latest version of Go
./build.sh: 13: go: not found
./build.sh: 14: go: not found
Have  installed at 
Downloading go1.19.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    75  100    75    0     0    681      0 --:--:-- --:--:-- --:--:--   681
100  141M  100  141M    0     0  80.6M      0  0:00:01  0:00:01 --:--:-- 92.1M
Unpacking go1.19.1
 141MiB 0:00:02 [54.8MiB/s] [============================================================================================================================================================================================================================================>] 100%            
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
ERROR: Have  installed at , which is still not right
Error: failed to update package: "go"

Caused by:
    0: error building package "go"
    1: failed to run command in environment "package-go"
    2: Non-zero exit status (1) from docker exec

Stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/mod.rs:53:14
   2: anyhow::backtrace::capture::Backtrace::create
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.61/src/backtrace.rs:216:13
   3: anyhow::backtrace::capture::Backtrace::capture
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.61/src/backtrace.rs:204:17
   4: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /Users/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.61/src/error.rs:533:25
   5: <T as core::convert::Into<U>>::into
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/convert/mod.rs:550:9
   6: anyhow::kind::Trait::new

poking around in build.sh and in the created docker image, it looks like the build.sh assumes ~/bin is in the path, but it doesn't appear to be. At least from docker exec -i cub-package-go sh from that shell, it lists the path as PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

The PATH issue may be a complete red herring, and the real issue is whatever causes the qemu logging. I'll continue to poke around.

opened by superfell 11

package build spew

Package builds currently print so much output that it's hard to tell what's going on. It'd be great to have a way to monitor progress without being overwhelmed by it.

(We might also want to think about parallel package builds and how to display that.)

opened by ongardie 0

Owner

Diego Ongaro

Co-author of @raft. Previously @homex, @eBay, @salesforce, @PlatformLab at Stanford.

GitHub

Vulkan rendering sandbox for raytracing

sol-rs ☀ sol-rs is a small rendering toolkit for Vulkan, with a focus on real-time raytracing (which is not currently available via other APIs such as

65 Dec 7, 2022

Sandbox is a pixel physics simulator inspired by other such like Sandspiel and Noita

Sandbox Sandbox is a pixel physics simulator inspired by other such like Sandspiel and Noita. It's also a precursor for an ongoing game project. Sandb

76 Nov 3, 2022

A sandbox library for making FAST voxel games

voxelize WIP A well-optimized web-based voxel engine. Development Before starting, make sure to install the following: rust node.js cargo-watch # clon

146 Dec 30, 2022

An online judge sandbox server in Rust, inspired by go-judge, for SAST OJ.

RsJudge An online judge sandbox server in Rust, inspired by go-judge, for SAST OJ. Table of Contents Features Build from source Prerequisites Build Fe

4 Dec 10, 2023

A CLI tool to manage your godot-rust projects

ftw A CLI tool to manage your godot-rust project! Table of contents General Information Setup Usage Contact General Information This is a tool to help

77 Dec 13, 2022

Libium is the backend of Ferium. It helps manage Minecraft mods from Modrinth, CurseForge, and Github Releases

Libium Libium is the backend of Ferium. It helps manage Minecraft mods from Modrinth, CurseForge, and Github Releases There are 3 main components in L

14 Dec 13, 2022

A Bevy helper to easily manage resources that need to persist across game sessions.

bevy-persistent A Bevy helper to easily manage resources that need to persist across game sessions. Background In games, there are a lot of resources

5 Mar 25, 2023

A Bevy plugin to easily create and manage windows that remember where they were.

bevy-persistent-windows A Bevy plugin to easily create and manage windows that remember where they were. Background When you're developing a game, thu

4 Aug 12, 2023

IDE for cross-platform software development

Diversity Space IDE for cross-platform software development | 日本語 | English | Русский | READMEの英語版とロシア語版はDeepl翻訳を使用して翻訳されています Английская и русская вер

0 Feb 23, 2022

Coordination repository of the Game Development Working Group

Rust Game Development Working Group ??️ The game development working group's main purpose is to make Rust a first-class option for game developers. Wh

448 Jan 2, 2023

2d collision test for game-development in rust (with optional integration and example for bevy)

impacted 2d collision test for game-development in rust (with optional integration and example for bevy) This provides a low-level "narrow-phase" coll

17 Nov 5, 2022

TestSuite4 is a framework designed to simplify development and testing of TON Contracts. It includes light-weight emulator of blockchain making it easy to develop contracts.

TestSuite4 0.1.2 TestSuite4 is a framework designed to simplify development and testing of TON Contracts. It contains lightweight blockchain emulator

26 Feb 3, 2022

Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

294 Dec 23, 2022

Create, open, manage your Python projects with ease, a project aimed to make python development experience a little better

7 Nov 18, 2022

Hamming Weight Tree from the paper Online Nearest Neighbor Search in Hamming Space

hwt Hamming Weight Tree from the paper Online Nearest Neighbor Search in Hamming Space To understand how the data structure works, please see the docs

7 Oct 9, 2021