Make any NixOS system netbootable with 10s cycle times.

Related tags

nix-netboot-serve
Overview

nix-netboot-serve

Dynamically generate netboot images for arbitrary NixOS system closures, profiles, or configurations with 10s iteration times.

Usage

Create working directories for it:

mkdir ./gc-roots ./profiles ./configurations ./cpio-cache

Then start up the server:

RUST_LOG=info cargo run -- --gc-root-dir ./gc-roots --config-dir ./configurations --profile-dir ./profiles/ --cpio-cache-dir ./cpio-cache/ --listen 127.0.0.1:3030

See ./boot.sh for an example of booting with QEMU.

Booting an absolute closure

How To

To boot from a specific closure like /nix/store/0m60ngchp6ki34jpwmpbdx3fby6ya0sf-nixos-system-nginx-21.11pre307912.fe01052444c, use /boot/0m60ngchp6ki34jpwmpbdx3fby6ya0sf-nixos-system-nginx-21.11pre307912.fe01052444c/netboot.ipxe as your chain url.

Behavior

As long as that closure exists on the host, that closure will always be booted, unchanged.

Booting a profile

How To

In the profiles directory, create symlinks to top level system paths. For example:

$ ls -la profiles/
example-host -> /nix/store/4y829p7lljdvwnmsk6pnig3mlh6ygklj-nixos-system-example-host-21.11pre130979.gfedcba

then use /dispatch/profile/example-host to boot it.

Behavior

The symlink will be resolved every time a machine boots.

Booting a configuration

How To

In the configurations directory, create a directory for each system, and create a default.nix inside. For example:

$ tree configurations/
configurations/
└── m1.small
    └── default.nix

In the default.nix, create an expression with your NixOS configuration ready to be built:

(import 
    {
    configuration = { pkgs, ... }: {
        networking.hostName = "m1small";
        environment.systemPackages = [ pkgs.hello ];
        fileSystems."/" = {
            device = "/dev/bogus";
            fsType = "ext4";
        };
        boot.loader.grub.devices = [ "/dev/bogus" ];
        boot.postBootCommands = ''
            PATH=${pkgs.nix}/bin /nix/.nix-netboot-serve-db/register
        '';
    };
}).system

Then use /dispatch/configuration/m1.small to boot it.

Behavior

The configuration will be nix-build once per boot, and create a symlink in the --gc-root-dir directory with the same name as the configuration.

If the build fails, the ipxe client will be told to retry in 5s.

Note: there is currently a buggy race condition. In the following circumstance:

  1. machine A turns on
  2. machine B turns on
  3. machine A hits the build URL and a long build starts
  4. you change the configuration to have a very short build
  5. machine B hits the build URL and the short build starts
  6. machine B's configuration finishes building
  7. machine B boots the short build configuration
  8. machine A's configuration finishes building
  9. machine A boots the short configuration instead of the long configuration

Notes on NixOS Configuration

Booting a machine from this server will completely ignore any of the defined fileSystems, everything will run out of RAM.

This system assumes a normal NixOS system booting off a regular disk: trying to use this to netboot a USB installer will not work.

If you don't have an existing configuration to start with, you could start with this:

{
    fileSystems."/" = {
        device = "/dev/bogus";
        fsType = "ext4";
    };
    boot.loader.grub.devices = [ "/dev/bogus" ];
}

Theory of Operation

Linux's boot process starts with two things:

  1. the kernel
  2. an initrd, or an initial ram disk

The ramdisk has all the files needed to mount any disks and start any software needed for the machine. Typically the ramdisk is constructed of a CPIO, a very simple file archive.

Linux supports a special case of its initrd being comprised of multiple CPIOs. By simply concatenating two CPIOs together, Linux's boot process will see the merged contents of both CPIOs.

Furthermore, individual CPIOs can be compressed independently, merged together with concatenation, and Linux will decompress and read each CPIO independently.

A NixOS system is comprised of hundreds of independent, immutable /nix/store paths.

Merging these together, we can dynamically create a single, compressed CPIO per Nix store path and cache it for later.

When a new boot request comes in, the software fetches the list of Nix store paths for the requested NixOS system. Then, every path has a CPIO built for it. Once each store path has a CPIO, the results are streamed back to the iPXE client. By caching the resulting CPIO, iterative development on a system configuration can result in just 3-5 new CPIOs per change.

Improvements over NixOS's NetBoot Support

NixOS's NetBoot image creation support works well, however iterating on a single closure involves recreating the CPIO and recompressing for every store path every time. This can add several minutes to cycle time.

Caveats

Loading the Nix Database

Before using Nix inside the booted machine, make sure to load the Nix database. To do that, add this to your NixOS configuration:

{ pkgs, ... }: {
    boot.postBootCommands = ''
        PATH=${pkgs.nix}/bin /nix/.nix-netboot-serve-db/register
    '';
}

This is not necessary if the system will not execute Nix commands.

Issues
  • initrd: remove kernel image

    initrd: remove kernel image

    We do not need to serve the kernel image in the initrd. This saves ~100MB of initrd on the default kernel configuration.

    opened by baloo 3
  • Doesn't support UEFI

    Doesn't support UEFI

    When trying to boot a VM over iPXE utilizing OVMF (QEMU EFI software), the following error can be observed:

    >>Start PXE over IPv4.
      Station IP address is 10.0.2.26
    
      Server IP address is 10.0.2.1
      NBP filename is http://10.0.2.1:3030/dispatch/configuration/small
      NBP filesize is 0 Bytes
      PXE-E99: Unexpected network error.
    BdsDxe: failed to load Boot0005 "UEFI PXEv4 (MAC:000000000000)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(000000000000,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0): Not Found
    

    After reading https://ipxe.org/appnote/uefihttp, it appears that iPXE expects an EFI file to be served, while this is serving the standard ipxe file.

    opened by cole-h 0
Releases(v0.1.0)
Owner
Determinate Systems
Determinate Systems
👩‍❤️‍💋‍👩 Synchronize installed packages on multiple machines

emplace Command-line tool to mirror installed software on multiple machines. Features Outputs a human-readable (RON) file to sync between machines: .e

Thomas Versteeg 135 Sep 9, 2021
Migrate C code to Rust

C2Rust helps you migrate C99-compliant code to Rust. The translator (or transpiler) produces unsafe Rust code that closely mirrors the input C code. T

Immunant 2k Sep 17, 2021
Lintje is an opinionated linter for Git.

Lintje Lintje is an opinionated linter for Git. It lints commit messages based on a preconfigured set of rules focussed on promoting communication bet

Tom de Bruijn 13 Aug 25, 2021
Astronomical algorithms in Rust

astro-rust Contents API Docs About Usage Contributing References About astro-rust is a library of advanced astronomical algorithms for the Rust progra

Saurav Sachidanand 165 Sep 9, 2021
Experimental playground for wiktionary data

wikt Experimental playground for wiktionary data. This document might not update as often as the code does. Set up You'll want a minimum of 10 GB free

Félix Saparelli 7 Jul 10, 2021
Combine internet connections, increase your download speed

dispatch A SOCKS proxy that balances traffic between network interfaces. Should work on macOS, Windows, and Linux. Only tested on macOS for now. This

Alexandre Kirszenberg 10 Aug 6, 2021
Minimal, flexible framework for implementing solutions to Advent of Code in Rust

This is advent_of_code_traits, a minimal, flexible framework for implementing solutions to Advent of Code in Rust.

David 5 May 27, 2021
A framework for iterating over collections of types implementing a trait without virtual dispatch

zero_v Zero_V is an experiment in defining behavior over collections of objects implementing some trait without dynamic polymorphism.

null 14 Jul 8, 2021
A lightning fast version of tmux-fingers written in Rust, copy/pasting tmux like vimium/vimperator

tmux-thumbs A lightning fast version of tmux-fingers written in Rust for copy pasting with vimium/vimperator like hints. Usage Press ( prefix + Space

Ferran Basora 419 Sep 17, 2021
cargo, make me a project

cargo-generate cargo, make me a project cargo-generate is a developer tool to help you get up and running quickly with a new Rust project by leveragin

null 789 Sep 7, 2021
Nannou/Rust tutorial based on Schotter by Georg Nees

Schotter (German for gravel) is a piece by computer art pioneer Georg Nees. It consists of a grid of squares 12 across and 22 down with random rotation and displacement that increases towards the bottom.

null 20 Sep 2, 2021
Simple ray tracer written in Rust

Simple ray tracer written in Rust from scratch I've just finished my first semester at the Faculty of Applied Mathematics and Computer Science at the

Vladislav 191 Aug 20, 2021
a hobby OS for x86_64 based on MikanOS.

a hobby OS for x86_64 based on MikanOS.

algon 17 Jul 2, 2021
Concatenate Amazon S3 files remotely using flexible patterns

S3 Concat This tool has been migrated into s3-utils, please use that crate for future updates. A small utility to concatenate files in AWS S3. Designe

Isaac Whitfield 31 May 22, 2021
Unlock vGPU functionality for consumer grade GPUs

Rust-based vgpu_unlock Unlock vGPU functionality for consumer-grade NVIDIA GPUs. This tool is to be used with the kernel patches from the main vgpu_un

Matt Bilker 3 Sep 17, 2021
A tool of generating and viewing dice roll success distributions.

AZDice A GUI tool for generating and visualising dice roll probability distributions. Aims Intended to help people trying to get game balance just rig

null 13 Mar 2, 2021
Utilities and tools based around Amazon S3 to provide convenience APIs in a CLI

s3-utils Utilities and tools based around Amazon S3 to provide convenience APIs in a CLI. This tool contains a small set of command line utilities for

Isaac Whitfield 35 Sep 13, 2021
A library to compile USDT probes into a Rust library

sonde sonde is a library to compile USDT probes into a Rust library, and to generate a friendly Rust idiomatic API around it. Userland Statically Defi

Ivan Enderlin 29 Aug 31, 2021
ᎩᎦᎨᎢ (IPA: [gigagei]) is a random quote fetching console utility. Written in Rust.

gigagei ᎩᎦᎨᎢ (IPA: [gigagei]) is a random quote fetching console utility. Written in Rust. Installing Use latest pre-built binary from releases Buildi

veleth 10 Sep 5, 2021