Dynamically generate netboot images for arbitrary NixOS system closures, profiles, or configurations with 10s iteration times.

Create working directories for it:

mkdir ./gc-roots ./profiles ./configurations ./cpio-cache

Then start up the server:

RUST_LOG=info cargo run -- --gc-root-dir ./gc-roots --config-dir ./configurations --profile-dir ./profiles/ --cpio-cache-dir ./cpio-cache/ --listen 127.0.0.1:3030

See ./boot.sh for an example of booting with QEMU.

To boot from a specific closure like /nix/store/0m60ngchp6ki34jpwmpbdx3fby6ya0sf-nixos-system-nginx-21.11pre307912.fe01052444c, use /boot/0m60ngchp6ki34jpwmpbdx3fby6ya0sf-nixos-system-nginx-21.11pre307912.fe01052444c/netboot.ipxe as your chain url.

As long as that closure exists on the host, that closure will always be booted, unchanged.

In the profiles directory, create symlinks to top level system paths. For example:

$ ls -la profiles/
example-host -> /nix/store/4y829p7lljdvwnmsk6pnig3mlh6ygklj-nixos-system-example-host-21.11pre130979.gfedcba

then use /dispatch/profile/example-host to boot it.

The symlink will be resolved every time a machine boots.

In the configurations directory, create a directory for each system, and create a default.nix inside. For example:

$ tree configurations/
configurations/
└── m1.small
    └── default.nix

In the default.nix, create an expression with your NixOS configuration ready to be built:

(import <nixpkgs/nixos> {
    configuration = { pkgs, ... }: {
        networking.hostName = "m1small";
        environment.systemPackages = [ pkgs.hello ];
        fileSystems."/" = {
            device = "/dev/bogus";
            fsType = "ext4";
        };
        boot.loader.grub.devices = [ "/dev/bogus" ];
        boot.postBootCommands = ''
            PATH=${pkgs.nix}/bin /nix/.nix-netboot-serve-db/register
        '';
    };
}).system

Then use /dispatch/configuration/m1.small to boot it.

Create a Hydra project and jobset which contains a job which produces a bootable system configuration.

Then use the URL /dispatch/hydra/HOSTNAME/PROJECT/JOBSET/JOB, substituting those URL sections with names from your system to boot it.

Note that nix-netboot-serve will query the provided Hydra for the store path to boot and will then try to substitute the closure. Nix must already be configured with the requested hydra's cache for this to work.

The configuration will be nix-build once per boot, and create a symlink in the --gc-root-dir directory with the same name as the configuration.

If the build fails, the ipxe client will be told to retry in 5s.

Note: there is currently a buggy race condition. In the following circumstance:

1. machine A turns on
2. machine B turns on
3. machine A hits the build URL and a long build starts
4. you change the configuration to have a very short build
5. machine B hits the build URL and the short build starts
6. machine B's configuration finishes building
7. machine B boots the short build configuration
8. machine A's configuration finishes building
9. machine A boots the short configuration instead of the long configuration

Linux's boot process starts with two things:

1. the kernel
2. an initrd, or an initial ram disk

The ramdisk has all the files needed to mount any disks and start any software needed for the machine. Typically the ramdisk is constructed of a CPIO, a very simple file archive.

Linux supports a special case of its initrd being comprised of multiple CPIOs. By simply concatenating two CPIOs together, Linux's boot process will see the merged contents of both CPIOs.

Furthermore, individual CPIOs can be compressed independently, merged together with concatenation, and Linux will decompress and read each CPIO independently.

A NixOS system is comprised of hundreds of independent, immutable /nix/store paths.

Merging these together, we can dynamically create a single, compressed CPIO per Nix store path and cache it for later.

When a new boot request comes in, the software fetches the list of Nix store paths for the requested NixOS system. Then, every path has a CPIO built for it. Once each store path has a CPIO, the results are streamed back to the iPXE client. By caching the resulting CPIO, iterative development on a system configuration can result in just 3-5 new CPIOs per change.

NixOS's NetBoot image creation support works well, however iterating on a single closure involves recreating the CPIO and recompressing for every store path every time. This can add several minutes to cycle time.

• Get the size of the initrd: HEAD /boot/PATH/initrd
• Pass additional kernel commandline arguments: /dispatch/...?cmdline_prefix_args=...&cmdline_suffix_args=...

Before using Nix inside the booted machine, make sure to load the Nix database. To do that, add this to your NixOS configuration:

{ pkgs, ... }: {
    boot.postBootCommands = ''
        PATH=${pkgs.nix}/bin /nix/.nix-netboot-serve-db/register
    '';
}

This is not necessary if the system will not execute Nix commands.