Hi aurae community,

This PR adds first steps to make auraed boot as pid1, mainly mounting the kernel filesystems (devfs, procfs and sysfs), but also some minor fixes to make auraed boot without kernel panic.

• Fixes kernel panic due to a rust panic thrown by syslog creation (missing unix sockets [1])
  • The rust panic causes auraed to stop, and if started as pid1 the kernel will in turn also panic because the init process ended.
• Fixes hack/hack.mk for zsh users
• Adds a SystemRuntime (analog to AuraedRuntime), which handles system preparation.
  • SystemRuntime Case: not-pid1
    • assume a systemd environment (currently only unix socket /var/run/syslog)
  • SystemRuntime Case: pid1
    • initializes logging without unix socket dependency. Using syslog crate would require creation of unix socket first, and I suggest to discuss this first if there is a use case to have a unix socket for logging.
    • mounts file based kernel interfaces
    • devfs, sysfs, procfs
• Added helper functions for basic file io (currently for devfs, sysfs, procfs interfaces)
  • get content of e.g. /dev, /sys or /proc
  • or /sys/class/net/ to see available network interfaces

ToDo:

• Discuss Logging targets
  • Same log targets for all SystemRuntime cases?
• Network initialization
• Stop auraed in pid1 case gracefully (kernel halt syscall to shutdown or reboot)

How to test

Steps to run auraed as pid in a qemu vm are documented in the hack/README.md. Copy pasted them here:

make build-container
make kernel
make initramfs

# create `vm-br0` bridge on your machine:
make network

# run auraed in a VM as pid 1:
make virsh-start virsh-console virsh-stop

# exit VM console with Ctrl+]

Links

[1] https://docs.rs/syslog/latest/src/syslog/lib.rs.html#232-243

Hi Kris,

together with @vincinator I am working on a project somewhat similar to aurae. We are writing a pid 1 in Rust to act as a VMM (basically a replacement for systemd+libvirt), that exposes a gRPC API to control VMs from a SmartNIC. It's not published yet, but will also be under Apache v2.

I have extracted our tooling around testing the program as pid 1 within a libvirt VM. Have a look at hack/README.md for details.

Basically this PR allows you to start a VM, boot a Kernel and load auraed as pid 1. With the current state of auraed you see a Kernel panic as auraed expects things on the Linux system, that don't exist (e.g. syslog).

We can discuss the way those scripts are implemented. I am open for changes.

Malte

Hi all,

• feature: add ipv4/ipv6 route to network interface
  • adding routes to a network interface required a workaround for issue https://github.com/aurae-runtime/auraed/issues/40
• refactor: extract network configuration in separate functions (loopback config, current default nic config)
  • needed to understand why adding an ip route failed, this refactor helped identifying the cause (described in #40)

Hi @future-highway, could you review this PR please?

This PR iterates over the error handling of the init module.

• added Err return values in cases where we silently failed before
• use expect instead of unwrap
• restrict function visibility of init functions to a minimum, to help maintain and select appropriate error handling

Next iteration on error handling could be the introduction of custom error types. see: https://github.com/aurae-runtime/auraed/issues/32

Hi Community,

this PR adds a power button listener.

• Listen to power button events
• No libc required for listening to events
  • libc only used for reboot syscall (easy to replace)

Test

Run auraed in a VM as pid1:

make build-container
make kernel
make initramfs

# create `vm-br0` bridge on your machine:
make network

# run auraed in a VM as pid 1:
make virsh-start virsh-console virsh-stop

# exit VM console with Ctrl+]

Send a shutdown signal:

make virsh-shutdown 
# before the vm is powered off, a log entry shows the received input event

Todos

• Notify Runtime first before shutting down - .await the auraed runtime
• detect correct power button input device (hardcoded to /dev/input/event0 which works for the libvirtd development vm)
• Handle reboot button with a reboot instead

Hi,

This PR uses the crate little-dude/netlink to initialize network devices

• Added network initialization for the libvirt-pid1 initialization flow
  • Initialize lo device
    • assign ::1/128
    • assign 127.0.0.1/8
  • Initialize eth0 device
    • assign fe80::2/64
  • Print network info after initialization
• Functions used in above initialization flow are a starting point for other scenarios
• Possibility to create sriov devices for a given network interface

How to test

As usual, copy paste steps from hack/README.md:

make build-container
make kernel
make initramfs

# create `vm-br0` bridge on your machine:
make network

# run auraed in a VM as pid 1:
make virsh-start virsh-console virsh-stop

# exit VM console with Ctrl+]

On a second terminal you are now able to ping the auraed node via

# name of network interface "vm-br0" is set up by make network
ping fe80::2%vm-br0 

Screenshot

This PR adds a few features to the project

• Removes the repeated protobuf definitions for the meta objects
• Establishes runtime as a synchronous subsystem (schedule will be async)
• Introduces a working exec
• Introduces some code cleanup in auraed to simplify our codebase

Definitely not where I would want to end up, but this seemed like a good place to do the pr/get a review. If the organization makes sense, I (with some guidance) or someone else can attempt to continue with network, fileio, and power.

Steps taken:

• Moved all the init code into the init module (specifically SystemRuntime)
• Attempted to architect the code in a more foolproof way (e.g., non-pid1 process cannot call pid1 init code)
• Divided related code into smaller mods
• Slimed down init's top level mod, and expose the minimum necessary
• Reduced the reliance on anyhow in favor of custom errors using thiserror

Clippy has a default configuration, so it will output warnings for more than explicitly configured. This should be overridable, but starting with the defaults seems like a good choice.

You can see an example of where I needed to override clippy in the build.rs file.

The Aurae project is based around the concept of objects.

Giving that an RPC Message is the lowest common definition for our data structures, we will need to be able to generate a substantial amount of code and boilerplate for each object.

Developers (and maybe one day consumers, clients, and end-users) should have an easy way of creating and expressing new and generic objects in the codebase.

For each object we define as an RPC message we will need to do the following.

• Establish a database table, and corresponding schema.
• Establish a source of truth for Rust structures
• Establish client code
• Establish server code
• Establish Authz (authorization) style traits which can be implemented to bring authz to each object and corresponding functions
• Establish AuraeScript definitions with corresponding getters and setters such that the objects can quickly be expressed in AuraeScript

We need to start taking advantage of rustdoc in the .proto files.

Ideally we can generate the documentation with a Make target or similar and have it write directly to the /stdlib directory.

When setting a network device up (via netlink) the function returns before the network device is actually in state up. Assigning routes to a network device (via netlink) requires the network device to be up.

I think the caller should safely assume that .await for the async function init/network.rs:set_link_up assures that the link is up, so that follow up network configuration steps can safely assume that the link is really up after the call.

In addition, I would like to provide access to the netdevice state during network configuration, so that a config functions relying on a link state can implement safety checks.

• [x] Compiling Qemu and adding to initramfs
• [ ] Check if Qemu is able to run within auraed environment
• [ ] Add API methods to create VMs
• [ ] Start Qemu VMs from auraed
• [ ] Wire up VM's serial console with auraed internal log data streams

it should be possible to trigger a system reboot or a shutdown via the grpc API. If auraed has a process ID > 1 this action should result in a graceful shutdown and exit of auraed instead.

See also #36

We need to implement a graceful shutdown flow. This flow must be executed when auraed gets a SIGTERM signal, the power button is pressed (if executed as pid 1) or when auraed gets a reboot or shutdown request via GRPC.

1. An event has to be sent to all GRPC clients to inform them about the imminent shutdown.
2. No new workloads (processes, containers, VMs, ...) may be scheduled.
3. All threads, processes, containers, VMs, MicroVMs must be shut down gracefully (e.g. sending SIGTERM to processes, waiting for x seconds and then sending SIGKILL if they have not shut down; sending ACPI shutdown to VMs, power off after timeout).
4. The GRPC API needs to be stopped.
5. The API socket needs to be deleted.