Description

I tried to use the rancher/k3d image from DockerHub and create a VM with it. I assumed that this would fail, but I would love to piece together why.

[root at fedora in ~/krunvm] (101) (main)
% RUST_BACKTRACE=1 krunvm create rancher/k3d --name k3d-test
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/create.rs:134:36
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: krunvm::create::create
   4: krunvm::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I believe in the .unwrap() call on line 134, an error was returned from the below function.

https://github.com/containers/krunvm/blob/367898a940585ce592e1e5e90f0690b568a78b62/src/create.rs#L11-L18

In order to figure out what happened, I would love to be able to see the exact error encountered from the called function. From my own edits, I can see that it occurred at this line...

let mut file = File::create(resolvconf)?; 

However, I did not find the above line without running cargo run with some edits and logging added. Perhaps, there could be recommended suggestions based on what kind of error occurred and where :)

Environment

Command | Output --- | --- krunvm -V | krunvm 0.1.4 uname -r | 5.10.15-200.fc33.x86_64 cat /etc/os-release \| grep "^PRETTY_NAME" | PRETTY_NAME="Fedora 33 (Server Edition)"

Other

Thank you! This is a wonderful project, by the way.

Environment: MacOS with VirtualBox 6.1.18 hosting Ubuntu 20.04 with Guest additions enabled

1. Installed libkrunfw from source
2. Installed libkrun from source.
3. git clone krunvm and build

Normally running it with LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH ./target/release/krunvm list throws a Please re-run krunvm inside a "buildah unshare" session

So I do a buildah unshare and then create the VM with LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH ./target/release/krunvm create debian --name deb-test This creates the image which can be verified with krunvm list

When trying to execute this with LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH ./target/release/krunvm start deb-test /bin/bash the Virtualbox VM completely crashes. Looking at the VBox log file just has this:

00:01:35.155450 emR3Debug: rc=VERR_EM_INTERNAL_ERROR

Not sure if Im making some basic errors, but pls let know if you need any more debug logs.

On a new ubuntu VM I'm running nvm install 14 which compiles nodejs from source. After a while, the VM crashes with:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: DecodeMessage(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })', src/devices/src/virtio/fs/device.rs:176:18
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: devices::virtio::fs::device::Fs::process_queue
   4: devices::virtio::fs::event_handler::<impl polly::event_manager::Subscriber for devices::virtio::fs::device::Fs>::process
   5: _krun_start_enter
   6: krunvm::start::start
   7: krunvm::main

Some context, using macOS 12.4 (21F79):

stephen@boris-godunov ~ % uname -a
Darwin boris-godunov 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000 arm64
stephen@boris-godunov ~ % krunvm --version
krunvm 0.2.1
stephen@boris-godunov ~ % id
uid=501(stephen) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae)

To reproduce:

stephen@boris-godunov ~ % krunvm create --name test ubuntu:latest
microVM created with name: test
stephen@boris-godunov ~ % krunvm start test
root@test:/# mkdir test
root@test:/# cd test
root@test:/test# touch foo
root@test:/test# ls -lh
total 0
-rw-r--r-- 1 root root 0 Aug 10 22:47 foo
root@test:/test# ln -s foo bar
root@test:/test# ls -lh
total 0
lrwxrwxrwx 1  501 dialout 3 Aug 10 22:47 bar -> foo
-rw-r--r-- 1 root root    0 Aug 10 22:47 foo
root@test:/test# stat bar
  File: bar -> foo
  Size: 3         	Blocks: 0          IO Block: 4096   symbolic link
Device: 15h/21d	Inode: 1814521     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (  501/ UNKNOWN)   Gid: (   20/ dialout)
Access: 2022-08-10 22:47:49.795944806 +0000
Modify: 2022-08-10 22:47:49.795944806 +0000
Change: 2022-08-10 22:47:49.795944806 +0000
 Birth: 

Expected: a symlink created with ln -s should have user/group root/root. Actual: the symlink has uid/gid 501/20, same as the macOS user who started the krunvm vm.

$ krunvm create --name nix nixos/nix
microVM created with name: nix
$ krunvm start nix      
mount(/proc): No such file or directory

I'm not sure where the problem is here, but I'd be willing to help debug if necessary.

Hi,

I've been exploring krunvm to run a podman service that I will use from my M1 MacBook. I've been able to get it to work, except that port mappings are being ignore. Here are the steps to reproduce it:

• install podman 3.4.0 on a fresh fedora image krunvm create docker.io/library/fedora:latest
• run podman run -ti --rm -p 8080:80 nginx on the fedora host
• Logs say: Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
• I'm able to get a response from the container on port 80 (within the fedora host)
• When specifying the network explictly (podman run -ti --rm --net podman -p 8080:80 nginx) I get:

ERRO[0000] error loading cached network config: network "podman" not found in CNI cache
WARN[0000] falling back to loading from existing plugins on disk
ERRO[0000] Error tearing down partially created network namespace for container ecaaf8be15d7b321afdfa1454881af8d2b2aca3172f30589ef2af8cbc2b8ecc3: error removing pod elated_bell_elated_bell from CNI network "podman": neither iptables nor ip6tables usable
Error: error configuring network namespace for container ecaaf8be15d7b321afdfa1454881af8d2b2aca3172f30589ef2af8cbc2b8ecc3: error adding pod elated_bell_elated_bell to CNI network "podman": failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported

• Running the command from the MacBook results in:

Error: error preparing container f4558401e6f433e2c35c198296b6d53bc0e6a22f580ef9b344f5c7fd15a40ae7 for attach: error configuring network namespace for container f4558401e6f433e2c35c198296b6d53bc0e6a22f580ef9b344f5c7fd15a40ae7: error adding pod great_margulis_great_margulis to CNI network "podman": failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported

Any help is greatly appreciated!

host:
  arch: arm64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc34.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    variant: container
    version: "34"
  eventLogger: file
  hostname: fedora
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.10
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 629608448
  memTotal: 1038118912
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.aarch64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 19m 14.23s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.0
  Built: 1633030810
  BuiltTime: Thu Sep 30 19:40:10 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/arm64
  Version: 3.4.0

cat /etc/cni/net.d/87-podman.conflist

{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [{ "dst": "0.0.0.0/0" }],
        "ranges": [
          [
            {
              "subnet": "10.88.0.0/16",
              "gateway": "10.88.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "tuning"
    }
  ]
}

cat /etc/containers/containers.conf

[containers]
default_capabilities = [
    "CHOWN",
    "DAC_OVERRIDE",
    "FOWNER",
    "FSETID",
    "KILL",
    "NET_BIND_SERVICE",
    "SETFCAP",
    "SETGID",
    "SETPCAP",
    "SETUID",
    "SYS_CHROOT"
]
default_sysctls = [
 "net.ipv4.ping_group_range=0 0",
]
log_driver = "k8s-file"
netns = "host"
[secrets]
[secrets.opts]
[network]
[engine]
cgroup_manager = "cgroupfs"
events_logger = "file"
[engine.runtimes]
[engine.volume_plugins]

Thanks for providing such an excellent tool! But I haven't found how to use this tool Using docker on macOS is very complicated. Docker can only be installed in vm, which is very resource-intensive and inefficient. After seeing this tool, I want to create a vm, and then run docker or k3s in the vm, instead of qemu, docker for mac or other virtual machines, but I have no success, no matter it is docker, podman, k3s, can't Started in the image of krunvm, is there any way to use docker or k3s in the image of krunvm?

using make or cargo build --release, I get an error /usr/bin/ld: cannot find -lkrun.

I don't get where dependency comes from : os (apt) or rust (cargo).

Any idea ?

Adds the SECURITY.md and CODE-OF-CONDUCT.md templates that point to the main one in containers common. If you do not like these, please supply your own.

Signed-off-by: TomSweeneyRedHat [email protected]

Running the following shows an example of the problem:

% krunvm start fedora-rawhide /bin/ls -- -l /bin/bash
-rwxr-xr-x 1 root root 1433752 Jul 20 22:00 /bin/bash
% krunvm start fedora-rawhide /bin/bash -- -c 'echo $PATH'
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
% krunvm start fedora-rawhide bash

The last command does not succeed. You need to pass /bin/path for it to work.

Note that this is somewhat inconsistent with podman usage for example, where podman run -it ubi8 bash works.

Is there a way to run a VM headless and stop the VM without having to exit the process that was started by the VM? Without having to delete it? Maybe I missed it in the docs.

First, I'm not entirely sure how useful this report will be.

I was experimenting with Yocto Project builds inside a krunvm on my M1 MBP (16GB RAM). About ten minutes into the build my pointer got very laggy, it started beachballing, and then the machine rebooted (on restart, the diagnosis was that the WindowServer watchdog timed out).

Yocto was building a whole Linux distro including the cross toolchain from scratch, so it's CPU and I/O heavy.

I started by blaming the combination of both, but when I went to delete the build tree (a good 30GB) it crashed again, so it looks like it's just heavy I/O.

FWIW, my krunvm guest was configured with 8GB of RAM and 6 cores.

what did you do?

• cloned down $ git clone https://github.com/containers/krunvm
• cd into the newly cloned folder $ cd krunvm/
• started the build as per README.md $ cargo build --release

what was the outcome?

it failed to compile with the following error message:

error: failed to run custom build command for `krunvm v0.2.2 (/.../krunvm)`

Caused by:
  process didn't exit successfully: `/.../krunvm/target/release/build/krunvm-93a7e7b5f83e43bd/build-script-build` (exit status: 101)
  --- stderr
  Error from running 'asciidoctor': No such file or directory (os error 2)
  thread 'main' panicked at 'failed to generate man page: No such file or directory (os error 2)', build.rs:25:13
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

what was the expected outcome?

it should compile on a clean fresh clone

what are you system/OS software versions?

• Linux Ubuntu 20.04.5 LTS
• Rust v1.58.1

When trying to build a project within krunvm, I noticed relatively low CPU utilization compared to a Linux build.

Two screenshots attached to illustrate the effect, one building on Linux, the other building the same project on krunvm, in both cases using make -j12. I suspect that the use of virtiofs limits the I/O bandwidth sufficiently to starve the build, but I wonder if there is a good way to prove that. What makes me doubt, though, is that the WIO state (in red) is very low. Is that metric reliable on krunvm, or is there something that would impact it?

Linux build:

krunvm build:

The following message is a bit puzzling:

% krunvm  changevm f35 -v /Users/ddd:/home/ddd
Invalid volume, only single direct root children are supported as guest_path

With podman, it is perfectly fine to pass a directory like this. This can be worked around using a bind mount from a top-level directory, but I'm curious about the reason for that restriction to start with.

Running krunvm -v and any command has not shown any usable output for me so far. What is the expected usage for that option?

For example, is there a way to show the boot process output? I can run dmesg after the boot, but I suspect there must be a way for you to debug what's going on during boot.

I noticed that because of #20:

% krunvm start fedora-rawhide  /bin/ls -- -l /bin/false
-rwxr-xr-x 1 root root 69808 Aug  1 15:36 /bin/false
% krunvm start fedora-rawhide  /bin/false
% echo $?
0

The expected returned value would be 1, since that's what /bin/false returns.

This may make it difficult to use krunvm in scripts. For example, I could not write a host script that does a build and test whether the build succeeded.