Platform

• Hardware: t1.small.x86 Bare metal server from Packet with 1 x Intel Atom C2550 @ 2.4Ghz processor, 8GB RAM, 80 GB SSD
• OS: Ubuntu 16.04.5 LTS
• Kernel: 4.4.0-134-generic

Issue Runing InstanceStart action causes Firecracker process to crash with logging error.

Error Message 018-11-28T06:51:31.336002913 [:ERROR:vmm/src/lib.rs:1157] Failed to log metrics on abort. Failed to log metrics. Logger was not initialized.:?

Steps to reproduce

• Download the firecracker executable from Firecracker release page
• Start firecracker process using ./firecracker --api-sock /tmp/firecracker.sock command. No messages seen here.
• On a second console download kernel and rootfs to same directory as firecracker binary from links : kernel and rootfs
• Set guest kernel using command curl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/boot-source' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kernel_image_path": "./hello-vmlinux.bin", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off" }'
• set the guest rootfs using commandcurl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/drives/rootfs' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "drive_id": "rootfs", "path_on_host": "./hello-rootfs.ext4", "is_root_device": true, "is_read_only": false }'
• Start VM using command curl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/actions' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "action_type": "InstanceStart" }'

Summary

• Logger is not initialised causing the process to crash when relevant logging is initiated code
• InstanceStart action fails leading to above case (?)

While developing the vhost-user-vsock application for cloud-hypervisor, I had an issue that happens also with the vsock device emulation: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1001

As @sboeuf suggested, I tried with Firecracker and I had a very similar issue:

2020-04-06T11:52:33.632842533 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1031, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

After the error, the throughput goes to 0 and the peers are able to communicate only restarting the connection, but the error happens every time I tried to run iperf-vsock (guest connecting to the host)

How to reproduce

I used iperf-vsock to stress the vsock connection.

Kernel: hello-vmlinux.bin (Linux (none) 4.14.55-84.37.amzn2.x86_64 #1 SMP Wed Jul 25 18:47:15 UTC 2018 x86_64 Linux) Firecracker: firecracker-v0.21.1-x86_64

VM config

{
  "boot-source": {
    "kernel_image_path": "./hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "ht_enabled": false
  },
  "vsock": {
      "vsock_id": "1",
      "guest_cid": 4,
      "uds_path": "/tmp/vm4.vsock"
  }
}

guest

~# mkdir /tmp     # iperf3 uses mkstemp(3)
~# iperf3 --vsock -c 2
Connecting to host 2, port 5201
[  5] local 4 port 1033 connected to 2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  91.0 MBytes   763 Mbits/sec                  
2020-04-06T11:55:51.001682775 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1033, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
[  5]   1.00-2.00   sec   422 MBytes  3.54 Gbits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   513 MBytes   431 Mbits/sec                  sender
[  5]   0.00-10.01  sec   513 MBytes   430 Mbits/sec                  receiver

iperf Done.

host

> ./iperf3 --vsock -s -B /tmp/vm4.vsock
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 40:8c44:a220:e9bb:30b4:aa01::, port 43521
[  5] local 6d34:2e76:736f:636b:5f35:3230:3100:0 port 12148 connected to :: port 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   431 MBytes   361 Mbits/sec                  receiver

Reason for This PR

The latest cargo-audit used in the Docker images requires Rust >=1.40. The current image still uses Rust 1.39.

Blocked by #1913

Fixes #1904

Description of Changes

The Dockerfiles have been changed to use Rust 1.43.1, the latest stable Rust version.

• [x] This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] The reason for this PR is clearly provided (issue no. or explanation).
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] Any newly added unsafe code is properly documented.
• [x] Any API changes are reflected in firecracker/swagger.yaml.
• [x] Any user-facing changes are mentioned in CHANGELOG.md.

Reason for This PR

The 2020 Roadmap welcomes contributions that adds initrd support. A previous PR was opened to add it (#670), but cannot be reused as is, because it also added support for loading a compressed kernel.

Hence #670 was reworked and rebased to only include initrd support. Fixes #208 .

Description of Changes

• Add initrd_path property to the boot-source API.
• Pass ramdisk_image (memory offset) and ramdisk_size to the guest bootparams table.
• Implement the load_initrd loader function, to load the initrd file content into the guest memory ~~at a (for the moment) fixed address~~ at then end of the available low memory (for x86_64).

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria. Where there are two options, keep one.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] Either this PR is linked to an issue, or, the reason for this PR is clearly provided.
• [x] The description of changes is clear and encompassing.
• [x] Either no docs need to be updated as part of this PR, or, the required doc changes are included in this PR. Docs in scope are all *.md files located either in the repository root, or in the docs/ directory.
• [x] Either no code has been touched, or, code-level documentation for touched code is included in this PR.
• [x] Either no API changes are included in this PR, or, the API changes are reflected in firecracker/swagger.yaml.
• [x] Either the changes in this PR have no user impact, or, the changes in this PR have user impact and have been added to the CHANGELOG.md file.

Fixed the issue where GET /vm/config would return the default config for a VM restored from a snapshot. The config now updates after restoring from a snapshot.

Reason for This PR

This PR exists to fix the bug described in issue #2783.

Description of Changes

Changed restoring a snapshot to include updating the VmResources so that the correct config would be returned for the GET /vm/config endpoint. The only parts of the config that are still using default values after restoring a snapshot are the following: boot-source (which is reasonable since there is no boot-source in the case of a restore), machine-config.smt and machine-config.cpu_template.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria.] [Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] The issue which led to this PR has a clear conclusion.
• [x] This PR follows the solution outlined in the related issue.
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] Any newly added unsafe code is properly documented.
• [x] Any API changes follow the Runbook for Firecracker API changes.
• [x] Any user-facing changes are mentioned in CHANGELOG.md.
• [x] All added/changed functionality is tested.

We want to have vsock support to enable container integration, but we don't want to use vhost since that would be another attack surface to directly exposes the host kernel. Instead, we'll write another back-end for vsock. See #194.

Currenty, virtio vsock exists in the codebase as an experimental development-only rust feature, with the vhost implementation.

Describe the bug

[Author TODO: A clear and concise description of what the bug is.]

root@ubuntu:~/myfirecracker/firecracker# firecracker --api-sock /tmp/firecracker.socket Bad system call (core dumped)

To Reproduce

1、 start firecrakcer: firecracker --api-sock /tmp/firecracker.socket 2、 send curl request curl --unix-socket /tmp/firecracker.socket -i
-X PUT 'http://localhost/boot-source'
-H 'Accept: application/json'
-H 'Content-Type: application/json'
-d "{ "kernel_image_path": "${kernel_path}", "boot_args": "keep_bootcon console=ttyS0 reboot=k panic=1 pci=off" }" curl: (52) Empty reply from server curl: (7) Couldn't connect to server curl: (7) Couldn't connect to server

Environment

uname -a Linux ubuntu 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:10:24 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

the firecrakcer is builded as following: root@ubuntu:~/myfirecracker# git clone https://github.com/firecracker-microvm/firecracker.git Cloning into 'firecracker'... remote: Enumerating objects: 82, done. remote: Counting objects: 100% (82/82), done. remote: Compressing objects: 100% (69/69), done. remote: Total 25405 (delta 31), reused 25 (delta 12), pack-reused 25323 Receiving objects: 100% (25405/25405), 17.58 MiB | 5.38 MiB/s, done. Resolving deltas: 100% (15370/15370), done. root@ubuntu:~/myfirecracker# cd firecracker root@ubuntu:~/myfirecracker/firecracker# tools/devtool build

This is a rebase of https://github.com/firecracker-microvm/firecracker/pull/428 - after various repo shuffles I'm not able to update that PR with the new version.

It adds support for loading bzImage kernels and initrds as an alternative to ELF64 kernels. It's slower than the vmlinux format because the kernel must decompress itself, but useful for trying kernels that I didn't build myself.

For now, it loads the initrd at a fixed address; it would be better to have a way of finding a free area of memory after the kernel has loaded.

• Updates Firecracker to the latest Tokio revision. This removes dependencies on deprecated libraries and introduces a global Tokio event loop, removing the need to juggle Tokio handles.
• As part of using the latest Tokio revision, I updated Firecracker to Hyper 0.12. While this allows Firecracker to use the new Tokio library, it does remove the typed headers. They are in development at hyperium/headers.
• Tokio now imposes a Sync constraint on spawned futures, as the future could potentially run on a threadpool. Since std::sync::mpsc::Sender is not Sync, I introduced Crossbeam, whose senders are Sync. I was not aware of SyncSender at the time. On bounded channels, Crossbeam greatly outperforms. This change can be reverted if necessary.

Still to do:

• Run integration tests on a Linux machine. This builds, but I didn’t have a Linux machine handy. Sorry about that; I’ll address that soon.
• Decide whether to keep Crossbeam.

Signed-off-by: David Barsky [email protected]

There are a few Jailers already around that support creation of Linux/BSD Jails the two biggest are Docker CE (containerD, moby) and SystemdNS

I would love to see that we get this Project to choose one of this Options as additional jailer

It will enable firecracker to be spawned in all this environments so it can be deployed in a local Kubernetes Cluster (containerd & moby) or CoreOs installation (SystemdNS)

Reason for This PR

Increase cgroups integration flexibility for the Jailer. Fixes #1937 Fixes #1617

Description of Changes

New --cgroups argument have been added to the Jailer. This argument takes care of setting the passed cgroups generically and check that the expected value is correctly written.

The Cgroup type have been modified in order to represent each of the cgroups passed by --cgroups command line argument.

• [ ] This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria.] [Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [ ] All commits in this PR are signed (git commit -s).
• [ ] The reason for this PR is clearly provided (issue no. or explanation).
• [ ] The description of changes is clear and encompassing.
• [ ] Any required documentation changes (code and docs) are included in this PR.
• [ ] Any newly added unsafe code is properly documented.
• [ ] Any API changes are reflected in firecracker/swagger.yaml.
• [ ] Any user-facing changes are mentioned in CHANGELOG.md.

Feature Request

I'm aware of the vsock device limitations described in snapshot-support.md.

However there's no mentioning of how to support multiple clones of a snapshot with a configured vsock on the same host (they'll all share the same CID). For network interfaces, Network Connectivity for Clones describes pretty nicely how multiple clones can be run on the same host by using network namespaces. But to my knowledge, there's no network namespace support available for vsock sockets (I only found "vsock: support network namespace" which doesn't seem to have been finished yet).

I tried to configure a different vsock CID/path before restoring a snapshot, but in that case restoring fails with:

2022-12-30T20:46:38.507150111 [anonymous-instance:fc_api:ERROR:src/api_server/src/parsed_request.rs:190] Received Error. Status code: 400 Bad Request. Message: Loading a microVM snapshot not allowed after configuring boot-specific resources.

Describe the desired solution

The vsock man page mentions the following about "live migrations":

   Sockets are affected by live migration of virtual machines.
   Connected SOCK_STREAM sockets become disconnected when the
   virtual machine migrates to a new host.  Applications must
   reconnect when this happens.

   The local CID may change across live migration if the old CID is
   not available on the new host.  Bound sockets are automatically
   updated to the new CID.

I wonder if it wouldn't be possible to implement the vsock support in Firecracker such that the CID/socket path can be changed for a snapshot at restore time?

In addition to allowing multiple clones with configured vsockets, such a change would also make it possible to use the vsock device for obtaining a simple SysGenID (see #2546).

Please let me know if there's anything obvious that I'm missing?

Checks

• [x] Have you searched the Firecracker Issues database for similar requests?
• [x] Have you read all the existing relevant Firecracker documentation?
• [x] Have you read and understood Firecracker's core tenets?

Reason

Rust toolchain upgrade from 1.64 to 1.66.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [ ] All commits in this PR are signed (git commit -s).
• [ ] If a specific issue led to this PR, this PR closes the issue.
• [ ] The description of changes is clear and encompassing.
• [ ] Any required documentation changes (code and docs) are included in this PR.
• [ ] New unsafe code is documented.
• [ ] API changes follow the Runbook for Firecracker API changes.
• [ ] User-facing changes are mentioned in CHANGELOG.md.
• [ ] All added/changed functionality is tested.
• [ ] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.

Reason

Rust toolchain upgrade from 1.64 to 1.66.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [ ] All commits in this PR are signed (git commit -s).
• [ ] If a specific issue led to this PR, this PR closes the issue.
• [ ] The description of changes is clear and encompassing.
• [ ] Any required documentation changes (code and docs) are included in this PR.
• [ ] New unsafe code is documented.
• [ ] API changes follow the Runbook for Firecracker API changes.
• [ ] User-facing changes are mentioned in CHANGELOG.md.
• [ ] All added/changed functionality is tested.
• [ ] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.

Changes

Rename vm-memory to vm-memory-ext and remove renaming of vm-memory to vm-memory-upstream.

Reason

Renaming dependencies negatively affects readability, one will typically assume when seeing use some_crate in the code base it refers to the external crate some_crate which could be looked up at crates.io/crates/some_crate.

By removing this alias and renaming the local crate, readability is improved.

This was previously a part of https://github.com/firecracker-microvm/firecracker/pull/3079.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [ ] All commits in this PR are signed (git commit -s).
• [ ] If a specific issue led to this PR, this PR closes the issue.
• [ ] The description of changes is clear and encompassing.
• [ ] Any required documentation changes (code and docs) are included in this PR.
• [ ] New unsafe code is documented.
• [ ] API changes follow the Runbook for Firecracker API changes.
• [ ] User-facing changes are mentioned in CHANGELOG.md.
• [ ] All added/changed functionality is tested.
• [ ] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.

Changes

Specify the tap name in unittests via %d to make the kernel pick a name for us that is ensured to not be in used by a separate process already.

Reason

spurious failures with "device busy" because of name clashes with tap devices.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [x] All commits in this PR are signed (git commit -s).
• [x] If a specific issue led to this PR, this PR closes the issue.
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] New unsafe code is documented.
• [x] API changes follow the Runbook for Firecracker API changes.
• [x] User-facing changes are mentioned in CHANGELOG.md.
• [x] All added/changed functionality is tested.
• [x] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.