Platform

• Hardware: t1.small.x86 Bare metal server from Packet with 1 x Intel Atom C2550 @ 2.4Ghz processor, 8GB RAM, 80 GB SSD
• OS: Ubuntu 16.04.5 LTS
• Kernel: 4.4.0-134-generic

Issue Runing InstanceStart action causes Firecracker process to crash with logging error.

Error Message 018-11-28T06:51:31.336002913 [:ERROR:vmm/src/lib.rs:1157] Failed to log metrics on abort. Failed to log metrics. Logger was not initialized.:?

Steps to reproduce

• Download the firecracker executable from Firecracker release page
• Start firecracker process using ./firecracker --api-sock /tmp/firecracker.sock command. No messages seen here.
• On a second console download kernel and rootfs to same directory as firecracker binary from links : kernel and rootfs
• Set guest kernel using command curl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/boot-source' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kernel_image_path": "./hello-vmlinux.bin", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off" }'
• set the guest rootfs using commandcurl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/drives/rootfs' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "drive_id": "rootfs", "path_on_host": "./hello-rootfs.ext4", "is_root_device": true, "is_read_only": false }'
• Start VM using command curl --unix-socket /tmp/firecracker.sock -i \ -X PUT 'http://localhost/actions' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "action_type": "InstanceStart" }'

Summary

• Logger is not initialised causing the process to crash when relevant logging is initiated code
• InstanceStart action fails leading to above case (?)

While developing the vhost-user-vsock application for cloud-hypervisor, I had an issue that happens also with the vsock device emulation: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1001

As @sboeuf suggested, I tried with Firecracker and I had a very similar issue:

2020-04-06T11:52:33.632842533 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1031, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

After the error, the throughput goes to 0 and the peers are able to communicate only restarting the connection, but the error happens every time I tried to run iperf-vsock (guest connecting to the host)

How to reproduce

I used iperf-vsock to stress the vsock connection.

Kernel: hello-vmlinux.bin (Linux (none) 4.14.55-84.37.amzn2.x86_64 #1 SMP Wed Jul 25 18:47:15 UTC 2018 x86_64 Linux) Firecracker: firecracker-v0.21.1-x86_64

VM config

{
  "boot-source": {
    "kernel_image_path": "./hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "ht_enabled": false
  },
  "vsock": {
      "vsock_id": "1",
      "guest_cid": 4,
      "uds_path": "/tmp/vm4.vsock"
  }
}

guest

~# mkdir /tmp     # iperf3 uses mkstemp(3)
~# iperf3 --vsock -c 2
Connecting to host 2, port 5201
[  5] local 4 port 1033 connected to 2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  91.0 MBytes   763 Mbits/sec                  
2020-04-06T11:55:51.001682775 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1033, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
[  5]   1.00-2.00   sec   422 MBytes  3.54 Gbits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   513 MBytes   431 Mbits/sec                  sender
[  5]   0.00-10.01  sec   513 MBytes   430 Mbits/sec                  receiver

iperf Done.

host

> ./iperf3 --vsock -s -B /tmp/vm4.vsock
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 40:8c44:a220:e9bb:30b4:aa01::, port 43521
[  5] local 6d34:2e76:736f:636b:5f35:3230:3100:0 port 12148 connected to :: port 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   431 MBytes   361 Mbits/sec                  receiver

Reason for This PR

The latest cargo-audit used in the Docker images requires Rust >=1.40. The current image still uses Rust 1.39.

Blocked by #1913

Fixes #1904

Description of Changes

The Dockerfiles have been changed to use Rust 1.43.1, the latest stable Rust version.

• [x] This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] The reason for this PR is clearly provided (issue no. or explanation).
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] Any newly added unsafe code is properly documented.
• [x] Any API changes are reflected in firecracker/swagger.yaml.
• [x] Any user-facing changes are mentioned in CHANGELOG.md.

Reason for This PR

The 2020 Roadmap welcomes contributions that adds initrd support. A previous PR was opened to add it (#670), but cannot be reused as is, because it also added support for loading a compressed kernel.

Hence #670 was reworked and rebased to only include initrd support. Fixes #208 .

Description of Changes

• Add initrd_path property to the boot-source API.
• Pass ramdisk_image (memory offset) and ramdisk_size to the guest bootparams table.
• Implement the load_initrd loader function, to load the initrd file content into the guest memory ~~at a (for the moment) fixed address~~ at then end of the available low memory (for x86_64).

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria. Where there are two options, keep one.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] Either this PR is linked to an issue, or, the reason for this PR is clearly provided.
• [x] The description of changes is clear and encompassing.
• [x] Either no docs need to be updated as part of this PR, or, the required doc changes are included in this PR. Docs in scope are all *.md files located either in the repository root, or in the docs/ directory.
• [x] Either no code has been touched, or, code-level documentation for touched code is included in this PR.
• [x] Either no API changes are included in this PR, or, the API changes are reflected in firecracker/swagger.yaml.
• [x] Either the changes in this PR have no user impact, or, the changes in this PR have user impact and have been added to the CHANGELOG.md file.

Fixed the issue where GET /vm/config would return the default config for a VM restored from a snapshot. The config now updates after restoring from a snapshot.

Reason for This PR

This PR exists to fix the bug described in issue #2783.

Description of Changes

Changed restoring a snapshot to include updating the VmResources so that the correct config would be returned for the GET /vm/config endpoint. The only parts of the config that are still using default values after restoring a snapshot are the following: boot-source (which is reasonable since there is no boot-source in the case of a restore), machine-config.smt and machine-config.cpu_template.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria.] [Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [x] All commits in this PR are signed (git commit -s).
• [x] The issue which led to this PR has a clear conclusion.
• [x] This PR follows the solution outlined in the related issue.
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] Any newly added unsafe code is properly documented.
• [x] Any API changes follow the Runbook for Firecracker API changes.
• [x] Any user-facing changes are mentioned in CHANGELOG.md.
• [x] All added/changed functionality is tested.

We want to have vsock support to enable container integration, but we don't want to use vhost since that would be another attack surface to directly exposes the host kernel. Instead, we'll write another back-end for vsock. See #194.

Currenty, virtio vsock exists in the codebase as an experimental development-only rust feature, with the vhost implementation.

Describe the bug

[Author TODO: A clear and concise description of what the bug is.]

root@ubuntu:~/myfirecracker/firecracker# firecracker --api-sock /tmp/firecracker.socket Bad system call (core dumped)

To Reproduce

1、 start firecrakcer: firecracker --api-sock /tmp/firecracker.socket 2、 send curl request curl --unix-socket /tmp/firecracker.socket -i
-X PUT 'http://localhost/boot-source'
-H 'Accept: application/json'
-H 'Content-Type: application/json'
-d "{ "kernel_image_path": "${kernel_path}", "boot_args": "keep_bootcon console=ttyS0 reboot=k panic=1 pci=off" }" curl: (52) Empty reply from server curl: (7) Couldn't connect to server curl: (7) Couldn't connect to server

Environment

uname -a Linux ubuntu 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:10:24 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

the firecrakcer is builded as following: root@ubuntu:~/myfirecracker# git clone https://github.com/firecracker-microvm/firecracker.git Cloning into 'firecracker'... remote: Enumerating objects: 82, done. remote: Counting objects: 100% (82/82), done. remote: Compressing objects: 100% (69/69), done. remote: Total 25405 (delta 31), reused 25 (delta 12), pack-reused 25323 Receiving objects: 100% (25405/25405), 17.58 MiB | 5.38 MiB/s, done. Resolving deltas: 100% (15370/15370), done. root@ubuntu:~/myfirecracker# cd firecracker root@ubuntu:~/myfirecracker/firecracker# tools/devtool build

This is a rebase of https://github.com/firecracker-microvm/firecracker/pull/428 - after various repo shuffles I'm not able to update that PR with the new version.

It adds support for loading bzImage kernels and initrds as an alternative to ELF64 kernels. It's slower than the vmlinux format because the kernel must decompress itself, but useful for trying kernels that I didn't build myself.

For now, it loads the initrd at a fixed address; it would be better to have a way of finding a free area of memory after the kernel has loaded.

• Updates Firecracker to the latest Tokio revision. This removes dependencies on deprecated libraries and introduces a global Tokio event loop, removing the need to juggle Tokio handles.
• As part of using the latest Tokio revision, I updated Firecracker to Hyper 0.12. While this allows Firecracker to use the new Tokio library, it does remove the typed headers. They are in development at hyperium/headers.
• Tokio now imposes a Sync constraint on spawned futures, as the future could potentially run on a threadpool. Since std::sync::mpsc::Sender is not Sync, I introduced Crossbeam, whose senders are Sync. I was not aware of SyncSender at the time. On bounded channels, Crossbeam greatly outperforms. This change can be reverted if necessary.

Still to do:

• Run integration tests on a Linux machine. This builds, but I didn’t have a Linux machine handy. Sorry about that; I’ll address that soon.
• Decide whether to keep Crossbeam.

Signed-off-by: David Barsky [email protected]

There are a few Jailers already around that support creation of Linux/BSD Jails the two biggest are Docker CE (containerD, moby) and SystemdNS

I would love to see that we get this Project to choose one of this Options as additional jailer

It will enable firecracker to be spawned in all this environments so it can be deployed in a local Kubernetes Cluster (containerd & moby) or CoreOs installation (SystemdNS)

Reason for This PR

Increase cgroups integration flexibility for the Jailer. Fixes #1937 Fixes #1617

Description of Changes

New --cgroups argument have been added to the Jailer. This argument takes care of setting the passed cgroups generically and check that the expected value is correctly written.

The Cgroup type have been modified in order to represent each of the cgroups passed by --cgroups command line argument.

• [ ] This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria.] [Reviewer TODO: Verify that these criteria are met. Request changes if not]

• [ ] All commits in this PR are signed (git commit -s).
• [ ] The reason for this PR is clearly provided (issue no. or explanation).
• [ ] The description of changes is clear and encompassing.
• [ ] Any required documentation changes (code and docs) are included in this PR.
• [ ] Any newly added unsafe code is properly documented.
• [ ] Any API changes are reflected in firecracker/swagger.yaml.
• [ ] Any user-facing changes are mentioned in CHANGELOG.md.

Changes

Adds additional testing onto the cpuid crate in the feature/cpu-templates feature branch.

Reason

Increases safety.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [x] All commits in this PR are signed (git commit -s).
• [x] If a specific issue led to this PR, this PR closes the issue.
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] New unsafe code is documented.
• [x] API changes follow the Runbook for Firecracker API changes.
• [x] User-facing changes are mentioned in CHANGELOG.md.
• [x] All added/changed functionality is tested.
• [x] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.

Changes

Changes code coverage to grcov.

Reason

kcov outputs inaccurate coverage.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [x] All commits in this PR are signed (git commit -s).
• [x] If a specific issue led to this PR, this PR closes the issue.
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] New unsafe code is documented.
• [x] API changes follow the Runbook for Firecracker API changes.
• [x] User-facing changes are mentioned in CHANGELOG.md.
• [x] All added/changed functionality is tested.
• [x] New TODOs link to an issue.

• [ ] This functionality can be added in rust-vmm.

Changes

Follow-up cleanups after the latest release changes.

Reason

Extract release functionality out of devtool and simplify devtool.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [x] All commits in this PR are signed (git commit -s).
• ~[ ] If a specific issue led to this PR, this PR closes the issue.~
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• ~[ ] New unsafe code is documented.~
• ~[ ] API changes follow the Runbook for Firecracker API changes.~
• ~[ ] User-facing changes are mentioned in CHANGELOG.md.~
• [x] All added/changed functionality is tested.
• ~[ ] New TODOs link to an issue.~

• ~[ ] This functionality can be added in rust-vmm.~

Signed-off-by: Luminita Voicu [email protected]

Changes

Update AMD performance baselines for TCP throughput tests.

Reason

We have recently merged some network improvements (see #2958) and only ARM and Intel performance baselines were updated.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [X] All commits in this PR are signed (git commit -s).
• ~[ ] If a specific issue led to this PR, this PR closes the issue.~
• [X] The description of changes is clear and encompassing.
• ~[ ] Any required documentation changes (code and docs) are included in this PR.~
• ~[ ] New unsafe code is documented.~
• ~[ ] API changes follow the Runbook for Firecracker API changes.~
• ~[ ] User-facing changes are mentioned in CHANGELOG.md.~
• ~[ ] All added/changed functionality is tested.~
• ~[ ] New TODOs link to an issue.~

• ~[ ] This functionality can be added in rust-vmm.~

Describe the bug

Some intermittent failures can be noticed for tests that configure microVMs that exceed the one memory region layout on x86. Like test_create_large_diff_snapshot. The test fails only sometimes because most of the times the memory monitor does not get a chance to run a second time (we have a 1s gap between runs) before the test finishes.

To Reproduce

• decrease MEMORY_SAMPLE_TIMEOUT_S to 0.5
• run test_create_large_diff_snapshot

Expected behaviour

• aforementioned test passes all the time

Changes

Use dedicated kill switch mechanism to break out of micro-http loop and gracefully shutdown the API server thread.

Remove hacky "/shutdown-internal" HTTP request which was externally exposing a way to bring down parts of firecracker components.

Dependencies

Ideally needs https://github.com/firecracker-microvm/micro-http/pull/33 merged first, then point the dependency there.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

• [x] All commits in this PR are signed (git commit -s).
• [x] ~If a specific issue led to this PR, this PR closes the issue.~
• [x] The description of changes is clear and encompassing.
• [x] Any required documentation changes (code and docs) are included in this PR.
• [x] ~New unsafe code is documented.~
• [x] ~API changes follow the Runbook for Firecracker API changes.~
• [x] ~User-facing changes are mentioned in CHANGELOG.md.~
• [x] All added/changed functionality is tested.
• [x] ~New TODOs link to an issue.~

• [x] ~This functionality can be added in rust-vmm.~