It'd be nice if we could have our CI verifying that each and every commit from a PR can be properly built. The main reason for that is to be able to run a git bisect if later in the future we're trying to understand where something might have gone wrong.

As @atishp04 and I have found, the other confidential compute implementations (pKVM, TDX, SEV) have the TVM decide what's MMIO and what isn't -- pKVM registers the regions via hypercalls, TDX & SEV have a more convoluted scheme where a page fault gets reflected back to the TVM which it can then turn into a hypercall. So, let's make tvm_add_emulated_mmio_region() a guest-side SBI call in a separate TEE-Guest extension to be consistent and avoid having to introduce our own KVM ioctl() to register MMIO registers on the host side.

We should probably also do the same with shared memory. The host can tell the TVM via device-tree or whatever where it wants the shared memory region to be, and then the guest effectively "accepts" that as the shared memory region by making a tvm_add_shared_memory_regino() SBI call.

In both cases we can notify the host that the TVM has registered these regions by setting up the shared-memory block to look like an ECALL exit from the TVM.

This patch defines the shared-memory structure used to communicate non-confidential state between the TSM and host, and the API to register it.

Two open questions:

• Should we try to replicate more standard RISC-V registers, or continue to use the custom fields/values we've defined previously? i.e. use sanitized scause, htval, htinst, etc values rather than our custom exit_cause0/exit_cause1 plus associated enums. The former will have better re-use with any sort of nested virtualization extension (and existing KVM code in general), while the latter will be more space efficient and theoretically mean less work for the OS/VMM to do on vCPU exit (e.g. no need to decode instructions again, or look up if a page fault is confidential or not). I ended up going with the standard RISC-V register approach here to reduce the overall surface area of the API. (I realize that my re-use of htinst for WFI isn't standard-compliant.)
• Do we want to reserve entire pages for these shared memory structs, or let the OS/VMM place them wherever they want (in non-confidential memory, of course)? I'm leaning towards the latter since it's mostly informational for the OS/VMM, i.e. causing the structs to overlap or otherwise place them somewhere where they could be freely read/written won't violate the confidentiality of a TVM.

Thoughts?

I'm trying to use gdb to debug with the bazel-bin/salus, some steps are as below, but it looks like the gdb can't find the the source code for example main.rs based on symbol file bazel-bin/salus. Has anyone one worked with gdb ? Thanks

$ gdb-multiarch bazel-bin/salus $ set architecture riscv:rv64 $ target remote localhost:1234 $ set directories ./src/ $ add-symbol-file ./bazel-bin/test-workloads/guestvm $ add-symbol-file ./bazel-bin/test-workloads/tellus

In the recent updates to Nacl extension [0], feature probing was introduced and allows guest hypervisor to probe which features are supported by the host hypervisor.

Also, the SetShmem FID argument has changed from pfn to full address of the shared memory page.

[0] https://lists.riscv.org/g/sig-hypervisors/message/256

In anticipation for the huge pages support, the way to share/unshare memory from a TVM is updated with a more explicit model that expects the host to drive the entire operation. Relying on some new TEE-Host functions, the host can invalidate and remove a specific range of memory based on the request from the TVM.

Can we implement SBI debug console extension in Salus ? https://lists.riscv.org/g/tech-unixplatformspec/message/1815 The Linux patches can be found in the above email. Anup reports that it improves guest booting time by ~30%. It should help boot time of Linux in Salus as well

This patch series adds PMU counter support.

1. The first commit adds PMU SBI extension types
2. The second commit adds convenience wrappers for the SBI calls
3. The third commit adds a PMU driver
4. The fourth implements a pass-through layer for PMU SBI calls
5. The fifth commit adds some counter booking
6. The sixth commit adds rudimentary support for counter save and restore
7. The seventh commit adds PMU counter virtualization for HW counters
8. The final commit adds test functionality

If host scheduler tries to schedule more number of vcpus on a pcpu that its number of imsic file, it will result in TVM failure. This is a common scenario when we run two TVMs with 2 vcpus which is a common test case.

Increasing the number of imsic files only helps to run the bare minimum setup in multi-TVM SMP configuration.

In the long run, the host kernel can't do much about it except notifying the VMM about the correct reason i.e. insufficient imsic files. Its upto the VMM to decide whether it wants to show appropriate error to the user or perform lazy pinning if possible.

This needs to be merged after https://github.com/rivosinc/sbi-rs/pull/15

As mentioned in that PR, This patch only updates the extension name & ID as per the latest naming changes in the spec. We can modify all the reference of "tee" to cove as well to align it well or leave it as it is for now.

Instead of having single call that writes the layout to host memory and that the host must retry if they provide insufficient buffer space, take an approach similar to the SBI PMU interface where a one call returns the number of register sets (TvmCpuNumRegisterSets) and another returns the location of a specific register set (TvmCpuGetRegisterSet).

As part of this change, RegisterSetLocation is compressed to a single u32 so that it's guaranteed to fit in a single general-purpose register. The version field is dropped and instead the versioning is implicit from the RegisterSetId.

While playing around with Salus, I notice that a virtual IOMMU was enabled since there are prints(when booting debian image) like: Found RISC-V IOMMU version 0x2,and

[    0.518950] iommu: Default domain type: Translated 
[    0.519827] iommu: DMA domain TLB invalidation policy: strict mode 

Then I find a file called riscv_iommu.c (https://github.com/rivosinc/qemu/blob/salus-integration-10312022/hw/riscv/riscv_iommu.c), after reading the code I believe that this c file is the implementation codes for the virtual IOMMU. I noticed that the vIOMMU was designed as a PCI device as the TypeInfo implies:

static const TypeInfo riscv_iommu_pci = {
    .name = TYPE_RISCV_IOMMU_PCI,
    .parent = TYPE_PCI_DEVICE,
    .instance_size = sizeof(RISCVIOMMUStatePci),
    .class_init = riscv_iommu_pci_init,
    .interfaces = (InterfaceInfo[]) {
        { INTERFACE_PCIE_DEVICE },
        { },
    },
};

For comparison, the TypeInfo in intel_iommu.c(https://github.com/rivosinc/qemu/blob/salus-integration-10312022/hw/i386/intel_iommu.c) imples:

static const TypeInfo vtd_info = {
    .name          = TYPE_INTEL_IOMMU_DEVICE,
    .parent        = TYPE_X86_IOMMU_DEVICE,
    .instance_size = sizeof(IntelIOMMUState),
    .class_init    = vtd_class_init,
};

Here comes the first question. Why implement the IOMMU as a PCI device? As my humble knowledge goes, a typical IOMMU usually be integrated with the CPU. Maybe doing this for developing convenience?

Put aside the first question, I then notice that:

k->vendor_id = PCI_VENDOR_ID_RIVOS;
k->device_id = PCI_DEVICE_ID_RIVOS_IOMMU;
...
k->class_id = 0x0806;

in the riscv_iommu_pci_init function. I decided to observe the behaviors of the vIOMMU. In qemu-monitor, using info pci, it says:

(qemu) info pci
info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Class 0264: PCI device 1b36:0010
      PCI subsystem 1af4:1100
      IRQ 0, pin A
      BAR0: 64 bit memory at 0x100000000 [0x100003fff].
      id ""
  Bus  0, device   2, function 0:
    Class 2054: PCI device 1efd:edf1
      PCI subsystem 1af4:1100
      BAR0: 64 bit memory at 0x17ffff000 [0x17fffffff].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1041
      PCI subsystem 1af4:1100
      IRQ 0, pin A
      BAR1: 32 bit memory at 0x40000000 [0x40000fff].
      BAR4: 64 bit prefetchable memory at 0x100004000 [0x100007fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""

Since the PCI_VENDOR_ID_RIVOS,PCI_DEVICE_ID_RIVOS_IOMMU,0x0806 equals to 0x1efd,0xedf1,2054D respectively, I believed that device 2 is the riscv_iommu_pci device. But in debian, trying to find the vIOMMU using lspci -v , it only says:

root@debian:~# lspci -v 
00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
	Subsystem: Red Hat, Inc. QEMU PCIe Host bridge
	Flags: fast devsel
lspci: Unable to load libkmod resources: error -2

00:01.0 Non-Volatile memory controller: Red Hat, Inc. QEMU NVM Express Controller (rev 02) (prog-if 02 [NVM Express])
	Subsystem: Red Hat, Inc. QEMU NVM Express Controller
	Flags: bus master, fast devsel, latency 0
	Memory at 100000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] MSI-X: Enable+ Count=65 Masked-
	Capabilities: [80] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [60] Power Management version 3
	Kernel driver in use: nvme

00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
	Subsystem: Red Hat, Inc. Virtio network device
	Flags: bus master, fast devsel, latency 0
	Memory at 40000000 (32-bit, non-prefetchable) [size=4K]
	Memory at 100004000 (64-bit, prefetchable) [size=16K]
	Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
	Kernel driver in use: virtio-pci

That leads to the other question. Why the riscv_iommu_pci device hasn't been recognized correctly by pciutils, but qemu-monitor seems normal? Possibly driver issues? mistaken setting? or any other problem?

I am researching the CoVE(or AP-TEE) currently, it would be very helpful to me if you answer my questions. :smiling_face_with_tear:

I'm testing code of the scripts/run_tellus.sh, any way to let the guestvm to access to the flash device defined in qemu ?

The flash device address is 0x20000000 https://github.com/rivosinc/qemu/blob/ae9f6df84c966e3608ed40fdacb55e2567946c60/hw/riscv/virt.c#LL91C9-L91C9

Thanks if any input

The goal is to optimize the way the PageTracker updates huge pages, by acting only on the head page rather than updating all the 4k pages from the huge page range.

When a huge page is mapped into a TVM, we can manage only the head page through the block/unblock lifecycle. When the TVM gets destroyed, we need all the pages to be back to a proper state, which is why we force the subpages to be the same state as the head page.

We also need to handle the case where a page range is shared differently. In that case, we can't assume two TVMs share the page at the same granularity, which is why we can't manipulate only the head page, but instead we must update all the subpages.

Last thing, we must handle a page demotion in a special way, by updating all the subpages states, given the demotion happened while the PageTracker was operating only on the head page of the huge page.

We should be inserting the guestvm binary into the tellus binary via a linker section, similar to how the u-mode binary is injected into salus, rather than the janky script we have that just appends the two binaries together.

cc: @stillson

Probably needs some minor linker script massaging to put these on a separate page from the data and bss segments. We should also consider having a separate segment for "RO after boot" data that gets initialized during boot and then marked RO post-boot.