Unlock vGPU functionality for consumer grade GPUs

Matt Bilker

Last update: Jan 5, 2023

Related tags

Utilities vgpu_unlock-rs

Overview

Rust-based vgpu_unlock

Unlock vGPU functionality for consumer-grade NVIDIA GPUs.

This tool is to be used with the kernel patches from the main vgpu_unlock repository!

Dependencies

This tool requires Rust. You can install it via your package manager or via .
Rust requires a linker to be installed to be able to create the shared library. Typically, this is installed with the C compiler through your distribution's package manager.
The dependencies from the main vgpu_unlock project excluding Python and frida.

Installation

In the following instructions needs to be replaced with the path to this repository on the target system.

Install the NVIDIA vGPU driver and kernel driver patches as detailed in the main vgpu_unlock project README. Ignore the steps regarding editing the systemd service unit files.

Run cargo build --release to compile the shared library.

Create the directories /etc/systemd/system/nvidia-vgpud.service.d and /etc/systemd/system/nvidia-vgpu-mgr.service.d.

Create the files /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf and /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf with the following:

[Service]
Environment=LD_PRELOAD=
   
    /target/release/libvgpu_unlock_rs.so

Create the directory /etc/vgpu_unlock which will house the vGPU profile override configuration file.

Create the file /etc/vgpu_unlock/profile_override.toml with the profile fields that are to be overridden. The following is an example that sets the number of heads to 1, sets the framebuffer to be 1920x1080 (1920 * 1080 = 2073600 pixels), enables CUDA, and disables the frame-rate limiter.

num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0

Happy hacking!

Comments

Same issue as #5 - Fails to start VM

Same issue as #5 - https://github.com/mbilker/vgpu_unlock-rs/issues/5

Host Setup:

Proxmox 7.2 on Debian
Quadro RTX 4000

Guest Setup

20.04 ubuntu desktop
Quadro RTX 6000

Followed Jeff (Craft Computings) guide on this software. Able to get list using "mdevctl types". Looking to split RTX4000 into 4x instances of 2Gb each

Error

root@p53:~# qm start 311
no efidisk configured! Using temporary efivars disk.
mdev instance '00000000-0000-0000-0000-000000000311' already existed, using it.
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000311,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000311: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000311,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000311: failed to read device config space: Bad address
start failed: QEMU exited with code 1
root@p53:~#

. Setup

root@p53:~# nvidia-smi
Mon May 23 17:13:41 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 4000     On   | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P8    15W /  N/A |     54MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@p53:~#

root@p53:~# systemctl stop nvidia-vgpu-mgr.service
root@p53:~# systemctl status nvidia-vgpu-mgr.service
● nvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpu-mgr.service.d
             └─vgpu_unlock.conf
     Active: inactive (dead) since Mon 2022-05-23 17:04:21 +08; 39s ago
    Process: 48056 ExecStart=/usr/bin/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
    Process: 53989 ExecStopPost=/bin/rm -rf /var/run/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
   Main PID: 48057 (code=exited, status=0/SUCCESS)
        CPU: 18ms

May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f>
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: vgpu(+0x3c1d) [0x562973203c1d]
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 3 (v>
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 3
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: display_init failed for inst: 0
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration >
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x5b
May 23 17:04:20 p53.rubicon.local systemd[1]: Stopping NVIDIA vGPU Manager Daemon...
May 23 17:04:21 p53.rubicon.local systemd[1]: nvidia-vgpu-mgr.service: Succeeded.
May 23 17:04:21 p53.rubicon.local systemd[1]: Stopped NVIDIA vGPU Manager Daemon.

root@p53:~# systemctl start nvidia-vgpu-mgr.service
root@p53:~# systemctl status nvidia-vgpu-mgr.service
● nvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpu-mgr.service.d
             └─vgpu_unlock.conf
     Active: active (running) since Mon 2022-05-23 17:05:34 +08; 3s ago
    Process: 54378 ExecStart=/usr/bin/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
   Main PID: 54379 (nvidia-vgpu-mgr)
      Tasks: 1 (limit: 76740)
     Memory: 304.0K
        CPU: 6ms
     CGroup: /system.slice/nvidia-vgpu-mgr.service
             └─54379 /usr/bin/nvidia-vgpu-mgr

May 23 17:05:34 p53.rubicon.local systemd[1]: Starting NVIDIA vGPU Manager Daemon...
May 23 17:05:34 p53.rubicon.local systemd[1]: Started NVIDIA vGPU Manager Daemon.
May 23 17:05:34 p53.rubicon.local nvidia-vgpu-mgr[54379]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started
root@p53:~#

root@p53:~# systemctl status nvidia-vgpud.service
● nvidia-vgpud.service - NVIDIA vGPU Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpud.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpud.service.d
             └─vgpu_unlock.conf
     Active: inactive (dead) since Mon 2022-05-23 17:16:45 +08; 1s ago
    Process: 58289 ExecStart=/usr/bin/nvidia-vgpud (code=exited, status=0/SUCCESS)
    Process: 58291 ExecStopPost=/bin/rm -rf /var/run/nvidia-vgpud (code=exited, status=0/SUCCESS)
   Main PID: 58290 (code=exited, status=0/SUCCESS)
        CPU: 245ms

May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: BAR1 Length: 0x100
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Frame Rate Limiter enabled: 0x1
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Number of Displays: 1
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Max pixels: 1310720
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Display: width 1280, height 1024
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: License: GRID-Virtual-Apps,3.0
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: PID file unlocked.
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: PID file closed.
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Shutdown (58290)
May 23 17:16:45 p53.rubicon.local systemd[1]: nvidia-vgpud.service: Succeeded.
root@p53:~#

root@p53:~# cat /etc/vgpu_unlock/profile_override.toml
[profile.nvidia-257]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 60
framebuffer = 1968526677
pci_id = 0x1E3012BA
pci_device_id = 0x1E30
root@p53:~#

root@p53:~# cat /etc/pve/nodes/p53/qemu-server/311.conf
agent: 1
args: -uuid 00000000-0000-0000-0000-000000000311
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
hostpci0: 0000:01:00.0,mdev=nvidia-257
ide2: local:iso/ubuntu-20.04.4-desktop-amd64.iso,media=cdrom,size=3299872K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1653292108
name: u311.u2004
net0: virtio=26:69:CE:5F:52:16,bridge=vmbr1,firewall=1,tag=30
numa: 0
ostype: l26
scsi0: local:311/vm-311-disk-0.qcow2,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=d2e8b2af-319f-44a5-aeb5-c0c449b7d92d
sockets: 1
vmgenid: f2a8f0f4-7b18-4a9c-903e-c4161e6d7abc
root@p53:~#

. Even if I proceed with the Guest VM (ubuntu 20.04) setup and install the nvidia driver, the driver will not run

(from guest vm)

$ sudo apt install nvidia-driver-470
$ nvidia-smi
NVIDIA_SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

Would appreciate any help to move this along. Thanks!

opened by elmark22efk 21

Fails to start VM

Setup: i5-10400f GTX1660SUPER Followed Jeff (Craft Computings) guide on this software

Error in GUI:

kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000102: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000102: failed to read device config space: Bad address

Error in nvidia-vgpu-mgr:

Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0xeb43) [0x5652bb60eb43]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0xc3b6) [0x5652bb60c3b6]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0x3bda) [0x5652bb603bda]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7ff8cba6cd0a]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0x3c1d) [0x5652bb603c1d]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 1 (error setting vGPU configuration information f>
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 1
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: display_init failed for inst: 0
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration error
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x1f

Someone with a solution?

opened by TechyGuy17 6

mdev specific nvidia profile overrides

In PoIIoLocos vGPU Proxmox guide there is chapter about the possibility to override certain VgpuConfig options. Is it possible to override the option [profile.nvidia-XXX] for specific VMs with [mdev.00000000-0000-0000-0000-000000000XXX]? Or is this a global variable which can’t be defined for specific VMs? For example I want to use [profile.nvidia-47]for [mdev.00000000-0000-0000-0000-000000000100] and [profile.nvidia-49]for [mdev.00000000-0000-0000-0000-000000000102]. I don't want use the VMs at the same time because I do unterstand that it's not possible to mix and match profiles. Currently if I want to use a linux guest I will have to use a nvidia b profile because the performance of nvidia q profiles is utterly terrible. This conflicts with the nvidia q profile which I want to use for a windows guest. At line 146 of lib.rs I can see that there are options like gpu_type and vgpu_type. What are these for?

opened by tristan-k 5
CUDA in win/linux guest not available when device override used
It's a rather curious case that affects CUDA capabilities when there's a device name override. I've got a few Teslas, M40, M40 24G, P100, and P4 and they all seem to exhibit the same issue.

For the windows guest - the acceleration works fine, taking the physical P100 as an example, I can spoof it as Quadro GP100 and the 3d rendering works ok, but CUDA doesn't

For Linux guests (edit: same observed with Windows), as soon as the card device id and the subsystem id get overridden the CUDA capabilities are lost. Without the override, everything works fine, including CUDA. Seeing that nvidia-smi doesn't report an overridden GPU name (taking it directly from the mdev profile name?) - but lspci does, I suspect that the driver has a dafaq moment and doesn't know whether it's a CUDA capable device or not. But that's just a guess. Initially I thought it might have been caused by the ROM BAR region being set to only 256M but that's not the case here. Without the override it still shows 256M, so BAR has to be handled somehow differently with vGPU.

The host happily reports the GPU is in C+G mode, indicating both Compute and GFX should be available, so there's something in the device id override that is making CUDA unavailable.

Host smi:

another host smi:

guest SMI (with override active):

CUDA test with an override:

rliwoch@ubuntu-vpu-test:~$ docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2 [Vector addition of 50000 elements] Failed to allocate device vector A (error code all CUDA-capable devices are busy or unavailable)!

Windows CUDA test with an override:

Linux guest, CUDA without override:

rliwoch@ubuntu-vgpu-test:~$ docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2 [Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done

I'm posting it as a rather curious case, as I said in the beginning. I'm not sure if the override is needed in this for Linux Guest and CUDA, it happily took a normal driver (not a GRID one) - so there doesn't seem to be licensing requirements there. Having said that I don't want to start using it and find out that the performance of CUDA is crippled after some time (read somewhere that it can go into the limp mode after a while?)

Any ideas as to why with an override the device suddenly stops accepting CUDA API calls?

for competness:

[profile.nvidia-86] num_displays = 1 cuda_enabled = 1 frl_enabled = 0 #Quadro GP100 #pci_device_id = 0x15F0 #pci_id = 0x15F011C3 #Windows box [mdev.00000000-0000-0000-0000-000000000103] pci_device_id = 0x15F0 pci_id = 0x15F011C3 #Linux box [mdev.00000000-0000-0000-0000-000000000102]
opened by rliwoch 4

use of unstable library feature 'renamed_spin_loop' Error.

I'm running into the following when running cargo build --release

error[E0658]: use of unstable library feature 'renamed_spin_loop'
 --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.3/src/spinwait.rs:9:5
  |
9 | use core::hint::spin_loop;
  |     ^^^^^^^^^^^^^^^^^^^^^
  |
  = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information

error[E0658]: use of unstable library feature 'renamed_spin_loop'
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.3/src/spinwait.rs:16:9
   |
16 |         spin_loop()
   |         ^^^^^^^^^
   |
   = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0658`.
error: could not compile `parking_lot_core`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
```

opened by absenth 4

Can we support RTX 30 Series (Ampere, GA102) ?

If can not using a normal driver. Can we use the grid driver(512.78_grid_win10_win11_server2016_server2019_server2022_64bit_international.exe)?

I have a RTX 3090 Ti 24GB card. Let me know if i can help you to debug.

https://github.com/mbilker/vgpu_unlock-rs/blob/master/src/lib.rs#L413 A6000 not work. So i change it to A10:

                // Ampere
                0x2200..=0x2600 => {
                    // A10
                    (0x2236, 0x1482)
                }

It looks like one step away from success:

9月 07 12:36:41 v-amd nvidia-vgpu-mgr[839]: VgpuStart {
                                                 uuid: {00000000-0000-0000-0000-000000000001},
                                                 config_params: "vgpu_type_id=595",
                                                 qemu_pid: 2629,
                                                 unknown_414: [0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0],
                                             }
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: Unable to get VM name from QEMU commandline, using VM's UUID as VM name. 0x57
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 00000000-0000-0000-0000-000000000001 GPU PCI id 00:01:00.0 config params vgpu_type_id=595
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=595
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: Successfully updated env symbols!
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x2080014b failed.
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: VgpuConfig {
                                                  vgpu_type: 595,
                                                  vgpu_name: "NVIDIA A10-8Q",
                                                  vgpu_class: "Quadro",
                                                  vgpu_signature: 0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
                                                  features: "Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0",
                                                  max_instances: 3,
                                                  num_heads: 4,
                                                  max_resolution_x: 7680,
                                                  max_resolution_y: 4320,
                                                  max_pixels: 66355200,
                                                  frl_config: 60,
                                                  cuda_enabled: 1,
                                                  ecc_supported: 1,
                                                  mig_instance_size: 0,
                                                  multi_vgpu_supported: 0,
                                                  vdev_id: 0x223614bd,
                                                  pdev_id: 0x2236,
                                                  fb_length: 0x1dc000000,
                                                  mappable_video_size: 0x400000,
                                                  fb_reservation: 0x24000000,
                                                  encoder_capacity: 0x64,
                                                  bar1_length: 0x100,
                                                  frl_enable: 1,
                                                  adapter_name: "NVIDIA A10-8Q",
                                                  adapter_name_unicode: "NVIDIA A10-8Q",
                                                  short_gpu_name_string: "GA102-A",
                                                  licensed_product_name: "NVIDIA RTX Virtual Workstation",
                                                  vgpu_extra_params: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
                                              }
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Applying profile nvidia-595 overrides
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/num_heads: 4 -> 1
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_resolution_x: 7680 -> 1920
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_resolution_y: 4320 -> 1080
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_pixels: 66355200 -> 2073600
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): gpu-pci-id : 0x100
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vgpu_type : Quadro
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Framebuffer: 0x1dc000000
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Virtual Device Id: 0x2236:0x14bd
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: ######## vGPU Manager Information: ########
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: Driver Version: 510.85.03
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU supported range: (0x70001, 0xd0001)
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x20801322 failed.
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: error: vmiop_log: (0x0): Failed to get blacklisted pages:0x56
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Init frame copy engine: syncing...
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU migration enabled
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: display_init inst: 0 successful
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x20801322 failed.
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: error: vmiop_log: (0x0): Failed to get blacklisted pages:0x56
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: ######## Guest NVIDIA Driver Information: ########
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: Driver Version: 512.78
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: vGPU version: 0xd0001
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU license state: Unlicensed (Unrestricted)
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Guest driver unloaded!

opened by alx696 3

The card_name parameter is invalid, the virtual graphics card name will still be displayed

I found that using the card_name parameter is invalid, and using GPU-Z can still display: GRID RTX6000-1B in: WDDM, opengl, and cuda, even if it has been replaced. This ID can be found in the registry

opened by wnark 3
vGPU License
Hi. I have many questions about NVIDIA vGPU Licensing.

Does this still require a nvidia vGPU License ?

I just got a 90-day free license and unlocked my vGPU functionality of my consumer grades GPU cards and I used this vGPUs in my Proxmox VMs. Whats happen when that period (90 days) expires? Please Help me. Thanks.
opened by srkaviani 2
unable to compile --noob issue

Hi

I am working my way through some vgpu tutorial..

proxmox 7.2 fresh install this tutorial: https://wvthoog.nl/proxmox-7-vgpu-v2/

when i run cargo build --release it will error out with the following:

error[E0658]: use of unstable library feature 'assoc_char_funcs': recently added --> src/format.rs:77:21 | 77 | for item in char::decode_utf16(self.0.iter().copied().take_while(|&ch| ch != 0)) { | ^^^^^^^^^^^^^^^^^^ | = note: see issue #71763 https://github.com/rust-lang/rust/issues/71763 for more information

error[E0658]: use of unstable library feature 'assoc_char_consts': recently added --> src/format.rs:78:41 | 78 | f.write_char(item.unwrap_or(char::REPLACEMENT_CHARACTER))?; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = note: see issue #71763 https://github.com/rust-lang/rust/issues/71763 for more information

error: aborting due to 2 previous errors

i have no idea how to get past this..

Halp plz

opened by arcnet01 2
Changing the name of the graphics card does not take effect?
Similar to this, I changed the graphics card to RTX8000, but it still shows RTX6000 on Windows

[profile.nvidia-435] card_name = "GRID RTX8000" vgpu_type = "NVS" pci_id = 0x1e301437 pci_device_id = 0x1e30
opened by wnark 2
Unable to install driver on Debian

Using Proxmox 7.2 I've installed the vGPU unlocker on my host using NVIDIA-Linux-x86_64-510.85.03 drivers and have passed it in to windows with no issues.

I want to do the same on a linux VM, but since I can't do the vGPU spoofing in Linux I went to install the included guest drivers that came with the NVIDIA-Linux-x86_64-510.85.03 driver package.

I'm attempting to install it using dpkg --install , but I keep getting an error saying I haven't installed the linux-headers package. But I've already installed linux-headers-amd64.

Any ideas what I need to do to install the GPU in the guest OS for Linux?

opened by abishur 1
Failed to override nvidia profile
Machine infomations: OS: archlinux Kernel: archlinux 5.12.15-arch1-1 NVIDIA Driver Version: 460.73.01 GPU: RTX 1660 SUPPER

Override profile: ➜ vgpu_unlock-rs cat /etc/vgpu_unlock/profile_override.toml [profile.nvidia-258] num_displays = 1 display_width = 1920 display_height = 1280 max_pixels = 2457600 cuda_enabled = 1

Problem: Failed override nvidia profile. The error message "Parameters size for NVA082_CTRL_CMD_HOST_VGPU_DEVICE_GET_VGPU_TYPE_INFO was 1840 bytes, expected 1848 bytes" is reported.

Logs: 4.1 nvidia-vgpu logs: ➜ vgpu_unlock-rs journalctl -u nvidia-vgpu-mgr.service Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Manager Daemon... Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Manager Daemon. Jan 09 00:01:48 archlinux nvidia-vgpu-mgr[487]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[487]: Nv0000CtrlVgpuGetStartDataParams { mdev_uuid: {2b7bc7e4-8784-4933-8613-9bc92bbc718c}, config_params: "vgpu_type_id=258", qemu_pid: 3091, gpu_pci_id: 0x100, vgpu_id: 2, gpu_pci_bdf: 256, } Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 2b7bc7e4-8784-4933-8613-9bc92bbc718c GPU PCI id 00:01:00.0 config params vgpu_type_id=258 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=258 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: Successfully updated env symbols! Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x20801322 failed. Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x2080014b failed. Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: Parameters size for NVA082_CTRL_CMD_HOST_VGPU_DEVICE_GET_VGPU_TYPE_INFO was 1840 bytes, expected 1848 bytes Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): gpu-pci-id : 0x100 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): vgpu_type : Quadro Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Framebuffer: 0xb0000000 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Virtual Device Id: 0x1e30:0x1327 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): FRL Value: 60 FPS Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: ######## vGPU Manager Information: ######## Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: Driver Version: 460.73.01 Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x2080012f failed. Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Cannot query ECC status. vGPU ECC support will be disabled. Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Init frame copy engine: syncing... Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): vGPU migration disabled Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: display_init inst: 0 successful Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: ######## Guest NVIDIA Driver Information: ######## Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: Driver Version: 462.31 Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: vGPU version: 0x90001 ...... 4.2 nvidia-vgpud logs: ➜ vgpu_unlock-rs journalctl -u nvidia-vgpud
Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Daemon... Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Verbose syslog connection opened Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Started (482) Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Daemon. Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Global settings: Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Size: 16 Version 1 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Homogeneous vGPUs: 1 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: vGPU types: 517 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: pciId of gpu [0]: 0:1:0:0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Physical GPU: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: PciID: 0x0000 / 0x0001 / 0x0000 / 0x0000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Size: 52 Version 1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevID: 0x10de / 0x1e30 / 0x10de / 0x12ba Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported vGPUs count: 23 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x100: max 24 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x100: GRID RTX6000-1Q Class: Quadro Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1325 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x38000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x8000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 17694720 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 5120, height 2880 Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Daemon... Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Verbose syslog connection opened Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Started (482) Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Daemon. Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Global settings: Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Size: 16 Version 1 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Homogeneous vGPUs: 1 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: vGPU types: 517 Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: pciId of gpu [0]: 0:1:0:0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Physical GPU: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: PciID: 0x0000 / 0x0001 / 0x0000 / 0x0000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Size: 52 Version 1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevID: 0x10de / 0x1e30 / 0x10de / 0x12ba Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported vGPUs count: 23 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x100: max 24 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x100: GRID RTX6000-1Q Class: Quadro Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1325 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x38000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x8000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 17694720 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 5120, height 2880 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: License: Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x101: max 12 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x101: GRID RTX6000-2Q Class: Quadro Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1326 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x74000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0xc000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 35389440 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 7680, height 4320 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: License: Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x102: max 8 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x102: GRID RTX6000-3Q Class: Quadro Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1327 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0xb0000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x10000000 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 35389440 Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 7680, height 4320 ......
opened by oushijie812 0

Unable to override with two mdev profiles within one VM

OS: Proxmox 7.3 Kernel: 5.15.74-1-pve Host Driver: 525.60.12 Guest Driver: 525.60.13 Hardware: RTX 2060 12G * 2

Host Config:

[mdev.00000000-0000-0000-0000-000000000102]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0x162000000

[mdev.00000001-0000-0000-0000-000000000102]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0x162000000

VM Config

hostpci0: 0000:82:00.0,mdev=nvidia-259,pcie=1
hostpci1: 0000:01:00.0,mdev=nvidia-259,pcie=1

PVE vm Start

kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: warning: vfio 00000000-0000-0000-0000-000000000102: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000001-0000-0000-0000-000000000102,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0: warning: vfio 00000001-0000-0000-0000-000000000102: Could not enable error recovery for the device
TASK OK

Host journalctl nvidia-vgpu-mgr.service pastebin

Problem: When two vGPUs are added to a virtual machine, the mdev override does not work properly, but if override with a profile it works fine.

opened by nyacat 1

how to recompile nvidia-vgpu-vfio with NV_KVM_MIGRATION_UAPI ?

Hi, I'm currently working to add vgpu live migration on proxmox.

I have seen in your doc that nvidia-vgpu-vfio need to be recompiled with NV_KVM_MIGRATION_UAPI=1.

Can you tell me how we can send this variable when building dmks driver ?

opened by aderumier 15