vGPU NVIDIA TITAN Xp for Windows Clients

[Mike85]

Hello,

currently I am trying to integrate two nvida titanium xp gpus on proxmox.
But I am running in to the following issue by addin the GPU incl. profile to a VM:


Error by adding the GPU to a second System :

kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000100,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000100: error getting device from group 91: Input/output error
Verify all devices in group 91 are bound to vfio-<bus> or pci-stub and not already in use
TASK ERROR: start failed: QEMU exited with code 1

Output of dmesg:

dmesg |grep vgpu
[    9.153501] nvidia-vgpu-vfio 00000000-0000-0000-0000-000000000100: Adding to iommu group 91
[    9.153505] nvidia-vgpu-vfio 00000000-0000-0000-0000-000000000100: MDEV: group_id = 91

What I did:

- Installed the pve binary-amd64 for current version
- Installed v15.0 nVidia vGPU Driver


nano /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
- OR -
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Save file and close

update-grub

-Load VFIO modules at boot-

nano /etc/modules

echo 'vfio' >> /etc/modules
echo 'vfio_iommu_type1' >> /etc/modules
echo 'vfio_pci' >> /etc/modules
echo 'vfio_virqfd' >> /etc/modules

Save file and close

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

update-initramfs -u



chmod -R +x vgpu_unlock
chmod +x NVIDIA------.run

./NVIDIA------.run --dkms

nano /usr/src/nvidia-510.85.03/nvidia/os-interface.c
#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"

nano /usr/src/nvidia-450.80/nvidia/nvidia.Kbuild
ldflags-y += -T /root/vgpu_unlock/kern.ld

cd vgpu_unlock-rs
cargo build --release

mkdir /etc/systemd/system/nvidia-vgpud.service.d
mkdir /etc/systemd/system/nvidia-vgpu-mgr.service.d

nano /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf

[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so

nano /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf

[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so

- Created a custom profile for nvidia-48

[profile.nvidia-48]
num_displays = 1
display_width = 2560
display_height = 1440
max_pixels = 3686400
cuda_enabled = 1
frl_enabled = 60
framebuffer = 3937053354
pci_id = 0x1B3011A0
pci_device_id = 0x1B30

nano /etc/pve/qemu/[VM#].conf

args: -uuid 00000000-0000-0000-0000-000000000###

Added the GPU and to the vm



Any idea how this Issue can be solved?