[SOLVED] Request for help GVT-g and mdev

[DerekG]

HI all,

I've been running my homelab on 2 Proxmox nodes for about 6 months now, I consider Proxmox the best thing since sliced bread, but I have an issue with GVT-g which I can't overcome despite 2 solid days of internet research.

The node is an Intel i7-8750 mini PC so only iGPU (no slot for graphics card), booting Proxmox 7 on zfs RAID1. with a few VM's Win10, Ubuntu and MacOS.

My aim is to passthrough GVT-g mdev to some of the key VM's. I've configured the passthrough according to https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_general_requirements

The IOMMU groups are broken out and the mdev_supported_types looks correct, but when I try to add the PCI vGPU from the web interface it shows 0 devices available.
If I pass the device from terminal, the VM fails to boot with Error: pci device '0000:00:02.0' has no available instances of 'i915-GVTg_V5_8'

I did get the vGPU passed through to a Win10 VM, it loaded the Intel driver and I could test the vGPU working with Handbrake transcoding, but after the next reboot I lost that vGPU and have not seen any available for use since then.

Note: I do not require display out via HDMI as all the VM's are remote accessed.

Spec:
CPU(s) 12 x Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz (1 Socket)
Kernel Version Linux 5.11.22-4-pve #1 SMP PVE 5.11.22-8 (Fri, 27 Aug 2021 11:51:34 +0200)
PVE Manager Version pve-manager/7.0-11/63d82f4e

:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.11.22-4-pve\initrd.img-5.11.22-4-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on i915.enable_gvt=1 kvm.ignore_msrs=1

~# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630]
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1b.5 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation HM470 Chipset LPC/eSPI Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
01:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. A2000 NVMe SSD (rev 03)
02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
03:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter
04:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. A2000 NVMe SSD (rev 03)


:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
# Add for GVT-g Mediated passthrough
kvmgt
exngt
vfio-mdev
# Generated by sensors-detect on Wed Apr 28 23:44:17 2021
# Chip drivers
# coretemp
# nct6775

:~# dmesg | grep -i vfio
[ 3.768806] VFIO - User Level meta-driver version: 0.3
[ 5.779846] vfio_pci: add [8086:3e9b[ffffffff:ffffffff]] class 0x000000/00000000
[ 7.721089] vfio_mdev 337634dd-f18a-49c0-99fa-b9081d01d669: Adding to iommu group 14
[ 7.721094] vfio_mdev 337634dd-f18a-49c0-99fa-b9081d01d669: MDEV: group_id = 14
[ 7.733841] vfio_mdev 71d43450-5b6a-460f-bb32-3b34935ff12d: Adding to iommu group 15
[ 7.733847] vfio_mdev 71d43450-5b6a-460f-bb32-3b34935ff12d: MDEV: group_id = 15

:~# ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types
i915-GVTg_V5_4 i915-GVTg_V5_8

Log on VM Start:
Sep 06 22:17:49 pve-1 pvedaemon[123303]: <root@pam> starting task UPIDve-1:000349C0:0007E1C7:6136857D:qmstart:103:root@pam:
Sep 06 22:17:49 pve-1 pvedaemon[215488]: start VM 103: UPIDve-1:000349C0:0007E1C7:6136857D:qmstart:103:root@pam:
Sep 06 22:17:49 pve-1 pvedaemon[215488]: pci device '0000:00:02.0' has no available instances of 'i915-GVTg_V5_8'
Sep 06 22:17:49 pve-1 pvedaemon[123303]: <root@pam> end task UPIDve-1:000349C0:0007E1C7:6136857D:qmstart:103:root@pam: pci device '0000:00:02.0' has no available instances of 'i915-GVTg_V5_8'

Can anyone advise on this issue, as the only remotely relevant post I've found was with an Nvidia GPU with licensing issues.

Note, there are some confusing instructions in the manual, if they could be cleared up it might help:

"For some platforms, it may be necessary to allow unsafe interrupts. For this add the following line in a file ending with ‘.conf’ file in /etc/modprobe.d/:
options vfio_iommu_type1 allow_unsafe_interrupts=1" ?? What's the filename? existing or new file??

"Host Configuration
In this case, the host must not use the card. There are two methods to achieve this:
pass the device IDs to the options of the vfio-pci modules by adding
options vfio-pci ids=1234:5678,4321:8765
? to a .conf file in /etc/modprobe.d/ where 1234:5678 and 4321:8765 are the vendor and device IDs obtained by: ?? Again add to which file exactly?


Thanks in advance for any assistance.
DerekG

 

[dcsapak]

can you post your vm config?
why did you create the instances manually? pve already takes care of this.
it seems that you did create 2 instances, and now when pve wants to start the vm it cannot create new ones

to remove an instance do

echo 1 > /sys/bus/pci/devices/0000:00:02.0/UUID/remove

 

[DerekG]

HI Dominik,

You're a life saver!!

After removing those instances, I was able to see available vGPU's in Proxmox web interface.

However, after reboot, the old instances returned and I realised that I'd made them persistent during my 1st attempt about 4 months ago. I did not realise that Proxmox would auto create on VM start-up.
Removed the service creating them and now all is good, I've enabled vGPU on a Win10 and Ubuntu VM without issue.

Lessons learnt during this time, ignore web articles more than a year old, and read carefully as not all Ubuntu/KVM solutions are applicable to Proxmox VE.

Thank you for your assistance and best regards

DerekG