Dell R740 with Quadro 4000 forward to vm IOMMU/VGPU

[To0wnn]

Hi,

Many post on PCIE forward but i still get the code 43 on my vm windows10
I have a dell R740 with GF100GL [Quadro 4000] and the follwing is on in the bios:
Virtualization Technology I/OAT DMA Engine SR-IOV Global Enable

nvidia-error43-fixer_113 is working, but not working. Fixes the error but comes back again after reboot.
So no idea how to fix it. I did see some difficult kernel patches with an older kernel as well and such but is this nessesary?
Tried also nvflash a new vbios but no luck.

lspci -nnk | grep -i vga -C 3:
lspci -nnk | grep -i vga -C 3 Kernel driver in use: tg3 Kernel modules: tg3 02:00.0 PCI bridge [0604]: PLDA PCI Express Bridge [1556:be00] (rev 02) 03:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04) DeviceName: Embedded Video Subsystem: Dell Integrated Matrox G200eW3 Graphics Controller [1028:0715] Kernel modules: mgag200 -- Kernel driver in use: skx_uncore 3a:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04) Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000] 3b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3) Subsystem: Hewlett-Packard Company GF100GL [Quadro 4000] [103c:0780] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau

vfio.conf:
options vfio-pci ids=10de:06dd,10de:0be5 disable_vga=1

VM conf:
agent: 1 boot: order=sata0;ide2 cores: 24 cpu: host hostpci0: 3b:00,pcie=1,romfile=vbios.bin ide2: none,media=cdrom machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=6.2.0,ctime=1656349411 name: W10nonefi net0: virtio=22:27:30:7F:84:11,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 sata0: local-zfs:vm-107-disk-0,size=32G scsihw: virtio-scsi-pci smbios1: uuid=b2e7e2a4-01d1-4381-8ff7-1d8c2ec11739 sockets: 1 vmgenid: c3f7d198-ce5c-4db9-beb8-e02329480488

blacklist.conf:
blacklist radeon blacklist nouveau blacklist nvidia blacklist nvidiafb

iommu_unsafe_interrupts.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1

kvm.conf:
options kvm ignore_msrs=1

cat /proc/cmdline:
initrd=\EFI\proxmox\5.15.35-3-pve\initrd.img-5.15.35-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction nofb nomodeset video=efifb:off,vesafb:off

dmesg | grep -e DMAR -e IOMMU:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.011345] ACPI: DMAR 0x000000006FC0E000 0001B8 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.011400] ACPI: Reserving DMAR table memory at [mem 0x6fc0e000-0x6fc0e1b7] [ 0.542693] DMAR: IOMMU enabled [ 1.260975] DMAR: Host address width 46 [ 1.260978] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0 [ 1.260984] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.260990] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0 [ 1.260995] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.260999] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 [ 1.261004] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261008] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 1.261012] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261016] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0 [ 1.261020] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261024] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0 [ 1.261028] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261031] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0 [ 1.261042] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261046] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1 [ 1.261050] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261053] DMAR: RMRR base: 0x0000006f55f000 end: 0x0000006f561fff [ 1.261056] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000006f55f000-0x000000006f561fff], contact BIOS vendor for fixes [ 1.261061] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x000000006f55f000-0x000000006f561fff] [ 1.261066] DMAR: ATSR flags: 0x0 [ 1.261070] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6 [ 1.261073] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5 [ 1.261076] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4 [ 1.261079] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3 [ 1.261081] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2 [ 1.261084] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1 [ 1.261086] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0 [ 1.261088] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7 [ 1.261091] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7 [ 1.261093] DMAR-IR: HPET id 0 under DRHD base 0x9d7fc000 [ 1.261096] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 1.263327] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 2.178963] DMAR: No SATC found [ 2.178967] DMAR: dmar6: Using Queued invalidation [ 2.178975] DMAR: dmar5: Using Queued invalidation [ 2.178988] DMAR: dmar4: Using Queued invalidation [ 2.178995] DMAR: dmar0: Using Queued invalidation [ 2.179001] DMAR: dmar7: Using Queued invalidation [ 2.189065] DMAR: Intel(R) Virtualization Technology for Directed I/O

 

[bobmc]

I'm using the same video card as you

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3) Subsystem: Hewlett-Packard Company GF100GL [Quadro 4000] [103c:0780] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau

My host conf is slightly different
hostpci0: 0000:03:00,pcie=1,x-vga=1,romfile=175529.rom

I got the rom file from here
https://www.techpowerup.com/vgabios/

remember the rom file needs to be saved here
/usr/share/kvm/

 

[To0wnn]

Your rom is different i can also not flash it to my gpu this is my rom that matches 244098.rom
What is your cmdline?

if i start trough ssh the vm i get teh following
qm start 107 kvm: -device vfio-pci,host=0000:3b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/usr/share/kvm/244098.rom: vfio 0000:3b:00.0: failed getting region info for VGA region index 8: Invalid argument device does not support requested feature x-vga start failed: QEMU exited with code 1

 

[bobmc]

/proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.35-1-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off
pretty much the same as yours

you can try the rom file, it's only loaded in the VM so it won't make any changes to your hardware