Dell R740 with Quadro 4000 forward to vm IOMMU/VGPU

T

To0wnn

New Member
May 31, 2022
7
0
1
Hi,

Many post on PCIE forward but i still get the code 43 on my vm windows10
I have a dell R740 with GF100GL [Quadro 4000] and the follwing is on in the bios:
Virtualization Technology I/OAT DMA Engine SR-IOV Global Enable

nvidia-error43-fixer_113 is working, but not working. Fixes the error but comes back again after reboot.
So no idea how to fix it. I did see some difficult kernel patches with an older kernel as well and such but is this nessesary?
Tried also nvflash a new vbios but no luck.

lspci -nnk | grep -i vga -C 3:
lspci -nnk | grep -i vga -C 3 Kernel driver in use: tg3 Kernel modules: tg3 02:00.0 PCI bridge [0604]: PLDA PCI Express Bridge [1556:be00] (rev 02) 03:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04) DeviceName: Embedded Video Subsystem: Dell Integrated Matrox G200eW3 Graphics Controller [1028:0715] Kernel modules: mgag200 -- Kernel driver in use: skx_uncore 3a:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04) Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000] 3b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3) Subsystem: Hewlett-Packard Company GF100GL [Quadro 4000] [103c:0780] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau

vfio.conf:
options vfio-pci ids=10de:06dd,10de:0be5 disable_vga=1

VM conf:
agent: 1 boot: order=sata0;ide2 cores: 24 cpu: host hostpci0: 3b:00,pcie=1,romfile=vbios.bin ide2: none,media=cdrom machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=6.2.0,ctime=1656349411 name: W10nonefi net0: virtio=22:27:30:7F:84:11,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 sata0: local-zfs:vm-107-disk-0,size=32G scsihw: virtio-scsi-pci smbios1: uuid=b2e7e2a4-01d1-4381-8ff7-1d8c2ec11739 sockets: 1 vmgenid: c3f7d198-ce5c-4db9-beb8-e02329480488

blacklist.conf:
blacklist radeon blacklist nouveau blacklist nvidia blacklist nvidiafb

iommu_unsafe_interrupts.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1

kvm.conf:
options kvm ignore_msrs=1

cat /proc/cmdline:
initrd=\EFI\proxmox\5.15.35-3-pve\initrd.img-5.15.35-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction nofb nomodeset video=efifb:off,vesafb:off

dmesg | grep -e DMAR -e IOMMU:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.011345] ACPI: DMAR 0x000000006FC0E000 0001B8 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.011400] ACPI: Reserving DMAR table memory at [mem 0x6fc0e000-0x6fc0e1b7] [ 0.542693] DMAR: IOMMU enabled [ 1.260975] DMAR: Host address width 46 [ 1.260978] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0 [ 1.260984] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.260990] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0 [ 1.260995] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.260999] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 [ 1.261004] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261008] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 1.261012] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261016] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0 [ 1.261020] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261024] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0 [ 1.261028] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261031] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0 [ 1.261042] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261046] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1 [ 1.261050] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 1.261053] DMAR: RMRR base: 0x0000006f55f000 end: 0x0000006f561fff [ 1.261056] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000006f55f000-0x000000006f561fff], contact BIOS vendor for fixes [ 1.261061] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x000000006f55f000-0x000000006f561fff] [ 1.261066] DMAR: ATSR flags: 0x0 [ 1.261070] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6 [ 1.261073] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5 [ 1.261076] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4 [ 1.261079] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3 [ 1.261081] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2 [ 1.261084] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1 [ 1.261086] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0 [ 1.261088] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7 [ 1.261091] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7 [ 1.261093] DMAR-IR: HPET id 0 under DRHD base 0x9d7fc000 [ 1.261096] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 1.263327] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 2.178963] DMAR: No SATC found [ 2.178967] DMAR: dmar6: Using Queued invalidation [ 2.178975] DMAR: dmar5: Using Queued invalidation [ 2.178988] DMAR: dmar4: Using Queued invalidation [ 2.178995] DMAR: dmar0: Using Queued invalidation [ 2.179001] DMAR: dmar7: Using Queued invalidation [ 2.189065] DMAR: Intel(R) Virtualization Technology for Directed I/O
 
Last edited:
I'm using the same video card as you

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3) Subsystem: Hewlett-Packard Company GF100GL [Quadro 4000] [103c:0780] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau

My host conf is slightly different
hostpci0: 0000:03:00,pcie=1,x-vga=1,romfile=175529.rom

I got the rom file from here
https://www.techpowerup.com/vgabios/

remember the rom file needs to be saved here
/usr/share/kvm/
 
Your rom is different i can also not flash it to my gpu this is my rom that matches 244098.rom
What is your cmdline?

if i start trough ssh the vm i get teh following
qm start 107 kvm: -device vfio-pci,host=0000:3b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/usr/share/kvm/244098.rom: vfio 0000:3b:00.0: failed getting region info for VGA region index 8: Invalid argument device does not support requested feature x-vga start failed: QEMU exited with code 1
 
Last edited:
/proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.35-1-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off
pretty much the same as yours

you can try the rom file, it's only loaded in the VM so it won't make any changes to your hardware
 
You must log in or register to reply here.
Top Bottom