[SOLVED] Problem with GPU Passthrough

[dust_bro]

Just had this affect my system. After a couple days of pulling my hair out, I found this thread. The remove/rescan script worked for me.

 

[trevmlt]

I am at a loss on this one. HELP!

I have 2 GPU in the system, an AMD RX580 that passes through no problem to Win 10 and a second Nvidia 1660Ti that I can not get to work in a separate Win 11 VM. I use proxmox-boot-tool and have tried pinning kernel 5.15.39-4-pve in addition to using the 5.18.19 edge kernel - same result.

The REALLY weird part is I cannot even RDP into the VM (114) as it does not present any IP to my network after the drivers are installed - I have connection right up until the Nvidia USB drivers install - then it goes out. Here are my settings. Let me know if there is anything else I can supply - this SHOULD be working...

cat /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs video=vesafbff video=efifbff video=simplefbff initcall_blacklist=sysfb_init quiet intel_iommu=on iommu=pt


cat /proc/cmdline

initrd=\EFI\proxmox\5.15.39-4-pve\initrd.img-5.15.39-4-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs video=vesafbff video=efifbff video=simplefbff initcall_blacklist=sysfb_init quiet intel_iommu=on iommu=pt


cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:67df,1002:aaf0,10de:2182,10de:1aeb,10de:1aec,10de:1aed disable_vga=1


/etc/modprobe.d/blacklist.conf

blacklist amdgpu
blacklist nouveau
blacklist radeon
blacklist nvidia
blacklist i2c_nvidia_gpu
blacklist nvidiafb

cat /etc/modules

coretemp
nct6775
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net


/etc/pve/qemu-server/114.conf

agent: 1
bios: ovmf
boot: order=scsi0;ide2
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: vm-disks1:vm-114-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 03:00,pcie=1,x-vga=1
ide0: none,media=cdrom
ide2: none,media=cdrom
machine: pc-q35-7.0
memory: 4096
meta: creation-qemu=7.0.0,ctime=1663733622
name: Win11-1660
net0: virtio=56:60:BC:EF:C5:16,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: vm-disks1:vm-114-disk-1,cache=unsafe,discard=on,iothread=1,size=128G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=ada83453-f7a1-40fd-aac0-4f8fde22c2fa
sockets: 1
tpmstate0: vm-disks1:vm-114-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 7d5f2532-1049-4fdd-a1f8-1e7a99c82458


grep vfio /var/log/syslog (since last reboot and VM start)

Sep 21 17:19:03 pve kernel: [ 231.775896] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+memwns=none
Sep 21 17:19:03 pve kernel: [ 231.855280] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=nonewns=none
Sep 21 17:20:44 pve kernel: [ 333.263601] vfio-pci 0000:81:00.0: enabling device (0000 -> 0003)
Sep 21 17:20:44 pve kernel: [ 333.263950] vfio-pci 0000:81:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Sep 21 17:20:44 pve kernel: [ 333.263960] vfio-pci 0000:81:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Sep 21 17:20:44 pve kernel: [ 333.263966] vfio-pci 0000:81:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370
Sep 21 17:20:44 pve kernel: [ 333.282623] vfio-pci 0000:81:00.1: enabling device (0000 -> 0002)
Sep 21 17:20:50 pve kernel: [ 338.798159] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
Sep 21 17:20:50 pve kernel: [ 338.902762] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Sep 21 17:20:50 pve kernel: [ 338.902781] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Sep 21 17:27:35 pve kernel: [ 744.054038] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Sep 21 17:27:35 pve kernel: [ 744.054058] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Sep 21 17:30:40 pve kernel: [ 928.926154] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Sep 21 17:30:40 pve kernel: [ 928.926174] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900

lspci -nnk (problem GPU)

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd TU116 [GeForce GTX 1660 Ti] [1458:3fbe]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
03:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd TU116 High Definition Audio Controller [1458:3fbe]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd TU116 USB 3.1 Host Controller [1458:3fbe]
Kernel driver in use: vfio-pci
Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd TU116 USB Type-C UCSI Controller [1458:3fbe]
Kernel driver in use: vfio-pci
Kernel modules: i2c_nvidia_gpu

(good GPU)
81:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
Subsystem: Sapphire Technology Limited Radeon RX 570 Pulse 4GB [1da2:e353]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
81:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
Subsystem: Sapphire Technology Limited Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1da2:aaf0]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

dmesg | grep -e DMAR -e IOMMU

[ 0.015740] ACPI: DMAR 0x000000007DDEA580 000140 (v01 A M I OEMDMAR 00000001 INTL 00000001)
[ 0.015763] ACPI: Reserving DMAR table memory at [mem 0x7ddea580-0x7ddea6bf]
[ 0.197897] DMAR: IOMMU enabled
[ 0.470748] DMAR: Host address width 46
[ 0.470750] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[ 0.470757] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020df
[ 0.470761] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
[ 0.470765] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[ 0.470768] DMAR: RMRR base: 0x0000007f231000 end: 0x0000007f23efff
[ 0.470771] DMAR: ATSR flags: 0x0
[ 0.470772] DMAR: RHSA base: 0x000000fbffe000 proximity domain: 0x1
[ 0.470774] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
[ 0.470778] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffe000 IOMMU 0
[ 0.470780] DMAR-IR: IOAPIC id 0 under DRHD base 0xdfffc000 IOMMU 1
[ 0.470782] DMAR-IR: IOAPIC id 2 under DRHD base 0xdfffc000 IOMMU 1
[ 0.470783] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
[ 0.470785] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.471562] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.091764] DMAR: [Firmware Bug]: RMRR entry for device 03:00.2 is broken - applying workaround
[ 1.091802] DMAR: No SATC found
[ 1.091807] DMAR: dmar0: Using Queued invalidation
[ 1.091813] DMAR: dmar1: Using Queued invalidation
[ 1.096685] DMAR: Intel(R) Virtualization Technology for Directed I/O


cat /proc/iomem

00000000-00000fff : Reserved
00001000-0009ffff : System RAM
00000000-00000000 : PCI Bus 0000:00
000a0000-000dffff : PCI Bus 0000:00
000c0000-000c7fff : Video ROM
000f0000-000fffff : System ROM
00100000-733c4017 : System RAM
733c4018-733e0e57 : System RAM
733e0e58-733e1017 : System RAM
733e1018-733f1857 : System RAM
733f1858-733f2017 : System RAM
733f2018-73402857 : System RAM
73402858-73403017 : System RAM
73403018-7340b457 : System RAM
7340b458-7340c017 : System RAM
7340c018-7341c857 : System RAM
7341c858-7341d017 : System RAM
7341d018-7342d857 : System RAM
7342d858-7342e017 : System RAM
7342e018-7344d857 : System RAM
7344d858-7dca2fff : System RAM
7dca3000-7dcd2fff : Reserved
7dcd3000-7ddeafff : ACPI Tables
7ddeb000-7e012fff : ACPI Non-volatile Storage
7e013000-7f350fff : Reserved
7f351000-7f351fff : System RAM
7f352000-7f3d7fff : ACPI Non-volatile Storage
7f3d8000-7f7fffff : System RAM
7f800000-7fffffff : RAM buffer
80000000-dfffffff : PCI Bus 0000:00
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
80000000-8fffffff : Reserved
c0000000-d20fffff : PCI Bus 0000:03
c0000000-cfffffff : 0000:03:00.0
c0000000-cfffffff : vfio-pci
d0000000-d1ffffff : 0000:03:00.0
d0000000-d1ffffff : vfio-pci
d2000000-d203ffff : 0000:03:00.2
d2000000-d203ffff : vfio-pci
d2040000-d204ffff : 0000:03:00.2
d2040000-d204ffff : vfio-pci
dc000000-dd0fffff : PCI Bus 0000:08
dc000000-dd0fffff : PCI Bus 0000:09
dc000000-dcffffff : 0000:09:00.0
dd000000-dd01ffff : 0000:09:00.0
dd100000-dd103fff : 0000:00:04.7
dd100000-dd103fff : ioatdma
dd104000-dd107fff : 0000:00:04.6
dd104000-dd107fff : ioatdma
dd108000-dd10bfff : 0000:00:04.5
dd108000-dd10bfff : ioatdma
dd10c000-dd10ffff : 0000:00:04.4
dd10c000-dd10ffff : ioatdma
dd110000-dd113fff : 0000:00:04.3
dd110000-dd113fff : ioatdma
dd114000-dd117fff : 0000:00:04.2
dd114000-dd117fff : ioatdma
dd118000-dd11bfff : 0000:00:04.1
dd118000-dd11bfff : ioatdma
dd11c000-dd11ffff : 0000:00:04.0
dd11c000-dd11ffff : ioatdma
dd120000-dd1200ff : 0000:00:1f.3
dd121000-dd1217ff : 0000:00:1f.2
dd121000-dd1217ff : ahci
dd122000-dd1223ff : 0000:00:1d.0
dd122000-dd1223ff : ehci_hcd
dd123000-dd1233ff : 0000:00:1a.0
dd123000-dd1233ff : ehci_hcd
dd125000-dd12500f : 0000:00:16.1
dd126000-dd12600f : 0000:00:16.0
dd127000-dd127fff : 0000:00:05.4
de000000-df0fffff : PCI Bus 0000:03
de000000-deffffff : 0000:03:00.0
de000000-deffffff : vfio-pci
df000000-df07ffff : 0000:03:00.0
df080000-df083fff : 0000:03:00.1
df080000-df083fff : vfio-pci
df084000-df084fff : 0000:03:00.3
df084000-df084fff : vfio-pci
df200000-df3fffff : PCI Bus 0000:0c
df200000-df2fffff : 0000:0c:00.0
df200000-df2fffff : e1000e
df300000-df33ffff : 0000:0c:00.0
df340000-df35ffff : 0000:0c:00.0
df340000-df35ffff : e1000e
df360000-df363fff : 0000:0c:00.0
df360000-df363fff : e1000e
df400000-df5fffff : PCI Bus 0000:0b
df400000-df4fffff : 0000:0b:00.0
df400000-df4fffff : e1000e
df500000-df53ffff : 0000:0b:00.0
df540000-df55ffff : 0000:0b:00.0
df540000-df55ffff : e1000e
df560000-df563fff : 0000:0b:00.0
df560000-df563fff : e1000e
df600000-df7fffff : PCI Bus 0000:07
df600000-df6fffff : 0000:07:00.0
df600000-df6fffff : e1000e
df700000-df73ffff : 0000:07:00.0
df740000-df75ffff : 0000:07:00.0
df740000-df75ffff : e1000e
df760000-df763fff : 0000:07:00.0
df760000-df763fff : e1000e
df800000-df9fffff : PCI Bus 0000:06
df800000-df8fffff : 0000:06:00.0
df800000-df8fffff : e1000e
df900000-df93ffff : 0000:06:00.0
df940000-df95ffff : 0000:06:00.0
df940000-df95ffff : e1000e
df960000-df963fff : 0000:06:00.0
df960000-df963fff : e1000e
dfa00000-dfafffff : PCI Bus 0000:0a
dfa00000-dfa0ffff : 0000:0a:00.0
dfa10000-dfa107ff : 0000:0a:00.0
dfa10000-dfa107ff : ahci
dfb00000-dfbfffff : PCI Bus 0000:02
dfb00000-dfb7ffff : 0000:02:00.0
dfb80000-dfb81fff : 0000:02:00.0
dfb80000-dfb81fff : ahci
dfb82000-dfb83fff : 0000:02:00.0
dfb82000-dfb83fff : ahci
dfffc000-dfffcfff : dmar1
e0000000-fbffffff : PCI Bus 0000:80
e0000000-f01fffff : PCI Bus 0000:81
e0000000-efffffff : 0000:81:00.0
e0000000-efffffff : vfio-pci
f0000000-f01fffff : 0000:81:00.0
f0000000-f01fffff : vfio-pci
fbe00000-fbefffff : PCI Bus 0000:81
fbe00000-fbe3ffff : 0000:81:00.0
fbe00000-fbe3ffff : vfio-pci
fbe40000-fbe5ffff : 0000:81:00.0
fbe60000-fbe63fff : 0000:81:00.1
fbe60000-fbe63fff : vfio-pci
fbf00000-fbf03fff : 0000:80:04.7
fbf00000-fbf03fff : ioatdma
fbf04000-fbf07fff : 0000:80:04.6
fbf04000-fbf07fff : ioatdma
fbf08000-fbf0bfff : 0000:80:04.5
fbf08000-fbf0bfff : ioatdma
fbf0c000-fbf0ffff : 0000:80:04.4
fbf0c000-fbf0ffff : ioatdma
fbf10000-fbf13fff : 0000:80:04.3
fbf10000-fbf13fff : ioatdma
fbf14000-fbf17fff : 0000:80:04.2
fbf14000-fbf17fff : ioatdma
fbf18000-fbf1bfff : 0000:80:04.1
fbf18000-fbf1bfff : ioatdma
fbf1c000-fbf1ffff : 0000:80:04.0
fbf1c000-fbf1ffff : ioatdma
fbf20000-fbf20fff : 0000:80:05.4
fbffe000-fbffefff : dmar0
fc000000-fcffffff : pnp 00:00
fd000000-fdffffff : pnp 00:00
fe000000-feafffff : pnp 00:00
feb00000-febfffff : pnp 00:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec40000-fec403ff : IOAPIC 2
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed08000-fed08fff : pnp 00:07
fed1c000-fed3ffff : Reserved
fed1c000-fed1ffff : pnp 00:07
fed1f410-fed1f414 : iTCO_wdt.1.auto
fed45000-fedfffff : pnp 00:00
fee00000-fee00fff : Local APIC
ff000000-ffffffff : Reserved
ff000000-ffffffff : pnp 00:07
100000000-107fffffff : System RAM
e72600000-e736025c7 : Kernel code
e73800000-e741f4fff : Kernel rodata
e74200000-e7463da7f : Kernel data
e74982000-e74ffffff : Kernel bss

 

[trevmlt]

I have made a little progress. I know the passthrough is working as I have made it to the point where I can see the 1660Ti in Windows, but no matter what I do, the Error 43 will not go away.

Update 2: I have a fully passed through 1660ti in ubuntu 22.04, but there is nothing I can do to make this work in Windows. The amdgpu was a snap - so frustrating. Final attempt with cmdline as follows - still either leaves me with an unbootable win vm at worse or an error 43 at best:

root=ZFS=rpool/ROOT/pve-1 boot=zfs pcie_aspm=off quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction video=vesafbff video=efifbff video=simplefbff initcall_blacklist=sysfb_init

 

[das1996]

Using the suggestions from this post, this is my working configuration:

MB: Asus ROG Strix X570-F Gaming
BIOS F4403 (AGESA V2 PI 1.2.0.7): enable IOMMU, SVM; disable CSM
CPU: Ryzen 5950x
GPU: GTX 1060

Proxmox 7.2-7
5.15.39-1-pve

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet initcall_blacklist=sysfb_init"

/etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c03,10de:10f1 disable_vga=1

/etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0

/etc/modprobe.d/snd-hda-intel.conf
options snd-hda-intel enable_msi=1

I was able to passthrough the GTX 1060, onboard audio and USB controllers (include ARGB) to a W10 VM.
VM: ovmf, q35, virtio drivers, (remove cdrom as it sometimes makes audio crack)

Thanks.

Thanks for the clear instructions. Testing using a gt-710 card, b550 board, 5700g proc. GPU passed through successfully with no error codes in device mangler.

I'm a bit puzzled by the last line, the snd-hda-intel.conf. Given you're using an amd platform, how is that applicable? thanks!

 

[leesteken]

I'm a bit puzzled by the last line, the snd-hda-intel.conf. Given you're using an amd platform, how is that applicable? thanks!

HDA is an Intel standard and does not require Intel hardware. The audio device on a Ryzen platform usually uses the snd_hda_intel driver as you can check with lspci -k.

 

[c3n21]

What a nice post! I've been looking for a solution to this BAR 3 issues all night. Big Thanks!

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifb:off" or "video=simplefb:off" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

I signed up to the forum just to thank you and confirm that this works for me too.
I'm running Proxmox VE 7.2-11 on my desktop with an i7-6700 and GTX 1060 6GB and this kernel parameter seems to do the trick.

 

[fateinabox]

What a nice post! I've been looking for a solution to this BAR 3 issues all night. Big Thanks!

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifb:off" or "video=simplefb:off" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

Bruh, error 43 has been plaguing me for HOURS. This literally was it. Thanks

 

[shutch]

Hi all,
I am having difficulties with passing through a GTX1070 to Windows 11. The VM recognizes it, but getting Code 43 in device manager of the VM.

The host is an Intel i7-3770 running on an Asus P8Z68-M PRO.

Windows has stopped this device because it has reported problems. (Code 43)

pveversion -v

proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3

/etc/pve/qemu-server/103.conf

bios: ovmf
boot: order=scsi0;net0;ide0
cores: 6
cpu: host
efidisk0: free-nas-test:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1
ide0: local:iso/virtio-win.iso,media=cdrom,size=519590K
machine: pc-q35-6.2
memory: 28128
meta: creation-qemu=6.2.0,ctime=1668891555
name: win11-gpu
net0: virtio=22:B4:49:C3:82:09,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: free-nas-test:vm-103-disk-1,size=128G
scsihw: virtio-scsi-single
smbios1: uuid=f1ea3bf5-f070-41b7-a780-1948e05bfd5c
sockets: 1
tpmstate0: free-nas-test:vm-103-disk-2,size=4M,version=v2.0
vga: none
vmgenid: c10b612a-cfba-4d4c-87f1-3b54c7f5bcab

/etc/default/grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction video=vesafb:off,efifb:off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1

/etc/modprove.d/kvm.conf

options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1

/etc/modprobe.d/blacklist.conf

options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1

Any help would be appreciated.

 

[William C]

What a nice post! I've been looking for a solution to this BAR 3 issues all night. Big Thanks!

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifb:off" or "video=simplefb:off" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

I. Just want to second this comment .

ive spent many hours and days trying to get working and this initcall_blacklist=sysfb_init stops the frame buffering nicking the memory. I also added graphics card without selecting pcie and all functions

 

[grootlab]

Thanks for your kind notes! I encountered a similar issue in Proxmox 7.2.5. Replacing video=efifb:off with initcall_blacklist=sysfb_init in /etc/default/grub indeed resolves the issue.

initcall_blacklist=sysfb_init in /etc/default/grub
This worked for me too.

 

[William C]

Bruh, error 43 has been plaguing me for HOURS. This literally was it. Thanks

initcall_blacklist=sysfb_init in /etc/default/grub
This worked for me too.

@grootlab Pls could you let me know how you get on with hdmi audio in guest OS ? Thx

 

[quiOt]

Hi,

Sorry for hijacking this thread.

I've an Intel Xeon E3 1220 v3 on a SuperMicro X10SLL-F with an old GT720 and I'm trying to set up the GPU Passthrough without success. The host crash after starting the guest vm.

uname -r:
5.15.53-1-pve

pveversion:
pve-manager/7.3-3/c3928077 (running kernel: 5.15.53-1-pve)

Grub comand:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_pci.ids=10de:1288,10de:0e0f vfio_iommu_type1.allow_unsafe_interrupts=1 vfio_pci.disable_vga=1 nofb nomodeset initcall_blacklist=sysfb_init video=vesafbff video=efifbff pcie_acs_override=downstream,multifunction"

modprobe.d/vfio.conf:
options vfio-pci ids=10de:1288,10de:0e0f disable_vga=1

modprobe.d/kvm.conf:
options kvm ignore_msrs=1 report_ignored_msrs=0

modprobe.d/iommu_unsafe_interrupts.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1

modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel

lspci -n -s 01:00:
01:00.0 0300: 10de:1288 (rev a1)
01:00.1 0403: 10de:0e0f (rev a1)

lspci -nnk:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 720] [10de:1288] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GK208B [GeForce GT 720] [1462:8c90]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c90]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

GPU Rom:
Exported with GPU-Z and copied to /usr/share/kvm/

VM Conf File:
hostpci0: 01:00,pcie=1,romfile=GK208.rom,x-vga=1

This setup can't run gpu passthrough or i've messed settings ?

Thanks!

 

[William C]

Hi,

Sorry for hijacking this thread.

I've an Intel Xeon E3 1220 v3 on a SuperMicro X10SLL-F with an old GT720 and I'm trying to set up the GPU Passthrough without success.

uname -r:
5.15.53-1-pve

pveversion:
pve-manager/7.3-3/c3928077 (running kernel: 5.15.53-1-pve)

Grub comand:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_pci.ids=10de:1288,10de:0e0f vfio_iommu_type1.allow_unsafe_interrupts=1 vfio_pci.disable_vga=1 nofb nomodeset initcall_blacklist=sysfb_init video=vesafbff video=efifbff"

modprobe.d/vfio.conf:
options vfio-pci ids=10de:1288,10de:0e0f disable_vga=1

modprobe.d/kvm.conf:
options kvm ignore_msrs=1 report_ignored_msrs=0

modprobe.d/iommu_unsafe_interrupts.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1

modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel

lspci -n -s 01:00:
01:00.0 0300: 10de:1288 (rev a1)
01:00.1 0403: 10de:0e0f (rev a1)

lspci -nnk:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 720] [10de:1288] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GK208B [GeForce GT 720] [1462:8c90]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c90]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

GPU Rom:
Exported with GPU-Z and copied to /usr/share/kvm/

VM Conf File:
hostpci0: 01:00,pcie=1,romfile=GK208.rom,x-vga=1

This setup can't run gpu passthrough or i'm have messed settings ?

Thanks!

Looks like you got everything covered

any errors on dmesg?

what output do you get to the display when you boot up? Does this change when you boot up the vm?

also check cat /proc/iomem

 

[quiOt]

Looks like you got everything covered

any errors on dmesg?

what output do you get to the display when you boot up? Does this change when you boot up the vm?

also check cat /proc/iomem

/proc/iomem:
f6000000-f70fffff : PCI Bus 0000:01
f6000000-f6ffffff : 0000:01:00.0
f7000000-f707ffff : 0000:01:00.0
f7080000-f7083fff : 0000:01:00.1

Proxmox crash right after i start the vm. What should I look on dmesg ? running "dmesg -w" nothing showed up between the start and the crash but found this that seems related to the graphic card.

[ 0.189120] pci 0000:00:01.0: bridge window [io 0xe000-0xefff]
[ 0.189122] pci 0000:00:01.0: bridge window [mem 0xf6000000-0xf70fffff]
[ 0.189124] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xf1ffffff 64bit pref]

[ 0.235385] pci 0000:00:01.0: Adding to iommu group 1
[ 0.235474] pci 0000:01:00.1: Adding to iommu group 1

 

[William C]

Is there anything useful in task history in proxnox for the vm? Usually if proxmox is immediately stopping the vm there’s some log entry.

I presume the vm starts ok without stopping if you run it in the console ? Try without pcie ticked if you haven’t already.

I’m sorry that’s about the lot of advice I can give . I’m fairly new to it all too and I’ve got AMD and a different graphics card .

What OS is the guest vm ? I will say with Linux and windows I’ve had no issue with a complete pass through of both video and audio without much fuss. However it’s OSX I’m wishing to use and so far not figured out why it works fine for graphics but not audio

In regards to dmesg I would make sure no errors or hint of the problem . You can get it to show the time with -t or might be -h fyi

 

[quiOt]

@William C , your help is welcome!

In Task History I got: "Unable to read tail ( got 0 bytes)" and sometimes "failed: got timeout". The guest OS is a macOS Monterey VM and it boots ok without the PCI Passthrough. With dmesg didn't showed any errors and syslog reports "VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout" right before the crash.

Edit: Tested now if this setup works with an Windows 11 VM and I was able to passthrough the gt720 and run the VM without crashes. Now that I know that the gpu passthrough works on my setup with win11 I must be missing some configuration on the macos guest vm.

 

[VartKat]

What a nice post! I've been looking for a solution to this BAR 3 issues all night. Big Thanks!

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifb:off" or "video=simplefb:off" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

This works for me NVidia 1060 on Intel PVE 7.3.3

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream pcie_acs_overrid=multifunction nofb nomodeset video=vesafb:off initcall_blacklist=sysfb_init"

Big thanks

 

[jpmn]

What a nice post! I've been looking for a solution to this BAR 3 issues all night. Big Thanks!

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifb:off" or "video=simplefb:off" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

This worked for me, Dell T5810 (Nvidia Quadro M2000)

 

[jaco]

I had the issue after an upgrade from 6.4 to 7.3-4, i tried all things in this thread. But ignored the issue would be in the BIOS as it worked before the upgrade and i had not changed it.
Anyway. On my Gigabyte motherboard, i disabled Boot>CSM Support and it worked.

 

[drx1]

Thank you to everyone who contributed to this thread, I was able to pass-through my nvidia k2200 card to a windows 10 VM successfully.