GPU Passthrough on B450 (Primary GPU)

R

Roy360

New Member
Aug 28, 2020
12
2
3
64
CPU: 2700X
Motherboard: ASROCK B450M Pro4. Bios: 3.30: AMD AGESA Combo-AM4 1.0.0.1 (3.5 breaks IOMMU)
Proxmox: 6.3.2

nano /etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nofb nomodeset video=efifb:off"

dmesg | grep -e DMAR -e IOMMU

Code: 
[    0.650043] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.651154] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.651522] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

/etc/modules
Code: 
vfio
# vfio_iommu_type1
vfio_pci
vfio_virqfd

dmesg | grep 'remapping'
Code: 
[    0.651158] AMD-Vi: Interrupt remapping enabled

find /sys/kernel/iommu_groups/ -type l
Code: 
/sys/kernel/iommu_groups/17/devices/0000:09:00.2
/sys/kernel/iommu_groups/7/devices/0000:00:07.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:0a:00.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/18/devices/0000:09:00.3
/sys/kernel/iommu_groups/8/devices/0000:00:07.1
/sys/kernel/iommu_groups/16/devices/0000:09:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.2
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.1
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:01.0
/sys/kernel/iommu_groups/14/devices/0000:03:04.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.1
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.1
/sys/kernel/iommu_groups/12/devices/0000:00:18.6
/sys/kernel/iommu_groups/12/devices/0000:00:18.4
/sys/kernel/iommu_groups/12/devices/0000:00:18.2
/sys/kernel/iommu_groups/12/devices/0000:00:18.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.7
/sys/kernel/iommu_groups/12/devices/0000:00:18.5
/sys/kernel/iommu_groups/2/devices/0000:00:01.3
/sys/kernel/iommu_groups/20/devices/0000:0a:00.2
/sys/kernel/iommu_groups/10/devices/0000:00:08.1
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0a:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:08.0

lspci -v | grep -e "NVIDIA"
Code: 
08:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050] (rev a1) (prog-if 00 [VGA controller])
08:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

VM Conf
1613863586483.png

I have an actual monitor hooked up to the GPU. When i start the VM it clears the screen and turns it blank.

Syslog after starting the VM:
Code: 
Feb 20 05:27:47 pve pvedaemon[3061]: start VM 100: UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam:
Feb 20 05:27:47 pve pvedaemon[1116]: <root@pam> starting task UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam:
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:48 pve systemd[1]: Created slice qemu.slice.
Feb 20 05:27:48 pve systemd[1]: Started 100.scope.
Feb 20 05:27:48 pve systemd-udevd[3096]: Using default interface naming scheme 'v240'.
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for tap100i0: No such file or directory
Feb 20 05:27:48 pve kernel: device tap100i0 entered promiscuous mode
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for fwbr100i0: No such file or directory
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for fwpr100p0: No such file or directory
Feb 20 05:27:48 pve systemd-udevd[3067]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3067]: Using default interface naming scheme 'v240'.
Feb 20 05:27:48 pve systemd-udevd[3067]: Could not generate persistent MAC address for fwln100i0: No such file or directory
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Feb 20 05:27:48 pve kernel: device fwln100i0 entered promiscuous mode
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Feb 20 05:27:48 pve kernel: device fwpr100p0 entered promiscuous mode
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Feb 20 05:27:51 pve kernel: vfio-pci 0000:08:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve pvedaemon[1116]: <root@pam> end task UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam: OK
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve QEMU[3100]: kvm: vfio-pci: Cannot read device rom at 0000:08:00.0
Feb 20 05:27:52 pve QEMU[3100]: Device option ROM contents are probably invalid (check dmesg).
Feb 20 05:27:52 pve QEMU[3100]: Skip option ROM probe with rombar=0, or load from file with romfile=
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
 
  • Like
Reactions: muckmuck
I've used this GPU before in a X399 setup with the same grub line.
CSM is disabled. SVM and SR_IOV enabled.
 
I have a 2700X on an ASRock X470 and PCI passthrough was broken until BIOS 3.70 with AMD AGESA Combo-AM4 1.0.0.4 Patch B. One of the signs that it is broken is this error message: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff. For your motherboard, this appears to be version 3.90. Have you tried upgrading your BIOS to this version?
Have you tried what this error message Skip option ROM probe with rombar=0, or load from file with romfile= suggest? You'll need to save the videocard ROM to a file, which is a bit of a hassle but there are guides on the internet.
EDIT: fixed AGESA version in the first sentence.
 
Last edited:
  • Like
Reactions: Roy360
avw said:
I have a 2700X on an ASRock X470 and PCI passthrough was broken until BIOS 3.70 with AGESA Combo-AM4 1.0.0.3 ABB. One of the signs that it is broken is this error message: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff. For your motherboard, this appears to be version 3.90. Have you tried upgrading your BIOS to this version?
Have you tried what this error message Skip option ROM probe with rombar=0, or load from file with romfile= suggest? You'll need to save the videocard ROM to a file, which is a bit of a hassle but there are guides on the internet.
Click to expand...
I haven't tried anything beyond 3.5 since the description says: ASRock do NOT recommend updating this BIOS if Pinnacle, Raven or Summit Ridge CPU is being used on your system. I'm scared the Cpu will stop booting if I flash too high

Under CPU support, all the Pinnacle CPUs are listed to support all the bios unlike the 1000 and 3000 series, which turn on and off based on the excat sku.
 
Roy360 said:
I haven't tried anything beyond 3.5 since the description says: ASRock do NOT recommend updating this BIOS if Pinnacle, Raven or Summit Ridge CPU is being used on your system. I'm scared the Cpu will stop booting if I flash too high
Click to expand...
Maybe you can try version 2.00 (AGESA 1.0.0.6), which is from before AMD AGESA broke PCI passthrough. However, ASRock probably won't allow you to flash this earlier version.
Your problem is very like the issue with my motherboard: 1.90 worked but it was shipped with 3.20 (ASRock does not allow flashing older version) and I had to wait until 3.70 (ASRock do NOT recommend updating this BIOS if Pinnacle, Raven or Summit Ridge CPU is being used on your system), before it worked again.
EDIT: more infromation about AGESA breaking PCI passthrough.
 
Last edited:
  • Like
Reactions: Roy360
avw said:
Maybe you can try version 2.00 (AGESA 1.0.0.6), which is from before AMD AGESA broke PCI passthrough. However, ASRock probably won't allow you to flash this earlier version.
Your problem is very like the issue with my motherboard: 1.90 worked but it was shipped with 3.20 (ASRock does not allow flashing older version) and I had to wait until 3.70 (ASRock do NOT recommend updating this BIOS if Pinnacle, Raven or Summit Ridge CPU is being used on your system), before it worked again.
EDIT: more infromation about AGESA breaking PCI passthrough.
Click to expand...
Well I bite the bullet and flashed 3.90
Update AMD AGESA Combo-AM4 1.0.0.4 Patch B​

I don't see any IOMMU groups now.

There is no output now:

1614288805292.png


Flashing the bios seems to be doing something....keep going up the bios update ladder?

I have SVM (virtualization) and SR-IOV root enabled.
 
I would advise against going with a higher BIOS version; it completely broke my setup with X470 and 2700X.
SVM and SR-IOV don't really matter for PCI passthrough (but keep them turned on). You need to enable IOMMU (not just Auto, but Enabled), AER and ACS.
There should be some output about IOMMU and groups if you did a CMOS reset and enabled all necessary virtualization settings. Best way to check IOMMU groups is for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done.
EDIT: the IOMMU and ACS settings are under NBIO Common Options/NB Configuration according to the manual (they are often hard to find).
EDIT: AGESA Combo-AM4 1.0.0.4 Patch B is the right BIOS version that fixes passthrough for many systems according to Level1Techs forum.
 
Last edited:
  • Like
Reactions: Roy360
avw said:
I would advise against going with a higher BIOS version; it completely broke my setup with X470 and 2700X.
SVM and SR-IOV don't really matter for PCI passthrough (but keep them turned on). You need to enable IOMMU (not just Auto, but Enabled), AER and ACS.
There should be some output about IOMMU and groups if you did a CMOS reset and enabled all necessary virtualization settings. Best way to check IOMMU groups is for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done.
EDIT: the IOMMU and ACS settings are under NBIO Common Options/NB Configuration according to the manual (they are often hard to find).
EDIT: AGESA Combo-AM4 1.0.0.4 Patch B is the right BIOS version that fixes passthrough for many systems according to Level1Techs forum.
Click to expand...
Thanks. ACS and IOMMU were disabled by default in 3.90. Would not have found it without you,
 
Roy360 said:
CPU: 2700X
Motherboard: ASROCK B450M Pro4. Bios: 3.30: AMD AGESA Combo-AM4 1.0.0.1 (3.5 breaks IOMMU)
Proxmox: 6.3.2

nano /etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nofb nomodeset video=efifb:off"

dmesg | grep -e DMAR -e IOMMU

Code: 
[    0.650043] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.651154] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.651522] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

/etc/modules
Code: 
vfio
# vfio_iommu_type1
vfio_pci
vfio_virqfd

dmesg | grep 'remapping'
Code: 
[    0.651158] AMD-Vi: Interrupt remapping enabled

find /sys/kernel/iommu_groups/ -type l
Code: 
/sys/kernel/iommu_groups/17/devices/0000:09:00.2
/sys/kernel/iommu_groups/7/devices/0000:00:07.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:0a:00.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/18/devices/0000:09:00.3
/sys/kernel/iommu_groups/8/devices/0000:00:07.1
/sys/kernel/iommu_groups/16/devices/0000:09:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.2
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.1
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:01.0
/sys/kernel/iommu_groups/14/devices/0000:03:04.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.1
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.1
/sys/kernel/iommu_groups/12/devices/0000:00:18.6
/sys/kernel/iommu_groups/12/devices/0000:00:18.4
/sys/kernel/iommu_groups/12/devices/0000:00:18.2
/sys/kernel/iommu_groups/12/devices/0000:00:18.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.7
/sys/kernel/iommu_groups/12/devices/0000:00:18.5
/sys/kernel/iommu_groups/2/devices/0000:00:01.3
/sys/kernel/iommu_groups/20/devices/0000:0a:00.2
/sys/kernel/iommu_groups/10/devices/0000:00:08.1
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0a:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:08.0

lspci -v | grep -e "NVIDIA"
Code: 
08:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050] (rev a1) (prog-if 00 [VGA controller])
08:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

VM Conf
View attachment 23858

I have an actual monitor hooked up to the GPU. When i start the VM it clears the screen and turns it blank.

Syslog after starting the VM:
Code: 
Feb 20 05:27:47 pve pvedaemon[3061]: start VM 100: UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam:
Feb 20 05:27:47 pve pvedaemon[1116]: <root@pam> starting task UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam:
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:47 pve kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 20 05:27:48 pve systemd[1]: Created slice qemu.slice.
Feb 20 05:27:48 pve systemd[1]: Started 100.scope.
Feb 20 05:27:48 pve systemd-udevd[3096]: Using default interface naming scheme 'v240'.
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for tap100i0: No such file or directory
Feb 20 05:27:48 pve kernel: device tap100i0 entered promiscuous mode
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for fwbr100i0: No such file or directory
Feb 20 05:27:48 pve systemd-udevd[3096]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3096]: Could not generate persistent MAC address for fwpr100p0: No such file or directory
Feb 20 05:27:48 pve systemd-udevd[3067]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 20 05:27:48 pve systemd-udevd[3067]: Using default interface naming scheme 'v240'.
Feb 20 05:27:48 pve systemd-udevd[3067]: Could not generate persistent MAC address for fwln100i0: No such file or directory
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Feb 20 05:27:48 pve kernel: device fwln100i0 entered promiscuous mode
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Feb 20 05:27:48 pve kernel: device fwpr100p0 entered promiscuous mode
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Feb 20 05:27:48 pve kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Feb 20 05:27:48 pve kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Feb 20 05:27:51 pve kernel: vfio-pci 0000:08:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve pvedaemon[1116]: <root@pam> end task UPID:pve:00000BF5:0001380A:60312A73:qmstart:100:root@pam: OK
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve QEMU[3100]: kvm: vfio-pci: Cannot read device rom at 0000:08:00.0
Feb 20 05:27:52 pve QEMU[3100]: Device option ROM contents are probably invalid (check dmesg).
Feb 20 05:27:52 pve QEMU[3100]: Skip option ROM probe with rombar=0, or load from file with romfile=
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 05:27:52 pve kernel: vfio-pci 0000:08:00.1: vfio_bar_restore: reset recovery - restoring BARs
Click to expand...
Dude, THANK YOU!!!!! You adding "(3.5 breaks IOMMU)" to your post saved me from almost giving up on my Proxmox migration and who knows how many hours!
 
Apologies for the thread necro, but had to offer my thanks as well. I'm just coming from TrueNAS-SCALE-22.12.3.3. I also have the Asrock B450 Pro4 (ATX version) and had upgraded to BIOS version 3.5 forever ago. I too have a 2700X and saw the warning about upgrading further so I stopped upgrading the BIOS. I upgraded to version 3.9 which included AGESA Combo-AM4 1.0.0.4 Patch B based on the recommendations from this thread and GPU passthrough is working great after enabling SR-IOV, IOMMU, AER and ACS in settings. I also enabled PCI-E ARI, but might disable that now that things are working as expected.

I still have CSM enabled because my GPUS (a Radeon HD 7870 and HD 8570) don't have UEFI BIOS.

For reference I was getting a different error on v3.50 of the BIOS when running "dmesg | vfio" (I'm not 100% on that command, still new to linux and I don't have the terminal in front of me). The errors I was getting when starting my VM were several instances of:

"not ready 3905ms after bus reset; waiting"

Again, really appreciate the leg work done on discovering a solution and providing sources. Nice job.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back