[SOLVED] PVE7: vfio-pci xxxx:xx:xx.x: No more image in the PCI ROM

B

bishop f4

New Member
Apr 13, 2022
11
0
1
SOLVED BY SETTING CSM UEFI SETTINGS FOR THE PCIE LANES
IF YOU ARE USING AN X79 CHINESE MOTHERBOARD WITH AMERICAN MEGATRENDS BIOS MAKE SURE TO TURN ON UEFI EXPOSURE FOR PCIE LANES IF YOU HAVE PVE INSTALLED WITH UEFI.
YOU ALSO WILL NOT NEED TO USE A ROM FILE AS THE ROM BAR WILL BE FULLY EXPOSED AND NO ERROR WILL OCCUR
Picture attached:
1652743271794.png


ORIGINAL POST:
Having an issue using main GPU passthrough on a PVE7 host. I have a patched ROM file I have attached which causes a black screen on start and then does not boot. Without the patched ROM, it doesn't recognize the GPU at all.

System Specs:
2x E5-4620v2
6x 16gb RAM
EVGA GTX1070ti

I did not add options vfio-pci ids=10de:1b82,10de:10f0 disable_vga=1 to /etc/modprobe.d/vfio.conf because it didn't make a difference in the black screen or "No more image in the PCI ROM" error.

Used this script to trim down ROM:
https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher
Video for another way I tried:
https://www.youtube.com/watch?v=1IP-h9IKof0&ab_channel=SpaceinvaderOne

cat /etc/pve/qemu-server/100.conf (trimmed down to the related items) (adding ROM file causes boot error and black screen, no vPOST)
Code: 
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off'
balloon: 0
bios: ovmf
cores: 6
cpu: host,hidden=1,flags=+pcid
hostpci0: 0000:03:00,pcie=1,romfile=PATCHED.EVGA.rom
machine: pc-q35-6.1
memory: 12288
meta: creation-qemu=6.1.0,ctime=1650000137
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=73696d32-f77b-4d9c-bebc-b6fbee433cc2
sockets: 2

dmesg | grep -e DMAR -e IOMMU -e 3:00
Code: 
[    0.015182] ACPI: DMAR 0x000000007C040A40 000138 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.015206] ACPI: Reserving DMAR table memory at [mem 0x7c040a40-0x7c040b77]
[    0.330222] DMAR: IOMMU enabled
[    0.696605] DMAR: Host address width 46
[    0.696607] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[    0.696614] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.696617] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
[    0.696621] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.696623] DMAR: RMRR base: 0x0000007c663000 end: 0x0000007c671fff
[    0.696626] DMAR: ATSR flags: 0x0
[    0.696627] DMAR: RHSA base: 0x000000fbffe000 proximity domain: 0x1
[    0.696629] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
[    0.696632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffe000 IOMMU 0
[    0.696634] DMAR-IR: IOAPIC id 0 under DRHD base  0xdfffc000 IOMMU 1
[    0.696636] DMAR-IR: IOAPIC id 2 under DRHD base  0xdfffc000 IOMMU 1
[    0.696637] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
[    0.696639] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.697308] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.157850] pci 0000:03:00.0: [10de:1b82] type 00 class 0x030000
[    1.157861] pci 0000:03:00.0: reg 0x10: [mem 0xde000000-0xdeffffff]
[    1.157870] pci 0000:03:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
[    1.157880] pci 0000:03:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
[    1.157887] pci 0000:03:00.0: reg 0x24: [io  0x8000-0x807f]
[    1.157894] pci 0000:03:00.0: reg 0x30: [mem 0xdf000000-0xdf07ffff pref]
[    1.157901] pci 0000:03:00.0: enabling Extended Tags
[    1.157917] pci 0000:03:00.0: BAR 3: assigned to efifb
[    1.157989] pci 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:02.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[    1.158033] pci 0000:03:00.1: [10de:10f0] type 00 class 0x040300
[    1.158044] pci 0000:03:00.1: reg 0x10: [mem 0xdf080000-0xdf083fff]
[    1.158080] pci 0000:03:00.1: enabling Extended Tags
[    1.159758] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.159791] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME AER PCIeCapability LTR]
[    1.184611] pci 0000:03:00.0: vgaarb: setting as boot VGA device
[    1.184613] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.184628] pci 0000:03:00.0: vgaarb: bridge control possible
[    1.264742] pci 0000:03:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.264784] pci 0000:03:00.1: D0 power state depends on 0000:03:00.0
[    1.265221] DMAR: No SATC found
[    1.265226] DMAR: dmar0: Using Queued invalidation
[    1.265244] DMAR: dmar1: Using Queued invalidation
[    1.266059] pci 0000:03:00.0: Adding to iommu group 25
[    1.266089] pci 0000:03:00.1: Adding to iommu group 25
[    1.269500] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    4.669564] snd_hda_intel 0000:03:00.1: Disabling MSI
[    4.669586] snd_hda_intel 0000:03:00.1: Handle vga_switcheroo audio client
[    4.772097] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input14
[    4.772162] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input15
[    4.772215] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input16
[    4.772272] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input17
[    4.772330] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input18
[    4.772383] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input19
[    4.772451] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input20
[  146.152872] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  146.172941] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  146.192738] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  151.708537] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  151.710666] vfio-pci 0000:03:00.0: No more image in the PCI ROM

uname -r
Code: 
5.13.19-6-pve

cat /etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

cat /proc/cmdline
Code: 
BOOT_IMAGE=/boot/vmlinuz-5.13.19-6-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off

cat /etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Any help would be appreciated, thank you
 
Last edited:
bishop f4 said:
SOLVED BY SETTING CSM UEFI SETTINGS FOR THE PCIE LANES
IF YOU ARE USING AN X79 CHINESE MOTHERBOARD WITH AMERICAN MEGATRENDS BIOS MAKE SURE TO TURN ON UEFI EXPOSURE FOR PCIE LANES IF YOU HAVE PVE INSTALLED WITH UEFI.
YOU ALSO WILL NOT NEED TO USE A ROM FILE AS THE ROM BAR WILL BE FULLY EXPOSED AND NO ERROR WILL OCCUR


Having an issue using main GPU passthrough on a PVE7 host. I have a patched ROM file I have attached which causes a black screen on start and then does not boot. Without the patched ROM, it doesn't recognize the GPU at all.

System Specs:
2x E5-4620v2
6x 16gb RAM
EVGA GTX1070ti

I did not add options vfio-pci ids=10de:1b82,10de:10f0 disable_vga=1 to /etc/modprobe.d/vfio.conf because it didn't make a difference in the black screen or "No more image in the PCI ROM" error.

Used this script to trim down ROM:
https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher
Video for another way I tried:
https://www.youtube.com/watch?v=1IP-h9IKof0&ab_channel=SpaceinvaderOne

cat /etc/pve/qemu-server/100.conf (trimmed down to the related items) (adding ROM file causes boot error and black screen, no vPOST)
Code: 
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off'
balloon: 0
bios: ovmf
cores: 6
cpu: host,hidden=1,flags=+pcid
hostpci0: 0000:03:00,pcie=1,romfile=PATCHED.EVGA.rom
machine: pc-q35-6.1
memory: 12288
meta: creation-qemu=6.1.0,ctime=1650000137
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=73696d32-f77b-4d9c-bebc-b6fbee433cc2
sockets: 2

dmesg | grep -e DMAR -e IOMMU -e 3:00
Code: 
[    0.015182] ACPI: DMAR 0x000000007C040A40 000138 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.015206] ACPI: Reserving DMAR table memory at [mem 0x7c040a40-0x7c040b77]
[    0.330222] DMAR: IOMMU enabled
[    0.696605] DMAR: Host address width 46
[    0.696607] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[    0.696614] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.696617] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
[    0.696621] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.696623] DMAR: RMRR base: 0x0000007c663000 end: 0x0000007c671fff
[    0.696626] DMAR: ATSR flags: 0x0
[    0.696627] DMAR: RHSA base: 0x000000fbffe000 proximity domain: 0x1
[    0.696629] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
[    0.696632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffe000 IOMMU 0
[    0.696634] DMAR-IR: IOAPIC id 0 under DRHD base  0xdfffc000 IOMMU 1
[    0.696636] DMAR-IR: IOAPIC id 2 under DRHD base  0xdfffc000 IOMMU 1
[    0.696637] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
[    0.696639] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.697308] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.157850] pci 0000:03:00.0: [10de:1b82] type 00 class 0x030000
[    1.157861] pci 0000:03:00.0: reg 0x10: [mem 0xde000000-0xdeffffff]
[    1.157870] pci 0000:03:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
[    1.157880] pci 0000:03:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
[    1.157887] pci 0000:03:00.0: reg 0x24: [io  0x8000-0x807f]
[    1.157894] pci 0000:03:00.0: reg 0x30: [mem 0xdf000000-0xdf07ffff pref]
[    1.157901] pci 0000:03:00.0: enabling Extended Tags
[    1.157917] pci 0000:03:00.0: BAR 3: assigned to efifb
[    1.157989] pci 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:02.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[    1.158033] pci 0000:03:00.1: [10de:10f0] type 00 class 0x040300
[    1.158044] pci 0000:03:00.1: reg 0x10: [mem 0xdf080000-0xdf083fff]
[    1.158080] pci 0000:03:00.1: enabling Extended Tags
[    1.159758] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.159791] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME AER PCIeCapability LTR]
[    1.184611] pci 0000:03:00.0: vgaarb: setting as boot VGA device
[    1.184613] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.184628] pci 0000:03:00.0: vgaarb: bridge control possible
[    1.264742] pci 0000:03:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.264784] pci 0000:03:00.1: D0 power state depends on 0000:03:00.0
[    1.265221] DMAR: No SATC found
[    1.265226] DMAR: dmar0: Using Queued invalidation
[    1.265244] DMAR: dmar1: Using Queued invalidation
[    1.266059] pci 0000:03:00.0: Adding to iommu group 25
[    1.266089] pci 0000:03:00.1: Adding to iommu group 25
[    1.269500] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    4.669564] snd_hda_intel 0000:03:00.1: Disabling MSI
[    4.669586] snd_hda_intel 0000:03:00.1: Handle vga_switcheroo audio client
[    4.772097] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input14
[    4.772162] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input15
[    4.772215] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input16
[    4.772272] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input17
[    4.772330] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input18
[    4.772383] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input19
[    4.772451] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:02.0/0000:03:00.1/sound/card0/input20
[  146.152872] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  146.172941] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  146.192738] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  151.708537] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  151.710666] vfio-pci 0000:03:00.0: No more image in the PCI ROM

uname -r
Code: 
5.13.19-6-pve

cat /etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

cat /proc/cmdline
Code: 
BOOT_IMAGE=/boot/vmlinuz-5.13.19-6-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off

cat /etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Any help would be appreciated, thank you
Click to expand...
I am also having the exact same issue with an X79 Motherbaord, can you be a bit more specific about what setting in the bios you changed and what you changed it to?

To be more specific;
- I cant find a setting for "UEFI EXPOSURE"
- I cant see settings per PCIE lane to change CSM setting; "SETTING CSM UEFI SETTINGS FOR THE PCIE LANES"

currently my CSM setting for the BIOS are;
Launch CSM: Enabled
Boot option filter: UEFI and legacy
Launch PXE OpROM policy: UEFI Only
Launch Storage OpROM policy: UEFI Only
Launch Video OpROM policy: Legacy Only

Other PCI device ROM priority: UEFI OpRom
 
greenie said:
I am also having the exact same issue with an X79 Motherbaord, can you be a bit more specific about what setting in the bios you changed and what you changed it to?

To be more specific;
- I cant find a setting for "UEFI EXPOSURE"
- I cant see settings per PCIE lane to change CSM setting; "SETTING CSM UEFI SETTINGS FOR THE PCIE LANES"

currently my CSM setting for the BIOS are;
Launch CSM: Enabled
Boot option filter: UEFI and legacy
Launch PXE OpROM policy: UEFI Only
Launch Storage OpROM policy: UEFI Only
Launch Video OpROM policy: Legacy Only

Other PCI device ROM priority: UEFI OpRom
Click to expand...
Should have been more specific, here is a screenshot of my CSM Parameters:
1652743164481.png

You can also try turning off CSM and seeing if that will work as well.

Since we are passing a GPU through, we need to pass Video OpROM.
I also have everything set to UEFI since I have UEFI boot on and do not plan on having anything other than UEFI.

Hope this helps!
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back