GPU Passthrough not working (Bar 3)

drdownload

Member
Sep 7, 2017
38
1
13
43
I'm already quite of options what to do next, I had a Server with working GPU passthrough (1050Ti) but I thought I want a new Thing also to game on and now it just not gonna work.

Hardware: Asrock B350 Pro4 Motherboard, 64GB RAM, i5 8600k, Zotac 1070 Mini, iGPU is still enabled in the BIOS because I wanted to use it for kvmgt. Bios ignores setting the iGPU to primary and by default uses the GPU DVI (not displayport or HDMI) and not the iGPU ones.

Software: Updated Proxmox 6, only extra installs are glusterfs-server and git.

After some playing around the following config results in only a black screen and in Windows only a non working display driver is listed.

Only real error messages are here:

root@server7:~# dmesg | grep vfio [ 8.137771] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem [ 8.159446] vfio_pci: add [10de:1b81[ffffffff:ffffffff]] class 0x000000/00000000 [ 8.179438] vfio_pci: add [10de:10f0[ffffffff:ffffffff]] class 0x000000/00000000 [ 214.054862] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 214.056079] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xb0000000-0xb1ffffff 64bit pref] [ 214.056287] vfio-pci 0000:01:00.0: No more image in the PCI ROM [ 983.544596] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 983.545567] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xb0000000-0xb1ffffff 64bit pref] [ 983.545782] vfio-pci 0000:01:00.0: No more image in the PCI ROM [ 1432.320898] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 1432.321868] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xb0000000-0xb1ffffff 64bit pref] [ 1432.322116] vfio-pci 0000:01:00.0: No more image in the PCI ROM

Grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb,efifb:eek:ff"
GRUB_CMDLINE_LINUX=""

lspci -v

Code:
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ZOTAC International (MCO) Ltd. GP104 [GeForce GTX 1070]
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at 82000000 (32-bit, non-prefetchable) [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 3000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GP104 High Definition Audio Controller
        Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at 83080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

/etc/modules
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
#vfio-mdev
#kvmgt

/etc/modprobe.d/blacklist
Code:
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist i915

Other Options
kvm.conf
options kvm ignore_msrs=1

iommu...conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

vfio.conf
options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1

qemu.conf
Code:
  GNU nano 3.2                                                       /etc/pve/qemu-server/100.conf                                                                 

agent: 1
bios: ovmf
bootdisk: scsi0
cores: 1
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
efidisk0: node7-ssd1:100/vm-100-disk-1.qcow2,size=128K
hostpci0: 01:00,pcie=1,romfile=vbios.bin,x-vga=1
ide0: node7-ssd1:iso/Win10_1909_German_x64.iso,media=cdrom,size=5315440K
ide2: node7-ssd1:iso/virtio-win-0.1.173.iso,media=cdrom,size=385062K
machine: q35
memory: 8192
name: win10
net0: e1000=D6:2C:E9:39:0D:F7,bridge=vmbr0,firewall=1
net1: virtio=36:25:25:6D:8D:52,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
parent: Installation
scsi0: node7-ssd1:100/vm-100-disk-0.qcow2,discard=on,size=256G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=a2c6a8cb-97fc-49e1-8362-ed929b5c4021
sockets: 2
vmgenid: a36ef9c8-3db5-4ceb-9ade-a1df85b6ab8e
 
Hey there,
did you ever fix your problem? I'm running into the exact same thing and I've been breaking my head over it for the past two weeks...

My setup
- Asus B350 am4 pro
- 64 gb ram
- Ryzen 7 1700
- gtx 1050 ti
- Proxmox 6.1 (latest updates)

NVidia card shows up in my VM, whether I use Win10 or Ubuntu. On both I get the drivers installed succesfully, only Win10 give me Code 43 and Ubuntu gives me a "No devices found" when running nvidia-smi.

I used to run my 6.1 install with a GTX970, had no problems with that card.

Did you patch your BIOS rom before using it? (source: nvidia-kvm-patcher)


When starting the VM I get:
Code:
kvm: -device vfio-pci,host=0000:08:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: Failed to mmap 0000:08:00.0 BAR 3. Performance may be slow


On the host I also get the same BAR 3 errors:

Code:
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]


EDIT: I think I just figured it out....
I changed the following option in my GRUB line:
Code:
video=vesafb:off,efifb:off

Changed it to:
Code:
video=vesafb:off video=efifb:off

I got the idea from this post: https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/post-257462
 
Last edited:
  • Like
Reactions: Andreas S.
oh my god, i've spent so much time.
Thanks, this is the solution!

WRONG video=vesafb:off,efifb:off
RIGHT video=vesafb:off video=efifb:off
 
oh my god, i've spent so much time.
Thanks, this is the solution!

WRONG video=vesafb:off,efifb:off
RIGHT video=vesafb:off video=efifb:off

Thanks for replying and confirming!

I'm glad this also worked for someone else :D
 
  • Like
Reactions: jhz
Just wanted to post and let others know that I was experiencing the same problem, and the above fix worked for me as well as of 01/29/21.
 
Hello,
Just wanted to bring up that this is what my rig needed as well. As of 2021-02-19
  • GTX 1660 Super
  • AMD R7 2700
My /etc/default/grub looks as follows:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=vesafb:off video=efifb:off"
 
Last edited:
I am posting from a TRX40 System with a single GPU which is supposed to be passed through on a headless (and gpu-less) PVE. I have the systemd command line set with video=vesafb:off video=efifb:off but i still have the problem unless i unplug everything from my GPU during boot and only plug it in once i am about to boot the VM. Has anybody had a similar issue? I once saw the option to specify which GPU is used for the Terminal (usually an iGPU) but i never saw an option that makes sure no GPU is grabbed no Matter what. The GPU is a 2080 ti and i am connecting a monitor with peripherals via USB-C.
 
I've been working on and off on this for months and these two lines are what fixed my issue as of 10.2.2021. Thank you soooo much!
oh my god, i've spent so much time.
Thanks, this is the solution!

WRONG video=vesafb:off,efifb:off
RIGHT video=vesafb:off video=efifb:off
 
Hey there,
did you ever fix your problem? I'm running into the exact same thing and I've been breaking my head over it for the past two weeks...

My setup
- Asus B350 am4 pro
- 64 gb ram
- Ryzen 7 1700
- gtx 1050 ti
- Proxmox 6.1 (latest updates)

NVidia card shows up in my VM, whether I use Win10 or Ubuntu. On both I get the drivers installed succesfully, only Win10 give me Code 43 and Ubuntu gives me a "No devices found" when running nvidia-smi.

I used to run my 6.1 install with a GTX970, had no problems with that card.

Did you patch your BIOS rom before using it? (source: nvidia-kvm-patcher)


When starting the VM I get:
Code:
kvm: -device vfio-pci,host=0000:08:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: Failed to mmap 0000:08:00.0 BAR 3. Performance may be slow


On the host I also get the same BAR 3 errors:

Code:
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
Jan 22 19:57:01 pve kernel: vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]


EDIT: I think I just figured it out....
I changed the following option in my GRUB line:
Code:
video=vesafb:off,efifb:off

Changed it to:
Code:
video=vesafb:off video=efifb:off

I got the idea from this post: https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/post-257462
If the host uses a kernel with version 5.15.30, (PVE 7.2-3) then the flag - "initcall_blacklist=sysfb_init" should be specified in GRUB_CMDLINE_LINUX_DEFAULT.
In this case, the flags "video=vesafb:eek:ff video=efifb:eek:ff video=simplefb:eek:ff" can be omitted.
This is exactly the problem that was solved.
My line: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init textonly"
Card Model: GIGABYTE GeForce GT 710 LP D5 [GV-N710D5-2GIL]
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!