[SOLVED] Seeking help with GPU passthrough

C

coen800

New Member
Jul 13, 2021
4
1
3
23
I just build my first proxmox server and one of the uses is that I want a VM to process my 4K CCTV foottage with motioneye. I have a single GPU in the proxmox server I want to use to offload this from the cpu. I want to passthrough my GPU to a ubuntu 20.04 VM. I can't seem to get around the BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] error.

This is my hardware:
AMD 2700X
Asus Prime X470-PRO
Gigabyte GTX 1050TI (in first x16 slot)
LSI HBA card (in second x16 slot)
And couple of disks
I run proxmox 7

I tried multiple things to get around this error but nothing seems to work.

Here are the steps I took and my current configuration.

On server (proxmox):
  • Included quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset textonly video=vesafb:off video=efifb:off in grub.
  • proxmox-boot-tool refresh (update-grub)
  • included the following in /etc/modules:
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  • IOMMU interrupt remaping:
    echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
    echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
  • Blacklist drivers
    echo "blacklist nvidiafb" >> /etc/modprobe.d/pve-blacklist.conf
    echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf
    echo "blacklist nvidia" >> /etc/modprobe.d/pve-blacklist.conf
  • reboot
  • added GPU to VFIO ( I made sure the I got the correct Id’s)
    echo "options vfio-pci ids=10de:1c82,10de:0fb9 disable_vga=1"> /etc/modprobe.d/vfio.conf
  • update-initramfs -u
  • reboot
Also on server (proxmox) to hide VM identifiers from Nvidia:
  • Hide VM identifiers from Nvidia
    SSH into Proxmox
    cd /etc/pve/qemu-server
    nano ###.conf (# is the VM identifier of the server)
    Added line:
    cpu: host,hidden=1
    Save file and exit.
On VM (Ubuntu 20.04 server):
  • Initial install done so SSH access is configured.
  • Added the GPU via proxmox hardware page with options:
    All functions, ROM-Bar and PCI-Express
  • Shutdown and start VM so GPU is added.
  • Download Nvidia drivers, tried both from ubuntu packages and Nvidia driver page.
  • Disable Nouveau drivers in kernel:
    echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nvidia-nouveau.conf
    echo "options nouveau modset=0" >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf
  • sudo update-initramfs -u
  • reboot
When I run sudo nvidia-smi in the ubuntu vm it hangs and in the dmesg log of proxmox I see thousands of BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] errors.

I did notice when booting the proxmox server it acctually displays the boot process and stops the display output untill I start the Ubuntu VM.
If someone has some suggestions I could try please let me know.

EDIT: I forgot to add I did try using the romfile option with the original vbios and a modified version with changed vendor ID and device ID but no luck.
Also here is my vm config:
Code: 
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: vm-mirror:vm-102-disk-0,size=1M
hostpci0: 0000:09:00,pcie=1,romfile=gtx1050ti.rom
machine: q35
memory: 4096
name: motioneye
net0: virtio=82:88:61:35:57:34,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: vm-mirror:vm-102-disk-1,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=e7cb7f74-4c8e-41ca-83da-15b14ef533d7
sockets: 1
vmgenid: 32ac0b3a-d55b-44c5-898d-d7db3215ed4c
 
Last edited:
Similar experience. I also try to passthrough my 3060ti in Proxmox 7.0 to my Windows10 vm. The CPU of machine is 5800x, AMD, too. I also got similar BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] and the vm hangs when I try to install latest(466.77, June 2021) Nvidia driver. Maybe we can share what we observe and see how to solve it.
 
To give an update on the thread, I've managed to fix it. As dcsapak pointed out it turns out I wasn't using grub thus the options I included in grub were not working.

Using the cat /proc/iomem command I noticed the GPU was actually in use by efifb.
Code: 
e0000000-f1ffffff : PCI Bus 0000:09
e0000000-efffffff : 0000:09:00.0
  e0000000-efffffff : vfio-pci
f0000000-f1ffffff : 0000:09:00.0
  f1000000-f12fffff : efifb

I used the echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind command to mannually unbind efifb. After that I was able to use the GPU in the VM.
 
  • Like
Reactions: leesteken
ohayo world said:
Similar experience. I also try to passthrough my 3060ti in Proxmox 7.0 to my Windows10 vm. The CPU of machine is 5800x, AMD, too. I also got similar BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] and the vm hangs when I try to install latest(466.77, June 2021) Nvidia driver. Maybe we can share what we observe and see how to solve it.
Click to expand...
You need to check if your GPU is used by some other proces with the cat /proc/iomem command. In your case the bar error is from memory address 0xd0000000-0xdfffffff so look for that.
 
coen800 said:
You need to check if your GPU is used by some other proces with the cat /proc/iomem command. In your case the bar error is from memory address 0xd0000000-0xdfffffff so look for that.
Click to expand...
Thanks for your reply, I've fixed my own problem, too. In my circumstence maybe the contributing factor is 1) install from 20H2 to 21H1 2) use geforce experince instead of downloading directly or 3) unplug DP cables from GPU. From your reply, I think the most probable factor is the third, which may result in some process running on the GPU.
 
You must log in or register to reply here.
Top Bottom