[SOLVED] Seeking help with GPU passthrough

[coen800]

I just build my first proxmox server and one of the uses is that I want a VM to process my 4K CCTV foottage with motioneye. I have a single GPU in the proxmox server I want to use to offload this from the cpu. I want to passthrough my GPU to a ubuntu 20.04 VM. I can't seem to get around the BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] error.

This is my hardware:
AMD 2700X
Asus Prime X470-PRO
Gigabyte GTX 1050TI (in first x16 slot)
LSI HBA card (in second x16 slot)
And couple of disks
I run proxmox 7

I tried multiple things to get around this error but nothing seems to work.

Here are the steps I took and my current configuration.

On server (proxmox):

• Included quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset textonly video=vesafb:off video=efifb:off in grub.
• proxmox-boot-tool refresh (update-grub)
• included the following in /etc/modules:
  vfio
  vfio_iommu_type1
  vfio_pci
  vfio_virqfd
• IOMMU interrupt remaping:
  echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
  echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
• Blacklist drivers
  echo "blacklist nvidiafb" >> /etc/modprobe.d/pve-blacklist.conf
  echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf
  echo "blacklist nvidia" >> /etc/modprobe.d/pve-blacklist.conf
• reboot
• added GPU to VFIO ( I made sure the I got the correct Id’s)
  echo "options vfio-pci ids=10de:1c82,10de:0fb9 disable_vga=1"> /etc/modprobe.d/vfio.conf
• update-initramfs -u
• reboot

Also on server (proxmox) to hide VM identifiers from Nvidia:

• Hide VM identifiers from Nvidia
  SSH into Proxmox
  cd /etc/pve/qemu-server
  nano ###.conf (# is the VM identifier of the server)
  Added line:
  cpu: host,hidden=1
  Save file and exit.

On VM (Ubuntu 20.04 server):

• Initial install done so SSH access is configured.
• Added the GPU via proxmox hardware page with options:
  All functions, ROM-Bar and PCI-Express
• Shutdown and start VM so GPU is added.
• Download Nvidia drivers, tried both from ubuntu packages and Nvidia driver page.
• Disable Nouveau drivers in kernel:
  echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nvidia-nouveau.conf
  echo "options nouveau modset=0" >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf
• sudo update-initramfs -u
• reboot

When I run sudo nvidia-smi in the ubuntu vm it hangs and in the dmesg log of proxmox I see thousands of BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] errors.

I did notice when booting the proxmox server it acctually displays the boot process and stops the display output untill I start the Ubuntu VM.
If someone has some suggestions I could try please let me know.

EDIT: I forgot to add I did try using the romfile option with the original vbios and a modified version with changed vendor ID and device ID but no luck.
Also here is my vm config:

agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: vm-mirror:vm-102-disk-0,size=1M
hostpci0: 0000:09:00,pcie=1,romfile=gtx1050ti.rom
machine: q35
memory: 4096
name: motioneye
net0: virtio=82:88:61:35:57:34,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: vm-mirror:vm-102-disk-1,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=e7cb7f74-4c8e-41ca-83da-15b14ef533d7
sockets: 1
vmgenid: 32ac0b3a-d55b-44c5-898d-d7db3215ed4c

 

[dcsapak]

do you boot with efi and zfs ? if yes the grub config is the wrong place to add kernel parameters, see https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline

to verify, you can check cat /proc/cmdline

 

[ohayo world]

Similar experience. I also try to passthrough my 3060ti in Proxmox 7.0 to my Windows10 vm. The CPU of machine is 5800x, AMD, too. I also got similar BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] and the vm hangs when I try to install latest(466.77, June 2021) Nvidia driver. Maybe we can share what we observe and see how to solve it.

 

[coen800]

To give an update on the thread, I've managed to fix it. As dcsapak pointed out it turns out I wasn't using grub thus the options I included in grub were not working.

Using the cat /proc/iomem command I noticed the GPU was actually in use by efifb.

e0000000-f1ffffff : PCI Bus 0000:09
e0000000-efffffff : 0000:09:00.0
  e0000000-efffffff : vfio-pci
f0000000-f1ffffff : 0000:09:00.0
  f1000000-f12fffff : efifb

I used the echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind command to mannually unbind efifb. After that I was able to use the GPU in the VM.

 

[coen800]

Similar experience. I also try to passthrough my 3060ti in Proxmox 7.0 to my Windows10 vm. The CPU of machine is 5800x, AMD, too. I also got similar BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] and the vm hangs when I try to install latest(466.77, June 2021) Nvidia driver. Maybe we can share what we observe and see how to solve it.

You need to check if your GPU is used by some other proces with the cat /proc/iomem command. In your case the bar error is from memory address 0xd0000000-0xdfffffff so look for that.

 

[ohayo world]

You need to check if your GPU is used by some other proces with the cat /proc/iomem command. In your case the bar error is from memory address 0xd0000000-0xdfffffff so look for that.

Thanks for your reply, I've fixed my own problem, too. In my circumstence maybe the contributing factor is 1) install from 20H2 to 21H1 2) use geforce experince instead of downloading directly or 3) unplug DP cables from GPU. From your reply, I think the most probable factor is the third, which may result in some process running on the GPU.