PVE 7.2 Failed to mmap BAR 3. Performance may be slow

[mailo95]

Hello,

I have small PVE homelab for personal/testing env. I have a strange problem with PCI-passthrough to Win10 VM when trying to start the VM get this error

kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: Failed to mmap 0000:01:00.0 BAR 3. Performance may be slow

. Strange is that I already have one VM with successfully passed other GPU without any problems. Will post all configs, logs and host hardware details.
Host hardware:
CPU: FX8300
MB: Asrock 990FX Extreme3
RAM: 24GB DDR3
GPU1: Nvidia 210 1st PCI slot (trying to pass to a VM and get error from above)
GPU2: HD6870 2nd PCI slot (successfully passed to a VM and working without any problems)

dmesg | grep -e DMAR -e IOMMU:
[    0.131134] AGP: Please enable the IOMMU option in the BIOS setup
[    0.637604] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.35-1-pve root=/dev/mapper/pve-root ro quiet iommu=pt amd_iommu=on nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off

cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

wireguard
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# Generated by sensors-detect on Fri Apr 29 12:04:18 2022
# Chip drivers
nct6775

dmesg | grep 'remapping'
[    0.637606] AMD-Vi: Interrupt remapping enabled

find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:04:00.1
/sys/kernel/iommu_groups/17/devices/0000:04:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:13.2
/sys/kernel/iommu_groups/7/devices/0000:00:13.0
/sys/kernel/iommu_groups/15/devices/0000:02:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:11.0
/sys/kernel/iommu_groups/13/devices/0000:00:16.2
/sys/kernel/iommu_groups/13/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:0a.0
/sys/kernel/iommu_groups/11/devices/0000:00:14.4
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.0
/sys/kernel/iommu_groups/16/devices/0000:03:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:12.2
/sys/kernel/iommu_groups/6/devices/0000:00:12.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.1
/sys/kernel/iommu_groups/4/devices/0000:00:0b.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.5
/sys/kernel/iommu_groups/2/devices/0000:00:05.0
/sys/kernel/iommu_groups/10/devices/0000:00:14.3
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:14.2

IOMMU group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14] (rev 02)
IOMMU group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
IOMMU group 11 00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
IOMMU group 12 00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
IOMMU group 13 00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 13 00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 14 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
IOMMU group 14 01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
IOMMU group 15 02:00.0 USB controller [0c03]: Etron Technology, Inc. EJ168 USB 3.0 Host Controller [1b6f:7023] (rev 01)
IOMMU group 16 03:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetLink BCM57781 Gigabit Ethernet PCIe [14e4:16b1] (rev 10)
IOMMU group 17 04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] [1002:6738]
IOMMU group 17 04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6790/6850/6870 / 7720 OEM] [1002:aa88]
IOMMU group 1 00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) [1002:5a16]
IOMMU group 2 00:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 1) [1002:5a19]
IOMMU group 3 00:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 5) [1002:5a1d]
IOMMU group 4 00:0b.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD990 PCI to PCI bridge (PCI Express GFX2 port 0) [1002:5a1f]
IOMMU group 5 00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
IOMMU group 6 00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 6 00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 7 00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 7 00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
IOMMU group 9 00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

/etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist radeon

/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:6738,1002:aa88 disable_vga=1 (GPU2)
options vfio-pci ids=10de:0a65,10de:0be3 disable_vga=1 (GPU1)

/etc/pve/qemu-server/101.conf
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 2
cpu: host,hidden=1
cpulimit: 2
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=6.2.0,ctime=1653742475
name: TestVM
net0: virtio=EA:78:CB:34:6C:19,bridge=vmbr0
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-1,cache=writeback,discard=on,size=50G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=dfd39b63-6b13-428a-a979-20eaf4475cd3
sockets: 1
vcpus: 2
vmgenid: 01f74ddf-5ff1-4904-9ffd-9860ee595937

Used these tutorials:
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/
https://pve.proxmox.com/wiki/Pci_passthrough

If there is another info that I can provide just ping me. Will be grateful if someone help me with this issue !
Thanks in advance.