VGA pass issues with Radeon RX 7900 XTX | kvm: VFIO_MAP_DMA failed: Invalid argument

T

tenhi_shadow

Active Member
Dec 21, 2016
3
0
41
34
Hello,


I use proxmox with kernel:
Code: 
# uname -r
6.8.12-2-pve



I have ASUS Radeon RX 7900 XTX TUF Gaming OC 24GB GDDR6 installed in PCI slot

From proxmox I can see it as:

Code: 
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] (rev c8)
    Subsystem: ASUSTeK Computer Inc. Navi 31 [Radeon RX 7900 XT/7900 XTX/7900M]
    Kernel driver in use: vfio-pci
    Kernel modules: amdgpu

but I also checked that it has more device, but it has not been documented on proxmox wiki:
Code: 
# lspci -nnk | grep AMD
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 10)
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 10)
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]


I have configured bootloader and have runnind params like this:

Code: 
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/rootvg-root ro intel_iommu=on iommu=pt nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off pcie_acs_override=downstream,multifunction

I configured test vm like that:
Code: 
# cat /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=ide2;net0
cores: 8
cpu: host
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:03:00,pcie=1,x-vga=1
ide2: sysdata:iso/archlinux-2024.03.01-x86_64.iso,media=cdrom,size=964940K
machine: q35
memory: 10240
meta: creation-qemu=9.0.2,ctime=1729277109
name: test-vga
net0: virtio=BC:24:11:16:23:C2,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=92b9a367-04e7-4c92-8127-61cc2609bce2
sockets: 1
tpmstate0: local:100/vm-100-disk-1.raw,size=4M,version=v2.0
vga: none
vmgenid: a4eb28e4-1339-44f5-8c18-25b3fac55bc7

when I trying to start it, I see this message in journald:
Code: 
Nov 01 21:06:42 vhost01 pvedaemon[2894]: start VM 100: UPID:vhost01:00000B4E:00004BCA:672534D2:qmstart:100:root@pam:
Nov 01 21:06:42 vhost01 pvedaemon[1845]: <root@pam> starting task UPID:vhost01:00000B4E:00004BCA:672534D2:qmstart:100:root@pam:
Nov 01 21:06:42 vhost01 systemd[1]: Created slice qemu.slice - Slice /qemu.
Nov 01 21:06:42 vhost01 systemd[1]: Started 100.scope.
Nov 01 21:06:44 vhost01 kernel: vfio-pci 0000:03:00.0: enabling device (0002 -> 0003)
Nov 01 21:06:44 vhost01 kernel: vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
Nov 01 21:06:44 vhost01 pvedaemon[1845]: <root@pam> end task UPID:vhost01:00000B4E:00004BCA:672534D2:qmstart:100:root@pam: OK
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 7/KVM/3005 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 6/KVM/3004 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 4/KVM/3002 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 3/KVM/3001 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 1/KVM/2999 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 2/KVM/3000 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:44 vhost01 kernel: x86/split lock detection: #AC: CPU 5/KVM/3003 took a split_lock trap at address: 0x7eedd050
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380000000000, 0x10000000, 0x773488000000) = -22 (Invalid argument)
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380010000000, 0x200000, 0x773758000000) = -22 (Invalid argument)
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380000000000, 0x10000000, 0x773488000000) = -22 (Invalid argument)
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380010000000, 0x200000, 0x773758000000) = -22 (Invalid argument)
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380000000000, 0x10000000, 0x773488000000) = -22 (Invalid argument)
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: VFIO_MAP_DMA failed: Invalid argument
Nov 01 21:06:45 vhost01 QEMU[2912]: kvm: vfio_container_dma_map(0x5f5401412250, 0x380010000000, 0x200000, 0x773758000000) = -22 (Invalid argument)


One doubt that I have here is that my VGA is located in 19 and in 20 groups:
Code: 
# pvesh get /nodes/vhost01/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬──────────────────────────────────────────────────┬──────┬──────────────────┬─
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                      │ mdev │ subsystem_device │
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════╪══════╪══════════════════╪═
│ 0x010601 │ 0x7a62 │ 0000:00:17.0 │          7 │ 0x8086 │                                                  │      │ 0xb005           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x010802 │ 0x5419 │ 0000:04:00.0 │         21 │ 0xc0a9 │                                                  │      │ 0x4100           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x010802 │ 0x5419 │ 0000:05:00.0 │         22 │ 0xc0a9 │                                                  │      │ 0x4100           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x010802 │ 0xa80a │ 0000:07:00.0 │         23 │ 0x144d │ NVMe SSD Controller PM9A1/PM9A3/980PRO           │      │ 0xa801           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x010802 │ 0xa80a │ 0000:0b:00.0 │         27 │ 0x144d │ NVMe SSD Controller PM9A1/PM9A3/980PRO           │      │ 0xa801           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x020000 │ 0x15f3 │ 0000:09:00.0 │         25 │ 0x8086 │ Ethernet Controller I225-V                       │      │ 0xe000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x020000 │ 0x00b1 │ 0000:0a:00.0 │         26 │ 0x1d6a │ AQC100 10G Ethernet MAC controller [AQtion]      │      │ 0x874a           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x028000 │ 0x7927 │ 0000:08:00.0 │         24 │ 0x14c3 │                                                  │      │ 0xe0fd           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x030000 │ 0x744c │ 0000:03:00.0 │         19 │ 0x1002 │ Navi 31 [Radeon RX 7900 XT/7900 XTX]             │      │ 0x0506           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x040300 │ 0xab30 │ 0000:03:00.1 │         20 │ 0x1002 │                                                  │      │ 0xab30           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x050000 │ 0x7a27 │ 0000:00:14.2 │          4 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060000 │ 0xa703 │ 0000:00:00.0 │          0 │ 0x8086 │                                                  │      │ 0x5000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060100 │ 0x7a04 │ 0000:00:1f.0 │         16 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0xa70d │ 0000:00:01.0 │          1 │ 0x8086 │                                                  │      │ 0x5000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0xa74d │ 0000:00:06.0 │          2 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a48 │ 0000:00:1a.0 │          9 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a40 │ 0000:00:1b.0 │         10 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a44 │ 0000:00:1b.4 │         11 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a38 │ 0000:00:1c.0 │         12 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a3a │ 0000:00:1c.2 │         13 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a30 │ 0000:00:1d.0 │         14 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x7a34 │ 0000:00:1d.4 │         15 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x1478 │ 0000:01:00.0 │         17 │ 0x1002 │ Navi 10 XL Upstream Port of PCI Express Switch   │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x060400 │ 0x1479 │ 0000:02:00.0 │         18 │ 0x1002 │ Navi 10 XL Downstream Port of PCI Express Switch │      │ 0x1479           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x078000 │ 0x7a68 │ 0000:00:16.0 │          6 │ 0x8086 │                                                  │      │ 0x1c3a           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x088000 │ 0xa74f │ 0000:00:08.0 │          3 │ 0x8086 │ GNA Scoring Accelerator module                   │      │ 0x5000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c0330 │ 0x7a60 │ 0000:00:14.0 │          4 │ 0x8086 │                                                  │      │ 0x5007           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c0500 │ 0x7a23 │ 0000:00:1f.4 │         16 │ 0x8086 │                                                  │      │ 0x5001           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a4c │ 0000:00:15.0 │          5 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a4d │ 0000:00:15.1 │          5 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a4e │ 0000:00:15.2 │          5 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a4f │ 0000:00:15.3 │          5 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a7c │ 0000:00:19.0 │          8 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a7d │ 0000:00:19.1 │          8 │ 0x8086 │                                                  │      │ 0x0000           │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─
│ 0x0c8000 │ 0x7a24 │ 0000:00:1f.5 │         16 │ 0x8086 │                                                  │      │ 0x0000           │
└──────────┴────────┴──────────────┴────────────┴────────┴──────────────────────────────────────────────────┴──────┴──────────────────┴─


My modprobe config is:

Code: 
### /etc/modprobe.d

# cat blacklist-ansible.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist cdc_acm
blacklist amdgpu
blacklist nvidia*
blacklist i915
blacklist nvidia-gpu
blacklist nvidiafb
blacklist snd_hda_codec_hdmi
blacklist snd_hda_intel

# cat vfio.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
options vfio-pci ids=1002:1478,1002:1479,1002:1479,1002:744c,1002:ab30 disable_vga=1 # Here I added more then just two, but seems like they belong to VGA
# options vfio-pci ids=1002:744c,1002:ab30 disable_vga=1 # I checked it, also not working


Please help me to resolve this issue
 
due to post size limitation, posting pve version:

Code: 
# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.12-2-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-2
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
ceph-fuse: 18.2.4-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240813.1~deb12u1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.8
libpve-cluster-perl: 8.0.8
libpve-common-perl: 8.2.5
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.4
pve-cluster: 8.0.8
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.13-2
pve-ha-manager: 4.0.5
pve-i18n: 3.2.4
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back