Hi,
I did replace my GPU from an old AMD R9 285 to a used 1080ti.
There are two "gaming" guest VMs in my system that utilize GPU passthrough:
Initially the Windows VM worked with the new gpu and the latest nvidia drivers but I always had crackling audio appearing after some time typically while gaming or doing some video watching on YT or similar.
Sometimes the crackling was not too heavy, sometimes it was so heavily distorted that i had to reboot the machine just to find myself in the same situation again after some time.
While I tried with https://pve.proxmox.com/wiki/PCI_Passthrough#HDMI_audio_crackling.2Fbroken from the wiki using msi utility tool in the windows vm didn't help too much, it seems for now that adding mitigations=off in /etc/default/grub of the proxmox host did the trick for the windows VM.
Within the ubuntu VM, the issue is way worse. Not only do you have a crackling audio, but playing simple yt video stutters heavily both in visual as well as audio so that you can't watch it. Even the simple sound output test ( Frontleft/Frontright within the setting sections of ubuntu) doesn't work properly (stutters). It doesn't matter if i use noveau driver in the vm or one from nvidia (535 i tried). For the noveau driver there is sometimes even a constant "knocking" sound coming from the speakers.
The suggested fix from the wiki i cant utilize, as it seems that MSI is enabled. This is the output of lspci -vv on the guest ubuntu:
At this time i am also not sure if my whole configuration is correct after all, as i fiddled it together from a lot of different tutorials.
Below is what i hope is useful. Any other log/config i can of course post.
Any idea how to fix my issue? Thanks in advance
My System
Hardware
Host Configuration
Question here: Do I actually need to add the vfio-pci.ids here??
Question: These pci ids are still of the old gpu, the new ones differ, lspci -n -s 06:00 give me:
VM Configuration
I did replace my GPU from an old AMD R9 285 to a used 1080ti.
There are two "gaming" guest VMs in my system that utilize GPU passthrough:
- Windows 10
- ubuntu 20.04
Initially the Windows VM worked with the new gpu and the latest nvidia drivers but I always had crackling audio appearing after some time typically while gaming or doing some video watching on YT or similar.
Sometimes the crackling was not too heavy, sometimes it was so heavily distorted that i had to reboot the machine just to find myself in the same situation again after some time.
While I tried with https://pve.proxmox.com/wiki/PCI_Passthrough#HDMI_audio_crackling.2Fbroken from the wiki using msi utility tool in the windows vm didn't help too much, it seems for now that adding mitigations=off in /etc/default/grub of the proxmox host did the trick for the windows VM.
Within the ubuntu VM, the issue is way worse. Not only do you have a crackling audio, but playing simple yt video stutters heavily both in visual as well as audio so that you can't watch it. Even the simple sound output test ( Frontleft/Frontright within the setting sections of ubuntu) doesn't work properly (stutters). It doesn't matter if i use noveau driver in the vm or one from nvidia (535 i tried). For the noveau driver there is sometimes even a constant "knocking" sound coming from the speakers.
The suggested fix from the wiki i cant utilize, as it seems that MSI is enabled. This is the output of lspci -vv on the guest ubuntu:
Code:
01:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GP102 HDMI Audio Controller
Physical Slot: 0
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 17
Region 0: Memory at c1000000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000At this time i am also not sure if my whole configuration is correct after all, as i fiddled it together from a lot of different tutorials.
Below is what i hope is useful. Any other log/config i can of course post.
Any idea how to fix my issue? Thanks in advance
My System
Hardware
- Asrock 890GX
- AMD Phenom 2 1090T
- 32GB RAM
- SSDs hosting the VMs
pve-manager/7.4-16/0f39f621 (running kernel: 5.15.116-1-pve)Host Configuration
- /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on mitigations=off vfio-pci.ids=1002:6939,1002:aad8,1002:6779,1002:aa98,dd01:0011"Question here: Do I actually need to add the vfio-pci.ids here??
- Iommu groups
Code:
┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────────────────────────────────┬──────┬──────────────────┬───────────────────────────┬──────────────────┬──────────────────────────
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_device │ subsystem_device_name │ subsystem_vendor │ subsystem_vendor_name
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════════╪══════════════════╪══════════════════════════
│ 0x010601 │ 0x4391 │ 0000:00:11.0 │ 7 │ 0x1002 │ SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] │ │ 0x4391 │ │ 0x1849 │ ASRock Incorporation
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x010601 │ 0x9120 │ 0000:01:00.0 │ 18 │ 0x1b4b │ 88SE9120 SATA 6Gb/s Controller │ │ 0x9120 │ │ 0x1849 │ ASRock Incorporation
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x020000 │ 0x8168 │ 0000:04:00.0 │ 21 │ 0x10ec │ RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller │ │ 0x8168 │ Motherboard (one of many) │ 0x1849 │ ASRock Incorporation
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x030000 │ 0x6779 │ 0000:05:00.0 │ 17 │ 0x1002 │ Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] │ │ 0x3203 │ │ 0x1682 │ XFX Pine Group Inc.
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x030000 │ 0x1b06 │ 0000:06:00.0 │ 22 │ 0x10de │ GP102 [GeForce GTX 1080 Ti] │ │ 0x3609 │ │ 0x1462 │ Micro-Star International
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x040300 │ 0x4383 │ 0000:00:14.2 │ 11 │ 0x1002 │ SBx00 Azalia (Intel HDA) │ │ 0x2892 │ │ 0x1849 │ ASRock Incorporation
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x040300 │ 0xaa98 │ 0000:05:00.1 │ 17 │ 0x1002 │ Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] │ │ 0xaa98 │ │ 0x1682 │ XFX Pine Group Inc.
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x040300 │ 0x10ef │ 0000:06:00.1 │ 22 │ 0x10de │ GP102 HDMI Audio Controller │ │ 0x3609 │ │ 0x1462 │ Micro-Star International
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x048000 │ 0x0011 │ 0000:08:00.0 │ 15 │ 0xdd01 │ Octopus CI DVB Adapter │ │ 0x0040 │ Octopus CI │ 0xdd01 │ Digital Devices GmbH
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────┼──────────────────┼──────────────────────────
│ 0x060000 │ 0x5a11 │ 0000:00:00.0 │ 0 │ 0x1002 │ RD890 Northbridge only single slot PCI-e GFX Hydra part │ │ 0x5a11 │ │ 0x1849 │ ASRock Incorporation- lspci -v for the 1080ti
Code:
06:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] GP102 [GeForce GTX 1080 Ti]
Flags: bus master, fast devsel, latency 0, IRQ 71, NUMA node 0, IOMMU group 22
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at ce000000 (64-bit, prefetchable) [size=32M]
I/O ports at ec00 [size=128]
Expansion ROM at fe800000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
06:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GP102 HDMI Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 70, NUMA node 0, IOMMU group 22
Memory at fe8fc000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel- /etc/modules
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci ids=1002:6939,1002:aad8,1002:6779,1002:aa98,dd01:0011
vfio_virqfd
Code:
06:00.0 0300: 10de:1b06 (rev a1)
06:00.1 0403: 10de:10ef (rev a1)- Blacklists
Code:
root@bigmox:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
root@bigmox:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
root@bigmox:~# cat /etc/modprobe.d/vfio_pci.conf
options vfio-pci ids=1002:6939,1002:aad8,1002:6779,1002:aa98,dd01:0011
softdep amdgpu pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pciVM Configuration
- Windows 10
Code:
balloon: 0
boot: order=ide2;ide0;net0
cores: 6
cpu: host
hostpci0: 0000:06:00,pcie=1,x-vga=1
ide0: extvmdisks:101/vm-101-disk-0.qcow2,size=250G,ssd=1
ide2: none,media=cdrom
machine: pc-q35-6.1
memory: 17000
meta: creation-qemu=6.1.0,ctime=1637058178
name: pWin10
net0: e1000=3E:4D:9B:B7:98:1B,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: gtx1080ti
scsihw: virtio-scsi-pci
smbios1: uuid=c9547027-5d60-4960-9280-8ee54a480cd7
sockets: 1
usb0: host=9-3,usb3=1
vga: none
vmgenid: 023a5571-b8b5-4e4b-b329-10d52b065e0e- Ubuntu 20.04
Code:
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host
efidisk0: extvmdisks:100/vm-100-disk-0.raw,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:06:00,pcie=1,x-vga=1
machine: q35
memory: 8192
meta: creation-qemu=6.1.1,ctime=1649171948
name: pUbuntu
net0: virtio=EE:58:A6:0B:49:A3,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: extvmdisks:100/vm-100-disk-1.qcow2,size=125G
scsihw: virtio-scsi-pci
smbios1: uuid=56892bcf-f29a-43c1-8a30-01ba620eac42
sockets: 1
usb1: host=9-2
vmgenid: dde477af-d88c-4da6-9455-bf71e4d2814b