no HDMI audio out after GPU and Audio passthrough

[udon]

I read a lot tutorial of gpu passthrough, and I did it in proxmox 8.HDMI did show what shows in the vm. But no matter what I try, there is no audio output.
Here is what i did:

efibootmgr -v

Boot0010* proxmox HD(2,GPT,0d01c215-bceb-4aad-9afc-165f4c3bed79,0x800,0x200000)/File(\EFI\PROXMOX\GRUBX64.EFI)
Which means bootloader is GRUB.

nano /etc/default/grub

change GRUB_CMDLINE_LINUX_DEFAULT="quiet" to this
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,igfx_off iommu=pt nomodeset initcall_blacklist=sysfb_init"

update-grub

And reboot.

dmesg | grep -i -e DMAR -e IOMMU

[ 0.095212] DMAR: IOMMU enabled [ 2.604266] DMAR: Intel(R) Virtualization Technology for Directed I/O
IOMMU enabled, good to go.

echo "vfio" >> /etc/modules
echo "vfio_iommu_type1" >> /etc/modules
echo "vfio_pci" >> /etc/modules
update-initramfs -u -k all

And reboot.

dmesg | grep -i vfio

[ 6.118244] VFIO - User Level meta-driver version: 0.3

dmesg | grep  'remapping'

[ 0.292240] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.294725] DMAR-IR: Enabled IRQ remapping in x2apic mode

lspci -nnk | grep VGA

00:02.0 VGA compatible controller [0300]: Intel Corporation GeminiLake [UHD Graphics 600] [8086:3185] (rev 06)

lspci -nnk | grep Audio

00:0e.0 Audio device [0403]: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio [8086:3198] (rev 06)

find /sys/kernel/iommu_groups/ -type l

Spoiler

/sys/kernel/iommu_groups/17/devices/0000:01:00.0 /sys/kernel/iommu_groups/7/devices/0000:00:13.2 /sys/kernel/iommu_groups/15/devices/0000:00:1e.0 /sys/kernel/iommu_groups/5/devices/0000:00:12.0 /sys/kernel/iommu_groups/13/devices/0000:00:19.2 /sys/kernel/iommu_groups/13/devices/0000:00:19.0 /sys/kernel/iommu_groups/13/devices/0000:00:19.1 /sys/kernel/iommu_groups/3/devices/0000:00:0e.0 /sys/kernel/iommu_groups/11/devices/0000:00:17.0 /sys/kernel/iommu_groups/11/devices/0000:00:17.3 /sys/kernel/iommu_groups/11/devices/0000:00:17.1 /sys/kernel/iommu_groups/11/devices/0000:00:17.2 /sys/kernel/iommu_groups/1/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:00.1 /sys/kernel/iommu_groups/18/devices/0000:02:00.0 /sys/kernel/iommu_groups/8/devices/0000:00:13.3 /sys/kernel/iommu_groups/16/devices/0000:00:1f.0 /sys/kernel/iommu_groups/16/devices/0000:00:1f.1 /sys/kernel/iommu_groups/6/devices/0000:00:13.0 /sys/kernel/iommu_groups/14/devices/0000:00:1c.0 /sys/kernel/iommu_groups/4/devices/0000:00:0f.0 /sys/kernel/iommu_groups/12/devices/0000:00:18.3 /sys/kernel/iommu_groups/12/devices/0000:00:18.1 /sys/kernel/iommu_groups/12/devices/0000:00:18.2 /sys/kernel/iommu_groups/12/devices/0000:00:18.0 /sys/kernel/iommu_groups/2/devices/0000:00:0c.0 /sys/kernel/iommu_groups/10/devices/0000:00:16.2 /sys/kernel/iommu_groups/10/devices/0000:00:16.0 /sys/kernel/iommu_groups/10/devices/0000:00:16.3 /sys/kernel/iommu_groups/10/devices/0000:00:16.1 /sys/kernel/iommu_groups/0/devices/0000:00:02.0 /sys/kernel/iommu_groups/19/devices/0000:03:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:15.0

pvesh get /nodes/home/hardware/pci --pci-class-blacklist ""

0x030000 │ 0x3185 │ 0000:00:02.0 │ 0 │ 0x8086 │ GeminiLake [UHD Graphics 600] 0x040300 │ 0x3198 │ 0000:00:0e.0 │ 3 │ 0x8086 │ Celeron/Pentium Silver Processor High Definition Audio
Everything indicate GPU and audio are in their own IOMMU group.

echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
echo "softdep snd_hda_intel pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep snd_hda_codec_hdmi pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep i915 pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "options vfio-pci ids=8086:3185,8086:3198 disable_vga=1" >> /etc/modprobe.d/vfio.conf
update-initramfs -u -k all

And reboot.
Then I frist created a debian vm without GPU passthrough.

Spoiler

balloon: 0 bios: ovmf boot: order=scsi0;ide2;net0 cores: 4 cpu: x86-64-v2-AES efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M ide2: local:iso/debian-12.1.0-amd64-DVD-1.iso,media=cdrom,size=3900480K machine: q35 memory: 4096 meta: creation-qemu=8.0.2,ctime=1696232110 name: debianDesktop net0: virtio=D2:E1:64:AD:76:7A,bridge=vmbr0 numa: 0 ostype: l26 runningcpu: qemu64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3 runningmachine: pc-q35-8.0+pve0 scsi0: local-lvm:vm-100-disk-1,iothread=1,size=64G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=eb33622b-216b-48cb-849c-aa71a40741f4 snaptime: 1696240734 sockets: 1 vmgenid: 5fc4389c-e827-46dd-867a-4227036d73fa

And I installed tigervnc. Shutdown wm, and add pci device of GPU and Audio, set vga to none.

Spoiler

balloon: 0 bios: ovmf boot: order=scsi0;net0 cores: 4 cpu: x86-64-v2-AES efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M hostpci0: 0000:00:02,pcie=1,x-vga=1 hostpci1: 0000:00:0e,pcie=1 machine: q35 memory: 4096 meta: creation-qemu=8.0.2,ctime=1696232110 name: debianDesktop net0: virtio=D2:E1:64:AD:76:7A,bridge=vmbr0 numa: 0 ostype: l26 parent: done scsi0: local-lvm:vm-100-disk-1,iothread=1,size=64G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=eb33622b-216b-48cb-849c-aa71a40741f4 sockets: 1 usb0: host=1915:1011 vga: none vmgenid: 696bfc21-52a2-4a43-8670-88a1bf6e933d

Start the vm, and monitor display the output of debian vm, and I knew GPU passthough successed.
And I opened a video, video is playing, but no sound.
Then I follow https://pve.proxmox.com/wiki/PCI_Passthrough#Troubleshooting. I did it in debian vm.

echo "options snd-hda-intel enable_msi=1" >> /etc/modprobe.d/snd-hda-intel.conf
update-initramfs -u -k all
systemctl reboot

Still no sound.

lspci -vv

Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
But Audio device MSI is disabled. And I don't know what to do now.

 

[BobhWasatch]

I think you might be over-thinking this. Most desktop environments use PulseAudio these days, not Alsa directly. If you have the qemu sound device enabled it will be the default, not the HDMI audio. So have you tried to use the desktop environment's mixer applet or a tool like pavuctl to select the right output?

That would be the first thing I would check.

 

[udon]

I think you might be over-thinking this. Most desktop environments use PulseAudio these days, not Alsa directly. If you have the qemu sound device enabled it will be the default, not the HDMI audio. So have you tried to use the desktop environment's mixer applet or a tool like pavuctl to select the right output?

That would be the first thing I would check.

Never mind, after another day trying, there is no sound come from HDMI, I gave up. There's a warning in vm dmesg "snd_hda_intel: no codec found". And I can't bring the sound from HDMI from all measures I found in the Internet.
But I find there is sound coming from 3.5mm audio port, I just need to buy a cable to connect my pc to monitor.

 

[udon]

I think you might be over-thinking this. Most desktop environments use PulseAudio these days, not Alsa directly. If you have the qemu sound device enabled it will be the default, not the HDMI audio. So have you tried to use the desktop environment's mixer applet or a tool like pavuctl to select the right output?

That would be the first thing I would check.

Hi, sorry to bother you again. I managed to spin up a kubuntu 18.04 vm to output audio with my monitor speaker with setting Digital stereo (HDMI) Output in audio setting without other configuration. But not 20.04 22.04 23.04 or 23.10. So it may be a kernel problem. But all 5 versions of kubuntu can output audio with monitor speaker on bare metal without problem. So I don't know where truely the problem is.
I have never mess up with linux kernel before. Although I try to google how to debug myself, but all the "beginner debug tutorial" I find is not for beginner, they are debugging kernel they complied. So I am sorry I have to just report what I find. And I did serach what shows in dmesg, but everything I tried not working.
I do want the problem to be solvesd because ubuntu 18.04 is out of long term support now. Although there is ubuntu pro 18.04 still under long term support, but Gnome eats too much RAM. It is just a 16GB RAM pc need to be a htpc and nas and server and router.
My host is a mini pc - KODLIX GD41, CPU is Intel Celeron N4120. Proxmox version is 8.0.3. Kernel version is Linux 6.2.16-3-pve.

OS version	kernel version	hdmi sound on bare metal	hdmi sound on proxmox 8	other
Kubuntu18.04	5.4.0-42-generic	yes	yes	5.4.0-152-generic has sound too
Kubuntu20.04	5.15.0-67-generic	yes	no
Kubuntu22.04	6.2.0-26-generic	yes	no
Kubuntu23.04	6.2.0-20-generic	yes	no
Kubuntu23.10	6.5.0-9-generic	yes	no

Attach files are dmesg message from vm.
And all the vm has the same setting like below, only thing diffrent is ISO they used.

Spoiler

balloon: 0 bios: ovmf boot: order=scsi0;net0 cores: 4 cpu: host efidisk0: local-lvm:vm-121-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M hostpci0: 0000:00:02,pcie=1,x-vga=1 hostpci1: 0000:00:0e,pcie=1 machine: q35 memory: 4096 meta: creation-qemu=8.0.2,ctime=1697639277 name: 26k2204 net0: virtio=72:A9:FD:A4:67:70,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: local-lvm:vm-121-disk-1,iothread=1,size=32G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=b0fcf284-234b-4041-a055-c6e9ed0c5de9 sockets: 1 usb0: host=1915:1011 vga: none vmgenid: 890c6741-7615-4add-bdec-a5a5d6ee7799

 

[HariSeldon]

Exactly same problem here.
Fresh server.
i5 12400
MSI PRO B760M

Everything working good except audio device.

I did exactly your same steps.

 

[HariSeldon]

Hi, sorry to bother you again. I managed to spin up a kubuntu 18.04 vm to output audio with my monitor speaker with setting Digital stereo (HDMI) Output in audio setting without other configuration. But not 20.04 22.04 23.04 or 23.10. So it may be a kernel problem. But all 5 versions of kubuntu can output audio with monitor speaker on bare metal without problem. So I don't know where truely the problem is.
I have never mess up with linux kernel before. Although I try to google how to debug myself, but all the "beginner debug tutorial" I find is not for beginner, they are debugging kernel they complied. So I am sorry I have to just report what I find. And I did serach what shows in dmesg, but everything I tried not working.
I do want the problem to be solvesd because ubuntu 18.04 is out of long term support now. Although there is ubuntu pro 18.04 still under long term support, but Gnome eats too much RAM. It is just a 16GB RAM pc need to be a htpc and nas and server and router.
My host is a mini pc - KODLIX GD41, CPU is Intel Celeron N4120. Proxmox version is 8.0.3. Kernel version is Linux 6.2.16-3-pve.

OS version	kernel version	hdmi sound on bare metal	hdmi sound on proxmox 8	other
Kubuntu18.04	5.4.0-42-generic	yes	yes	5.4.0-152-generic has sound too
Kubuntu20.04	5.15.0-67-generic	yes	no
Kubuntu22.04	6.2.0-26-generic	yes	no
Kubuntu23.04	6.2.0-20-generic	yes	no
Kubuntu23.10	6.5.0-9-generic	yes	no

Attach files are dmesg message from vm.
And all the vm has the same setting like below, only thing diffrent is ISO they used.

Spoiler

balloon: 0 bios: ovmf boot: order=scsi0;net0 cores: 4 cpu: host efidisk0: local-lvm:vm-121-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M hostpci0: 0000:00:02,pcie=1,x-vga=1 hostpci1: 0000:00:0e,pcie=1 machine: q35 memory: 4096 meta: creation-qemu=8.0.2,ctime=1697639277 name: 26k2204 net0: virtio=72:A9:FD:A4:67:70,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: local-lvm:vm-121-disk-1,iothread=1,size=32G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=b0fcf284-234b-4041-a055-c6e9ed0c5de9 sockets: 1 usb0: host=1915:1011 vga: none vmgenid: 890c6741-7615-4add-bdec-a5a5d6ee7799

I make it work at the end! Just by trial and errors!

The trick is to configure:
UEFI
i440FX

with Q35 hdmi sound doesn't work!

 

[leesteken]

with Q35 hdmi sound doesn't work!

Thank you for sharing the solution. q35 comes with a virtual (non-functional?) audio device, because the original had one. Have you tried binding it to pci-stub (pci_stub.ids=8086:293e for example) to practically disable it and test if that also fixed the issue?

 

[HariSeldon]

Thank you for sharing the solution. q35 comes with a virtual (non-functional?) audio device, because the original had one. Have you tried binding it to pci-stub (pci_stub.ids=8086:293e for example) to practically disable it and test if that also fixed the issue?

Can you explain a bit better? If I bind the audio physical device to pci stub it will not work...no? Or you mean from within the VM (and not the host) the virtual audio device? Can you please explain a bit more in detail?

 

[leesteken]

Can you explain a bit better? If I bind the audio physical device to pci stub it will not work...no? Or you mean from within the VM (and not the host) the virtual audio device? Can you please explain a bit more in detail?

Indeed the "virtual Q35 motherboard" comes with a virtual audio device (lspci -nn shows 8086:293e numeric ID), This virtual device (NOT your passed through real physical audio device) is seen as one of the audio devices. Since you have multiple audio devices (unless you make the virtual one unusable), maybe the software chooses the wrong one. That's why I think that it might help to disable the virtual fake audio device, to force the use of your real physical passed through audio device.

 

[udon]

Indeed the "virtual Q35 motherboard" comes with a virtual audio device (lspci -nn shows 8086:293e numeric ID), This virtual device (NOT your passed through real physical audio device) is seen as one of the audio devices. Since you have multiple audio devices (unless you make the virtual one unusable), maybe the software chooses the wrong one. That's why I think that it might help to disable the virtual fake audio device, to force the use of your real physical passed through audio device.

It's not the problem, the problem is the VM can't find the codec for passthroughed audio device. Only Ubuntu 18.04 find it. Other versions don't even show the passthroughed audio device in pulseaudio. Even there are codecs showing in /proc/asound/card*, but when cat it, the VM will crush.