[SOLVED] Can't passgtrough PCIe to PCI brigde of sound card

[Jungledede]

Hi,

I'am new here, i hope you can help me/ give hint to be able to use a PCIe sound card inside a windows VM.

Host machine:
HP Proliant 380 Gen 9
CPU(s) 48 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (2 Sockets)
Kernel Version Linux 5.11.22-3-pve #1 SMP PVE 5.11.22-5 (Wed, 21 Jul 2021 18:25:44 +0200)
PVE Manager Version pve-manager/7.0-10/d2f465d3

Guest:
Windows 10 Ent 2019
UEFI mode

The sound card i want to passthrough:
https://www.esi-audio.com/products/maya44ex/
the sound chip is PCI and it is behind a Texas instrument pcie to pci bridge.

As of today i can only passtrhough the sound chip and not pcie bridge. In the windows guest i have the driver for the board well installed, but when i want to play sound, nothing output. It is like the audio buffer is not fil, DMA not working, etc..


On the proxmox i found that the pcie bridge is in the same IOMMU group.
I put the device ID in the list to use the vfio-pci kernel driver, but on lspci, no driver at all show for this bridge.
If i force the PCIe bridge adress in the VM file, it crash.
The PCIe bridge is not showing in the list of pci device i can pass in the VM when using the proxmox web interface.

[fixed]04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
05:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
06:00.0 Multimedia audio controller: Device 19fe:7000 (rev 03)
08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)[/fixed]


Some data:

root@proxmox:~# lspci -vv
05:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge (prog-if 00 [Normal decode])
        Physical Slot: 1
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        NUMA node: 0
        IOMMU group: 38
        Bus: primary=05, secondary=06, subordinate=06, sec-latency=64
        I/O behind bridge: [disabled]
        Memory behind bridge: 93700000-937fffff [size=1M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Subsystem: Device 0000:0000
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
                Bridge: PM- B3+
        Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) PCI-Express to PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
[ICODE][/ICODE]
06:00.0 Multimedia audio controller: Device 19fe:7000 (rev 03)
        Subsystem: Device 0e51:0003
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        NUMA node: 0
        IOMMU group: 38
        Region 0: Memory at 93700000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D3 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: vfio-pci

To forcce the vfio driver i got IDs with this:

root@proxmox:~# lspci -nnk
05:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
06:00.0 Multimedia audio controller [0401]: Device [19fe:7000] (rev 03)
        Subsystem: Device [0e51:0003]
        Kernel driver in use: vfio-pci

I put the 3 ids in the file:

root@proxmox:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=104c:8240,19fe:7000,0e51:0003

But as we can see on the first lspci -vv it seems the pcie bridge don't use this. How i can be sure that the host is not using it or have reserved some memory/locks ?

The config file of the VM:

agent: 1
balloon: 8192
bios: ovmf
boot: order=sata0;ide2;net0
cores: 8
cpu: host
efidisk0: data2:vm-101-disk-1,size=4M
hostpci0: 0000:05:00,pcie=1
ide2: local:iso/virtio-win-0.1.190.iso,media=cdrom,size=489986K
machine: q35
memory: 16384
name: VM-SRV-RADIO
net0: e1000=3A:B5:97:6A:FE:D5,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
parent: FreshInstall
sata0: data2:vm-101-disk-0,size=120G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=c98fac8d-913a-4d56-88c2-5eb9c72c037b
sockets: 1
startup: order=2,up=120
vga: vmware,memory=512
vmgenid: 390e27cd-9ebb-4606-894d-c800058d60c5
[FreshInstall]
agent: 1
balloon: 8192
bios: ovmf
boot: order=sata0;ide2;net0
cores: 6
efidisk0: data2:vm-101-disk-1,size=4M
ide2: local:iso/virtio-win-0.1.190.iso,media=cdrom,size=489986K
memory: 16384
name: VM-SRV-RADIO
net0: e1000=3A:B5:97:6A:FE:D5,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win10
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-i440fx-5.1+pve0
sata0: data2:vm-101-disk-0,size=120G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=c98fac8d-913a-4d56-88c2-5eb9c72c037b
snaptime: 1622994090
sockets: 1
vga: vmware,memory=512
vmgenid: 390e27cd-9ebb-4606-894d-c800058d60c5
vmstate: data2:vm-101-state-FreshInstall

if i try to start the VM, quemu yell at me:

root@proxmox:~# qm start 101
kvm: -device vfio-pci,host=0000:05:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:05:00.0: error getting device from group 38: No such device
Verify all devices in group 38 are bound to vfio-<bus> or pci-stub and not already in use
start failed: QEMU exited with code 1

But the PCIe bridge is in the IOMMU group 38 if we look at the lscpi outputs, but i can't get to bound it to vfio drivers.

Any hint ?

The idea to use this board instead of more "classical" PC/gamer board is to use the ASIO driver which allow low latency.

 

[leesteken]

PCI(e) bridges should be handled automatically by VFIO. You should not pass them explicitly nor bind them to vfio-pci (which is not intended for bridges). If this worked in the past, it was probably a mistake. I could be wrong ofcourse; do you have any links to information that states that passing the bridge is necessary?
What did you change today? Maybe the fix for your issue is in another part of the configuration.

 

[Jungledede]

Hi,

It was an idea of mine, and by answer not the best, lol


If i put the address 00:06 in the VM config, the VM boot fine. In Windows i was able to install the driver of the soundcard. But i am not able to play any sound on any output.
For example the Windows Media Player is "stuck", the time do not move.


If i use radio automation software with ASIO driver mode, it yell at me.



The drivers seems fine.

I have try to use the q53 or the defat i440 setting of the VM, check or not the PCIe box, it change nothing.

 

[leesteken]

Please let us know what you changed between it working and it no longer working.

 

[Jungledede]

it never worked.
If i pass the PCI address (so 00:06), the driver is able to install, and windows see the board well, but no sound is played/output.
If i force the PCIe bridge address (00:05) the VM won't start.

But I am not sure where it don't work. It is a pass-through issue or a driver issue?
I don't have test the board yet in a bare metal computer.

I though that if i pass the PCIe bridge instead of the PCI device it will be better in case it is a driver issue (if the driver miss some interrupt/buffer access only done on the pcie bridge side).

but with your first post on this topic, am not sure i will be able to use this board on the VM :/

 

[leesteken]

If the device is visible inside the VM then PCI passthrough is working. If the device cannot be used by the driver, it is sometimes a passthrough configuration thing like the rombar setting or the romfile setting, or the device reset is not working correctly.
Or it is a driver issue, like it does not expect the device to be in the state that it is in after a reset. Sometimes this can be resolved by making sure no Linux driver initializes the device during the boot of the Proxmox host: blacklist the driver or bind the device to vfio-pci early.
Could you try the device in a Linux VM? You could just boot a Ubuntu 21.04 installer ISO in the VM and check whether you see the device with lspci -k and if you have sound.
Can you show a lspci -knnvv of the audio device (not the bridge) (and check for FLReset+ or FLReset-)?

 

[Jungledede]

inside an ubuntu 21.04 VM:

serveur@VM-TEST:~$ lspci -k
01:00.0 Multimedia audio controller: Device 19fe:7000 (rev 03)
        Subsystem: Device 0e51:0003

It is our guy. The ids are the same than on the host side.

serveur@VM-TEST:~$ sudo lspci -knnvv
01:00.0 Multimedia audio controller [0401]: Device [19fe:7000] (rev 03)
        Subsystem: Device [0e51:0003]
        Physical Slot: 0
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 10
        Region 0: Memory at fe800000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

No kernel used here.

and for example to ouput for the audio interface of the ich9 (for spice this one ? i don't ask to have it anyway)

00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 45
        Region 0: Memory at fea10000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee03000  Data: 0028
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

and the same on the host side

06:00.0 Multimedia audio controller [0401]: Device [19fe:7000] (rev 03)
        Subsystem: Device [0e51:0003]
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        NUMA node: 0
        IOMMU group: 38
        Region 0: Memory at 93700000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: vfio-pci

unfortunately, no driver for linux available :/

 

[leesteken]

I found others that have similar problems using unRAID and (plain) Linux. Maybe the lack of Linux support/driver is causing it to not reset properly.
Have you tried adding vfio-pci.ids=19fe:7000 to the kernel boot parameters (GRUB or systemd) to make sure nothing touches the device during boot? Maybe it will then work for a Windows VM once (but probably not after a shutdown/restart of the VM).
As PCI(e) passthrough is not a common use case, many devices/drivers don't support it and manufacturers do not test for or support it. Maybe someone else (here or on the internet) has experience with this device and/or ideas how to get it to work?

 

[Jungledede]

I give up.

I think, like you said, it an hardware incompatibility. I have plug an USB audio crappy sound card and this works just fine.

The dev of the automation software have warn few other people that DAW (digital audio workstation) in VM in not a good idea. I though that PCI-passthrough will help me, but it was not plan that it will be so difficult.
I will stay on bare metal for this machine.

Anyway, thank a lot to you. I learn lot of things since i do this only at home for hobby.