Xeon D-1528 on-board SATA passthrough failure

[Anthony VH]

I'm trying to pass through the on-board SATA controller on a Supermicro X10SDV-6C-TLN4F, which has a Xeon D-1528.

Passing through the SATA controller to a VM is possible, the VM sees the attached drives, but actually mounting/formatting/repartitioning/... them results in boatloads of errors. It seems those kinds of errors are due to "incorrect" passthrough. As far as I can see, this is due to the SATA controller being in the same IOMMU group as 3 other devices which I'm not passing through (ISA bridge, SMBus controller, and some thermal management controller).

Google searches seem to indicate this passthrough is be possible, at least on ESXi, so I can't imagine it's not on Proxmox. I would link to some forum topics and blog posts, but the forum won't let me as a new user. Searching for "esxi x10sdv passthrough sata" gave me 3 matches as the first results.

Proxmox boots from a NVMe SSD in the M.2 slot, so no issues there. (The onboard SATA definitely can't be passed through if the M.2 slot contains a SATA SSD.)

Here's the settings I'm using:

• BIOS: version 2.0a (newest one available on Supermicro's website)
  AFAIK all virtualization settings are enabled (i.e. VT-d, ACS, interrupt remapping). In fact, they were by default.
• Proxmox configuration:
  • /etc/default/grub:

    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

    Adding "pci_acs_override=downstream" didn't improve matters.
  • /etc/modules:
    

    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd

  • /etc/modprobe.d/vfio.conf:

    options vfio-pci ids=8086:8c02

  • Problematic IOMMU group:
    

    IOMMU Group 22 00:1f.0 ISA bridge [0601]: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller [8086:8c54] (rev 05)
    IOMMU Group 22 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
    IOMMU Group 22 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
    IOMMU Group 22 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05)

Does anyone have a clue on how to get this to work?

 

[wolfgang]

Hi,

As you see the IOMMU Group is not isolated and you can't eject the

Series Chipset Family Server Standard SKU LPC Controller

So this is why this will not work.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

 

[Anthony VH]

Hm, that's a shame.

I'm a little confused by the various messages I find online about people pulling this exact thing off with ESXi. The fact that it works there seems to indicate to me that it's due to some software issue that this can't (currently) be done with Proxmox.

 

[wolfgang]

You can't compare VMware with Proxmox VE.

 

[Anthony VH]

I guess this is kind of derailing the thread, but... Why can't I compare? From a user point of view both are supposed to act as a level 1 hypervisor. Both have access to exactly the same hardware. I'm assuming both use VT-x/VT-d/ACS/... to provide pass-through of physical hardware. So the fact that this SATA controller can be passed through in ESXi, but can't in Proxmox/KVM, seems to hint at a software issue in KVM to me. Clearly the underlying hardware is not at fault here, since ESXi can make it work...

 

[wolfgang]

To keep it generall.
Some Hardware Vendors do not tell the opensource community everything.
Proprietary Software Vendors have contracts to get this information,
but they have to keep this information secret what is no problem because the sourcecode is not open.