HPE ML350 Gen10 - integrated SAS passthrue issue

nektieri

Member
Oct 14, 2020
2
0
6
37
First of all hello to everyone!

I have a problem that I'm stuck on. I will be grateful for any suggestions.
I'm trying too passthru integrates sata (AHCI) and integrated sas controller to VM (ultimately for the needs of truenas).
For testing purpose I passthru to VM with windows 11

AHCI passthrue works fine.
SAS passthrue reboots whole server when i turn on VM.

What else can I test next ?

Problematic server:
HPE ML350 Gen10 - dual intel silver 4108
Proxmox is installed on two nvme disc in mirror configuration ZFS
Test VM uses q35 chipset.

Attaching few info that may be helpful:
pveversion pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.16-3-pve)
verbose - file attached

/etc/modules
vfio vfio_iommu_type1 vfio_pci vfio_virqfd

/etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt


/etc/modprobe.d/vfio.conf
options vfio-pci ids=9005:028f,8086:a182

/etc/modprobe.d/pve-blacklist.conf
blacklist nvidiafb blacklist smartpqi blacklist ahci

updated:
update-initramfs -u -k all

AHCI
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done | grep 'group 49' IOMMU group 49 00:17.0 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09) lspci -s 00:17.0 -v 00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 09) (prog-if 01 [AHCI 1.0]) DeviceName: Embedded SATA Controller #1 Subsystem: Hewlett Packard Enterprise C620 Series Chipset Family SATA Controller [AHCI mode] Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 255, NUMA node 0, IOMMU group 49 Memory at d9d94000 (32-bit, non-prefetchable) [size=8K] Memory at d9d96000 (32-bit, non-prefetchable) [size=256] I/O ports at 2040 [size=8] I/O ports at 2048 [size=4] I/O ports at 2020 [size=32] Memory at d9d00000 (32-bit, non-prefetchable) [size=512K] Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [70] Power Management version 3 Capabilities: [a8] SATA HBA v1.0 Kernel driver in use: vfio-pci Kernel modules: ahci

SAS
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done | grep 'group 16 ' IOMMU group 16 3b:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI SAS [9005:028f] (rev 01) lspci -s 3b:00.0 -v 3b:00.0 Serial Attached SCSI controller: Adaptec Smart Storage PQI SAS (rev 01) DeviceName: Embedded RAID 1 Subsystem: Hewlett-Packard Company Smart Array P408i-a SR Gen10 Flags: bus master, fast devsel, latency 0, IRQ 255, NUMA node 0, IOMMU group 16 Memory at e4400000 (64-bit, non-prefetchable) [size=32K] I/O ports at 6000 [size=256] Capabilities: [80] Power Management version 3 Capabilities: [b0] MSI-X: Enable- Count=64 Masked- Capabilities: [c0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [300] Secondary PCI Express Kernel driver in use: vfio-pci Kernel modules: smartpqi

Rest attached as a files - (post symbol limit)
 

Attachments

  • iommu groups.txt
    51.9 KB · Views: 0
  • pveversion --verbose.txt
    1.3 KB · Views: 0
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!