Dell H310 passthrough causing VM to fail on start, "Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK" [SOLVED]

[RadicalRingtail]

im attempting to passthrough a Dell H310 to a TrueNAS VM ive created (which i tested to make sure it boots up and works properly without passing through the SAS card), but whenever i try passing through the SAS card to the VM and attempt to start it, it seems to disable (?) all PCIe devices, and the only way to fix it is by rebooting the whole system

i enabled IOMMU and have verified that all IOMMU groups show up properly, made sure its not a hardware issue by swapping out power connectors, removing the drives that were connected to the SAS card, and ive also verified that the SAS card is flashed to IT mode firmware (though im not sure if that would make a difference with this issue). i thought this might also be an issue with the boot SSDs im using due to the "Synchronize Cache(10) falled" error it was throwing, but they seem to work perfectly fine when im not trying to passthrough this SAS card, and theyre completely brand new as well

ive attached images of the output i was getting while attempting to start the VM with the SAS card passed through, as well as the VM configuration

heres my system specs:

• CPU: AMD Ryzen 5 5600
• Mobo: MSI B550m VC-WiFi
• RAM: Corrsair Vengence LP DDR4 128gb (4x 32gb) 3200mhz
• GPU: Nvidia GT 1030
• PSU: Cooler Master V 850 SFX
• Boot Drive: 2x Crucial MX500 250gb SATA SSDs (in RAID 1 configuration)
• NVM Drives: 2x Sabrent Rocket M.2 500gb SSDs
• Storage Drives: 4x WD Ultrastar WUH721414ALE6L4
• HBA: Dell H310 (flashed to IT mode firmware)
• NIC: Dell BCM57810S

please let me know if theres anymore info you would like me to provide to help troubleshoot this, thank you in advance for the help!

 

[RadicalRingtail]

adding a bit more info here that might be useful regarding my config and PCIe info

heres my /etc/kernel/cmdline config:

root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on

and also my config for /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

and as for PCIe info...

heres the output from lspci:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7
01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E18 PCIe4 NVMe Controller (rev 01)
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
04:00.0 Non-Volatile memory controller: Phison Electronics Corporation E18 PCIe4 NVMe Controller (rev 01)
26:00.0 SATA controller: ASMedia Technology Inc. ASM1064 Serial ATA Controller (rev 02)
27:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
27:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
28:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
2a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
2b:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1)
2b:00.1 Audio device: NVIDIA Corporation GP108 High Definition Audio Controller (rev a1)
2c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
2d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
2d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
2d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

then the output of lspci -s 28:00.0 -v:

28:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: Dell 6Gbps SAS HBA Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 39, IOMMU group 15
        I/O ports at e000 [disabled] [size=256]
        Memory at fc340000 (64-bit, non-prefetchable) [size=64K]
        Memory at fc300000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at fc200000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

also ranlsblk to check if all the disks i connected were showing up:

NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda       8:0    0 232.9G  0 disk
├─sda1    8:1    0  1007K  0 part
├─sda2    8:2    0     1G  0 part
└─sda3    8:3    0   231G  0 part
sdb       8:16   0 232.9G  0 disk
├─sdb1    8:17   0  1007K  0 part
├─sdb2    8:18   0     1G  0 part
└─sdb3    8:19   0   231G  0 part
sdc       8:32   0  12.7T  0 disk <  these disks here
sdd       8:48   0  12.7T  0 disk <  are connected to
sde       8:64   0  12.7T  0 disk <  the SAS card
sdf       8:80   0  12.7T  0 disk <
zd0     230:0    0     1M  0 disk
zd16    230:16   0    64G  0 disk
nvme0n1 259:0    0 465.8G  0 disk
nvme1n1 259:1    0 465.8G  0 disk

they also show up in the proxmox web UI:


so the card seems to be working in proxmox just fine, it only seems to cause issues when its passed through to a VM. im not sure what would be causing this, going to be trying a few more tests today to see if theres any possible fixes i can find

again, if anyone has any tips or helpful info, that would be much appreciated!

 

[RadicalRingtail]

also another small thing i forgot to include in the original post, i get a "unable to read tail (got 0 bytes)" error when the TrueNAS VM tries to auto start with the SAS card passed through:

 

[leesteken]

Did you check the IOMMU groups and verified that H310 is the only device in a group ( https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation )? The Ryzen B550 chipset only supports passthrough for the x16 PCIe slot (and one M.2) that is connected directly to the CPU. All other devices are in the big "chipset group" and the Proxmox host loses all devices in a group when you passthrough one device in that group. Did you test with the H310 in that x16 PCIe slot?

EDIT: See other threads for more details about Ryzen chipsets and IOMMU: https://forum.proxmox.com/search/7895384/?q=B550&t=post&c[child_nodes]=1&c[nodes][0]=16&o=date

 

[RadicalRingtail]

Did you check the IOMMU groups and verified that H310 is the only device in a group ( https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation )? The Ryzen B550 chipset only supports passthrough for the x16 PCIe slot (and one M.2) that is connected directly to the CPU. All other devices are in the big "chipset group" and the Proxmox host loses all devices in a group when you passthrough one device in that group. Did you test with the H310 in that x16 PCIe slot?

just ran the command listed there and realized the card was in IOMMU group 15 with a bunch of other things (SATA controller, NIC, etc) so i switched the card to the primary PCIe slot and it seems to be working now! thank you for your help! i suppose i didnt realize this was a thing with the B550 chipset