[SOLVED] AMD-Vi IO_PAGE_FAULT (iommu issue)

P

PeterVanGrieken

New Member
Apr 18, 2022
7
4
3
Hi guys,

I've been running truenas core on my server for a while, but I'd like to virtualize this in proxmox and pass the drives for ZFS to Truenas by using PCI passthrough. But I keep running into the problem that when I start the computer, I get an error message:

ahci 0000:04:00.0: AMD-Vi: Event logged [ IO_PAGE_FAULT domain=0x0001 address=0x0 flags=0x0000]
ata7: COMRESET failed (errno=-16)
ata7: COMRESET failed (errno=-16)
ata7: COMRESET failed (errno=-16)
ata7: COMRESET failed (errno=-16)
ata7: reset failed, giving up

My config:
Asrock B550M Pro4 (updated to the latest bios 2.3)
AMD Ryzen 7 5800X
64Gb ECC ram
1x M.2 nvme as bootdrive for proxmox 7.1
4x 2Tb drives connected to the mobo SATA ports for zfs storage for proxmox
4x 3Tb drives connected to a PCIe sata controller for zfs storage for truenas (this is the card I want to forward to the Truenas VM)

In the bios IOMMU is set to enabled

In Grub I already changed the existing line below and updated grub (update-grub):
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

In /etc/modules I added the lines below and refreshed initramfs (update-initramfs -u -k all)
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I found some posts about this issue claiming to be issues in the kernel, but most of them are rather old (2016 and older) and refer to linux kernels 4 whereas I have kernel 5.13.19. Last post I found dates from 2019 (https://bugzilla.kernel.org/show_bug.cgi?id=202665) but also refers to kernel 4.20.12...

Anybody have a clue as to what I may be overlooking?
 

Attachments

  • Capture.JPG
    Capture.JPG
    631.2 KB · Views: 49
If you get the error when starting the VM, then probably the SATA controller is in the same iOMMU group as many other devices that are needed/used by the Proxmox host. This is likely because of your motherboard: only the first x16 PCIe slot and the M.42 slot (4 PCIe lanes) closest to the CPU are in their own groups. Everything else, including the other PCIe slots is part of one big group that also contains the on-board devices and SATA controller. Please see these posts of mine on a similar situation for more explanation.
 
@leesteken: I get the error as soon as I start the PC itself (not the VM). But I'm definately going to read that post carefully. I just succeeded in booting the PC with iommu enabled by swapping the nvme drive with an SSD drive (sata). Just got back up and running , imported a pool and created a VM with the 4-port PCIe sata card passed through.

I'm now looking how I can reuse the zvol with the previous VM, but I'll figure that out.

But as per your post: what worries me now is that the PCIe card I'm forwaring to the VM seems to be in the same IOMMU group as eg the network card in the PC. So I expect to run into issues there :-(.
(Note - I tested this after writing this post and confirmed the PC threw a bunch of errors.)

Command used: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done


output shows the 4-port sata controller I need is in the same group as the ethernet controller (group 0 ID 03:00.0 and 04:00.0):

Code: 
root@team1:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 0 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU group 0 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU group 0 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU group 0 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 0 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 0 03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe 2.0 x2 4-port SATA 6 Gb/s RAID Controller [1b4b:9230] (rev 11)
IOMMU group 0 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 11 06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 12 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 13 07:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 14 07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 2 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU group 2 05:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)

I tried adding another PCIe to sata card to see what effect this would have, but both SATA cards and the network controller have moved to group 13. Still in the same group (ID's 01:00.1 03:00.0 and 05:00.0).

Code: 
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 13 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU group 13 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU group 13 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU group 13 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 13 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 13 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 13 03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe 2.0 x2 4-port SATA 6 Gb/s RAID Controller [1b4b:9230] (rev 11)
IOMMU group 13 04:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller [1b4b:9128] (rev 20)
IOMMU group 13 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 14 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU group 14 06:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 15 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 16 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 17 08:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 18 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 1 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

From your post I saw you proposed 2 solutions. One is breaking up the groups or switching to an X570 board. Breaking the groups (and VM/Host isolation) doesn't sound like a good idea. But how would switching to an X570 board solve this?
 
PeterVanGrieken said:
@leesteken: I get the error as soon as I start the PC itself (not the VM).
Click to expand...
If you are getting an error about a connected SATA drive, then maybe the cable is not connected very well or has issues, or the hardware itself has problems.
I don't know about the IO_PAGE_FAULT, which appear to be about the onboard network controller. Maybe a kernel or BIOS issue? The best AMD BIOS versions are not always the latest. Although they appear to be getting better, some version just plain break passthrough or have terrible IOMMU groups. Maybe try pve-kernel-5.15 for newer drivers?
PeterVanGrieken said:
But as per your post: what worries me now is that the PCIe card I'm forwaring to the VM seems to be in the same IOMMU group as eg the network card in the PC. So I expect to run into issues there :-(.
Click to expand...
This happens a lot to people with Ryzen motherboards (except for X570).
PeterVanGrieken said:
I tried adding another PCIe to sata card to see what effect this would have, but both SATA cards and the network controller have moved to group 13. Still in the same group (ID's 01:00.1 03:00.0 and 05:00.0).
Click to expand...
Yes, because everything is always grouped into the big chipset group, except for PCIe lanes connected to the CPU (20 or 12 with APU) and sometimes a USB controller and/or an audio device.
PeterVanGrieken said:
From your post I saw you proposed 2 solutions. One is breaking up the groups or switching to an X570 board. Breaking the groups (and VM/Host isolation) doesn't sound like a good idea. But how would switching to an X570 board solve this?
Click to expand...
The X570 chipset is made/designed by AMD and it has much better IOMMU groups where almost every PCIe slot and even onboard devices are properly isolated into its own group. All other chipset are simpler (no PCIe 4.0), made/designed by another company (ASMedia?) and have poor support for passthrough.
 
leesteken said:
If you are getting an error about a connected SATA drive, then maybe the cable is not connected very well or has issues, or the hardware itself has problems.
I don't know about the IO_PAGE_FAULT, which appear to be about the onboard network controller. Maybe a kernel or BIOS issue? The best AMD BIOS versions are not always the latest. Although they appear to be getting better, some version just plain break passthrough or have terrible IOMMU groups. Maybe try pve-kernel-5.15 for newer drivers?
Click to expand...
Indeed maybe an idea to try the 5.15 kernel, but that will have to wait until at least next weekend. I have my hassio running on this server and the wife tends to get annoyed if the light stay out after sunset ;-). I'll doublecheck for any faulty connections, but I don't think this would have been the problem. I think I got the error at a moment only the bootdrve (an M.2 nvme) was connected and the error disappeared wen I switched off IOMMU support in the bios (though I'm not 100% sure - I'll have to retest this to make sure).

leesteken said:
The X570 chipset is made/designed by AMD and it has much better IOMMU groups where almost every PCIe slot and even onboard devices are properly isolated into its own group. All other chipset are simpler (no PCIe 4.0), made/designed by another company (ASMedia?) and have poor support for passthrough.
Click to expand...
Ok, that's clear. Then I prefer to opt for a new mobo using the X570 chipset. I've just been looking on the webshop of my preferred supplier and the Asrock X570S PG Riptide seems to be a good fit (X570 chipset, 3 PCIe 4.0 x16 Slots and 3 PCIe 4.0 x1 Slots).

I'll report back as soon as I have this up and running.

Thanks for the support! Really appreciated!
 
Hi leesteken,

I got a new Asrock X570 PG riptide mobo (X570 chipset) and indeed that dit the job. The IOMMU groups are hugely better than the Asrock B550M PRO4 (B550 chipst).

Code: 
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 14 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU group 15 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 16 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 17 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 17 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 17 05:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 17 05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 18 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 18 06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 19 02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 19 07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 1 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe 2.0 x2 4-port SATA 6 Gb/s RAID Controller [1b4b:9230] (rev 11)
IOMMU group 21 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Killer E3000 2.5GbE Controller [10ec:3000] (rev 06)
IOMMU group 22 08:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)
IOMMU group 23 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU group 23 09:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 24 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 25 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 26 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 27 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]

The PCIe sata controller is now in group 20 (03:00.0) and the onboard sata controller is in group 18/19 (06:00.0 and 07:00.0). All neatly separated!

The only strange thing is that the PCIe sata controller can only use 3 HDD at the same time. If I connect 4 HDD, only 3 of them show up in the bios. All HW is working properly (hdd, cables, sata controller) so I guess this might be a bug in the bios of the motherboard. But since I now have 3 sata controllers in separate IOMMU groups I was able to forward the 6 SATA ports of the motherboard to the Truenas VM and keep 2 available on the PCIe sata controller for storage of the VM's.

First tests look promising. I was able to import all the ZFS pools in the Truenas VM.

Thanks gain for the support!

Peter
 
PeterVanGrieken said:
The only strange thing is that the PCIe sata controller can only use 3 HDD at the same time. If I connect 4 HDD, only 3 of them show up in the bios.
Click to expand...

If you mean your onboard ports/controller, than have a look at your mainboard manual for something like this:
* If M2_2 is occupied by a PCIE-type M.2 device, SATA3_5_6 will be disabled.
Click to expand...

I only see one NVMe in your output; so did you install it in the second M.2-slot instead of the first?

But SATA-ports 1-4 should work anyway; so maybe check which ports exactly you have used/tried.
 
Last edited:
Neobin said:
If you mean your onboard ports/controller, than have a look at your mainboard manual for something like this:
Click to expand...
No, I mean the drives connected to the 4-port PCIe sata controller. When i connect 4 drives to the 4-port PCIe sata controller, only 3 drives show up in the bios. And there is no logic in which port works. if I connect drives to ports A/B/C/D, only A/B/C will show up. But if I disconnect eg C then A/B/D will show up in the bios. I tested all cables/drives and they are working properly.

And I'm sure the card is working properly - when I connect it to my B550 motherboard, I can use 4 drives without an issue. No idea what this doesn't work on the X570 board.

Neobin said:
I only see one NVMe in your output; so did you install it in the second M.2-slot instead of the first?
Click to expand...
I indeed used it in slot 1. Slot 2 is unused and all SATA ports on the motherboard work properly.
 
There have been reports about Ryzen SATA ports acting strange in this way. Somethings two drives work when then are connected one way, but not when you swap the ports.
 
Fyi:

Asrock reached out to me to offer support for the last issue where only 3 drives where recognized on the PCIe Sata controller. It seems this was actually due to an issue in the firmware in the PCIe card. We updated the firmware and the drives are working now in proxmox.

Card I used: Digitus DS-30104-1. Firmware is available on the Digitus website https://ftp.assmann.com/pub/DS-/DS-30104-1___4016032330240/DS-30104-1_firmware_mul_DS-30104-1%20Firmware_20181221.zip

Capture.JPG

So for proxmox and Truenas this setup works great:
- Asrock X570 PG Riptide
- AMD Ryzen 7 5800X
- 64Gb unbuffered ECC memory (ECC recommended but not mandatory for ZFS)
- Digitus DS-30104-1 PCIe-Sata controller
Both the onboard Sata controller and PCIe sata controller are in separate IOMMU groups, so I can use either one to forward to the Truenas VM.
 
  • Like
Reactions: Inten, Neobin and leesteken
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom