SATA-Controller Passthrough

N

npu

New Member
Oct 2, 2021
7
1
3
38
Hi,

I am new to Proxmox and tried to pass through a SATA controller to a VM (OMV). I am using an J5040-ITX system. This system has two SATA controllers. (Intel + ASM1062)
The Intel Controller is used for the Proxmox system and I want to pass through the ASMedia Controller to the VM.

I did load the modules required according to the documentation and also edited grub parameters accordingly.
So basically everything seems to be OK after these configuration but when I start the VM, Proxmox has severe issues. WebUI is not accessible anymore and most of the command do not work on bash anymore. i.e. reboot produces I/O errors.

I did check IOMMU groups and the SATA Controller seems be in a single group (0000:04:00.0):

Code: 
root@proxmox:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:13.2
/sys/kernel/iommu_groups/5/devices/0000:00:13.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:0f.0
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:13.3
/sys/kernel/iommu_groups/6/devices/0000:00:13.1
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:12.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:0e.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.1
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.1
/sys/kernel/iommu_groups/9/devices/0000:00:15.0


Code: 
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.008032] ACPI: DMAR 0x000000006D5A5000 0000A8 (v01 INTEL  GLK-SOC  00000003 BRXT 0100000D)
[    0.008098] ACPI: Reserving DMAR table memory at [mem 0x6d5a5000-0x6d5a50a7]
[    0.055467] DMAR: IOMMU enabled
[    0.168278] DMAR: Host address width 39
[    0.168280] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
[    0.168289] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap 1c0000c40660462 ecap 9e2ff0505e
[    0.168293] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
[    0.168300] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.168305] DMAR: RMRR base: 0x0000006d51b000 end: 0x0000006d53afff
[    0.168308] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007fffffff
[    0.168311] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
[    0.168314] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
[    0.168316] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.170161] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.639683] DMAR: No ATSR found
[    1.639686] DMAR: dmar0: Using Queued invalidation
[    1.639693] DMAR: dmar1: Using Queued invalidation
[    1.644074] DMAR: Intel(R) Virtualization Technology for Directed I/O

Do you have any ideas on that?
 
Can you run this command to get a nice overview of your groups for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done?
It does sound that the Proxmox host losses access to other devices such as drives and network. which suggest you are passing through a devices that shares a group with those devices. DId you use pcie_acs_override (which makes the command above useless) to break the IOMMU groups?
Can you show us your VM configuration file from /etc/pve/qemu-server/?
 
  • Like
Reactions: npu
SATA Controller seems to be in a single group 14 without any other devices. Ethernet controllers and the system SATA controller are in different groups.

Code: 
root@proxmox:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done 
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Gemini Lake Host Bridge [8086:31f0] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Gemini Lake Host Bridge [1849:0000] 
IOMMU group 0 00:00.1 Signal processing controller [1180]: Intel Corporation Celeron/Pentium Silver Processor Dynamic Platform and Thermal Framework Processor Participant [8086:318c] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Celeron/Pentium Silver Processor Dynamic Platform and Thermal Framework Processor Participant [1849:318c] 
    Kernel driver in use: proc_thermal 
    Kernel modules: processor_thermal_device 
IOMMU group 10 00:1f.0 ISA bridge [0601]: Intel Corporation Celeron/Pentium Silver Processor LPC Controller [8086:31e8] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Device [1849:31e8] 
IOMMU group 10 00:1f.1 SMBus [0c05]: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model [8086:31d4] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Celeron/Pentium Silver Processor Gaussian Mixture Model [1849:31d4] 
    Kernel driver in use: i801_smbus 
    Kernel modules: i2c_i801 
IOMMU group 11 01:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01) 
    Subsystem: Device [1dcf:0319] 
    Kernel driver in use: igb 
    Kernel modules: igb 
IOMMU group 12 01:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01) 
    Subsystem: Device [1dcf:0319] 
    Kernel driver in use: igb 
    Kernel modules: igb 
IOMMU group 13 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) 
    DeviceName: Onboard - RTK Ethernet 
    Subsystem: ASRock Incorporation Motherboard (one of many) [1849:8168] 
    Kernel driver in use: r8169 
    Kernel modules: r8169 
IOMMU group 14 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) 
    Subsystem: ASRock Incorporation Motherboard [1849:0612] 
    Kernel driver in use: ahci 
    Kernel modules: ahci 
IOMMU group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation GeminiLake [UHD Graphics 605] [8086:3184] (rev 06) 
    DeviceName: Onboard - Video 
    Subsystem: ASRock Incorporation UHD Graphics 605 [1849:2212] 
    Kernel driver in use: i915 
    Kernel modules: i915 
IOMMU group 2 00:0e.0 Audio device [0403]: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio [8086:3198] (rev 06) 
    DeviceName: Onboard - Sound 
    Subsystem: ASRock Incorporation Celeron/Pentium Silver Processor High Definition Audio [1849:f892] 
    Kernel driver in use: snd_hda_intel 
    Kernel modules: snd_hda_intel, snd_soc_skl, snd_sof_pci 
IOMMU group 3 00:0f.0 Communication controller [0780]: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface [8086:319a] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface [1849:319a] 
    Kernel driver in use: mei_me 
    Kernel modules: mei_me 
IOMMU group 4 00:12.0 SATA controller [0106]: Intel Corporation Celeron/Pentium Silver Processor SATA Controller [8086:31e3] (rev 06) 
    DeviceName: Onboard - SATA 
    Subsystem: ASRock Incorporation Device [1849:31e3] 
    Kernel driver in use: ahci 
    Kernel modules: ahci 
IOMMU group 5 00:13.0 PCI bridge [0604]: Intel Corporation Gemini Lake PCI Express Root Port [8086:31d8] (rev f6) 
    Kernel driver in use: pcieport 
IOMMU group 6 00:13.1 PCI bridge [0604]: Intel Corporation Gemini Lake PCI Express Root Port [8086:31d9] (rev f6) 
    Kernel driver in use: pcieport 
IOMMU group 7 00:13.2 PCI bridge [0604]: Intel Corporation Gemini Lake PCI Express Root Port [8086:31da] (rev f6) 
    Kernel driver in use: pcieport 
IOMMU group 8 00:13.3 PCI bridge [0604]: Intel Corporation Gemini Lake PCI Express Root Port [8086:31db] (rev f6) 
    Kernel driver in use: pcieport 
IOMMU group 9 00:15.0 USB controller [0c03]: Intel Corporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller [8086:31a8] (rev 06) 
    DeviceName: Onboard - Other 
    Subsystem: ASRock Incorporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller [1849:31a8] 
    Kernel driver in use: xhci_hcd 
    Kernel modules: xhci_pci
avw said:
DId you use pcie_acs_override (which makes the command above useless) to break the IOMMU groups?
Click to expand...
No, at least not intentionally. How can I check this?
avw said:
Can you show us your VM configuration file from /etc/pve/qemu-server/?
Click to expand...
Code: 
root@proxmox:~# cat /etc/pve/qemu-server/100.conf
boot: order=scsi0;ide2;net0
cores: 2
hostpci0: 0000:04:00.0
ide2: local:iso/openmediavault_5.6.13-amd64.iso,media=cdrom
memory: 2048
name: OMV
net0: virtio=5A:C6:7C:91:E3:62,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=857e1131-dff4-49be-a001-5889eb557d2e
sockets: 1
vmgenid: a6c79741-6748-4392-9890-c61006e58454
 
If pcie_acs_override does not occur in cat /proc/cmdline, then it is not used.
Maybe you need to add vfio_pci.ids=1b21:0612 to the kernel parameters or make sure ahci is not loaded for this device, as explained here.
Someone else here reported it started working OVMF instead of SeaBIOS.
Maybe you motherboard just crashes when doing passthrough (of this device) (using this BIOS version)?
 
npu said:
Both SATA controllers use the same kernel module "ahci", so I guess blacklisting the module is not an option.
Which IDs should I use for options vfio_pci ids=1b21:0612 in my system? 0000:0400?
Click to expand...
Put the line options vfio_pci ids=1b21:0612 (which is already correct for your SATA controller) in a .conf-file inside /etc/modprobe.d/ and you might need to add softdep ahci pre: vfio_pci as well, much like I explained in an editted post here (and don't forget to run update-initramfs -u).
However, if you have muiltiple SATA controllers with the same id (lspci -nn shows 1b21:0612 for more than one device and you don't want to passthrough all of them, then I don't know what to do or try. Fortunately, I don't see that in the IOMMU groups that you showed: the other one has id 8086:31e3.
 
  • Like
Reactions: npu
After adding options vfio_pci ids=1b21:0612 the behavior was exactly the same.
After adding the second line softdep ahci pre: vfio_pci the system did not boot anymore. It did not find volume group "pve" anymore. So it seems that this command also affects the system SATA controller.

Code: 
root@proxmox:~# cat /etc/modprobe.d/sata-passthrough.conf
options vfio_pci ids=1b21:0612
#softdep ahci pre: vfio_pci

/EDIT:
I compared for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done before and after starting the VM.
The only difference is the "kernel driver in use" which switches from "ahci" to "vfio_pci" for the ASM1062 controller.

Code: 
IOMMU group 14 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
    Subsystem: ASRock Incorporation Motherboard [1849:0612]
    Kernel driver in use: vfio-pci
    Kernel modules: ahci

I also tried a new VM with OVMF instead of SeaBIOS but this does not make any difference.
 
Last edited:
npu said:
After adding options vfio_pci ids=1b21:0612 the behavior was exactly the same.
After adding the second line softdep ahci pre: vfio_pci the system did not boot anymore. It did not find volume group "pve" anymore. So it seems that this command also affects the system SATA controller.

Code: 
root@proxmox:~# cat /etc/modprobe.d/sata-passthrough.conf
options vfio_pci ids=1b21:0612
#softdep ahci pre: vfio_pci
Click to expand...
Sorry, I did not expect that to be problematic. Are you sure your drives are connected to the other SATA controller? Can you show me ls -l /dev/disk/by-path/pci-*?
 
  • Like
Reactions: npu
Before starting the VM:
Code: 
root@proxmox:~# ls -l /dev/disk/by-path/pci-*
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-1 -> ../../sda
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-1.0 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-1.0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-1-part1 -> ../../sda1
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-2 -> ../../sdb
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-2.0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-2.0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:00:12.0-ata-2-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2 -> ../../sdc
lrwxrwxrwx 1 root root  9 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2.0 -> ../../sdc
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2.0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2.0-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2.0-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Oct  3 13:29 /dev/disk/by-path/pci-0000:04:00.0-ata-2-part3 -> ../../sdc3
After starting the VM:
Code: 
root@proxmox:~# ls -l /dev/disk/by-path/pci-*
lrwxrwxrwx 1 root root  9 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-1 -> ../../sda
lrwxrwxrwx 1 root root  9 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-1.0 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-1.0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-1-part1 -> ../../sda1
lrwxrwxrwx 1 root root  9 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-2 -> ../../sdb
lrwxrwxrwx 1 root root  9 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-2.0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-2.0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct  3 13:14 /dev/disk/by-path/pci-0000:00:12.0-ata-2-part1 -> ../../sdb1

/EDIT:

/dev/sdc is the system disk. Is there any command that shows which disk is connected to which controller?
 
Last edited:
OMG I am feeling so stupid. The disks were indeed connected to the wrong controllers. :rolleyes:
What a stupid mistake. I am sorry for wasting your time.
With correct cabling everything works as expected now. Even without the changes in /etc/modprobe.d/
 
  • Like
Reactions: leesteken
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back