Onboard SATA controller PCI Passthrough

Groovy6727 · 2025-05-02T20:12:42+0200

I am following Proxmox documentation
https://pve.proxmox.com/wiki/PCI_Passthrough
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough
and trying to passthrough onboard SATA controller to an OMV VM.

IOMMU is enabled in the BIOS/UEFI, SATA controller is the only device in IOMMU group #14.
If I remove PCI passthrough from VM, everything works like a charm, with SATA controller passthrough, host becomes unresponsive and hard reset is required.
Please help me understand what changes needs to be implemented for this PCI passthrough to work.

Host info:

Code:

OS: Proxmox VE 8.2.2 x86_64
Host: 11A4000HGE ThinkCentre M75q-1
Kernel: 6.8.4-2-pve
CPU: AMD Ryzen 5 PRO 3400GE w/ Radeon Vega Graphics (8) @ 3.300GHz
GPU: AMD ATI Radeon Vega Series / Radeon Vega Mobile Series

SATA controller info:

Code:

~# lspci -v
05:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61) (prog-if 01 [AHCI 1.0])
    Subsystem: Lenovo FCH SATA Controller [AHCI mode]
    Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 14
    Memory at fcc00000 (32-bit, non-prefetchable) [size=2K]
    Capabilities: [48] Vendor Specific Information: Len=08 <?>
    Capabilities: [50] Power Management version 3
    Capabilities: [64] Express Endpoint, MSI 00
    Capabilities: [a0] MSI: Enable- Count=1/2 Maskable- 64bit+
    Capabilities: [d0] SATA HBA v1.0
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [150] Advanced Error Reporting
    Capabilities: [270] Secondary PCI Express
    Capabilities: [2a0] Access Control Services
    Kernel driver in use: vfio-pci
    Kernel modules: ahci

Host config:

Code:

~# cat /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt vfio-pci.ids=1022:7901"
GRUB_CMDLINE_LINUX=""

~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

~# cat /etc/modprobe.d/pve-blacklist.conf
blacklist nvidiafb
blacklist ahci

~# cat /etc/modprobe.d/vfio_iommu.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

~# cat /etc/modprobe.d/vfio-pci.conf
options vfio-pci ids=1022:7901

Code:

~# update-grub && update-initramfs -u -k all && reboot now
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.4-2-pve
Found initrd image: /boot/initrd.img-6.8.4-2-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu entry for UEFI Firmware Settings ...
done
update-initramfs: Generating /boot/initrd.img-6.8.4-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

checking if PCIe passthrough is possible

Code:

~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi -e remapping
[    0.066674] AMD-Vi: Unknown option - 'on'
[    0.159073] AMD-Vi: Using global IVHD EFR:0x4f77ef22294ada, EFR2:0x0
[    0.439331] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.440703] AMD-Vi: Extended features (0x4f77ef22294ada, 0x0): PPR NX GT IA GA PC GA_vAPIC
[    0.440716] AMD-Vi: Interrupt remapping enabled
[    0.440872] AMD-Vi: Virtual APIC enabled
[    0.441003] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

VM config:

Code:

~# cat /etc/pve/qemu-server/100.conf
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 6
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:05:00.0
machine: q35,viommu=intel
memory: 24576
meta: creation-qemu=8.1.5,ctime=1745780999
name: OMVvm
net0: virtio=BC:24:11:74:A4:F5,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=64G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=c8365e26-85ba-431b-9a16-2f398e5f1ea4
sockets: 1
tags: openmediavault
tpmstate0: local-lvm:vm-100-disk-2,size=4M,version=v2.0
vmgenid: e8aeb4ed-def7-41e9-8ec9-b965f8647795

kernel log after starting the VM:

Code:

~# echo "starting VM with PCI PT" > /dev/kmsg && dmesg -wH
[May 2 19:06] starting VM with PCI PT
[May 2 19:07] tap100i0: entered promiscuous mode
[  +0.083950] vmbr0: port 2(fwpr100p0) entered blocking state
[  +0.000008] vmbr0: port 2(fwpr100p0) entered disabled state
[  +0.000028] fwpr100p0: entered allmulticast mode
[  +0.000064] fwpr100p0: entered promiscuous mode
[  +0.000054] vmbr0: port 2(fwpr100p0) entered blocking state
[  +0.000003] vmbr0: port 2(fwpr100p0) entered forwarding state
[  +0.015343] fwbr100i0: port 1(fwln100i0) entered blocking state
[  +0.000006] fwbr100i0: port 1(fwln100i0) entered disabled state
[  +0.000019] fwln100i0: entered allmulticast mode
[  +0.000061] fwln100i0: entered promiscuous mode
[  +0.000053] fwbr100i0: port 1(fwln100i0) entered blocking state
[  +0.000002] fwbr100i0: port 1(fwln100i0) entered forwarding state
[  +0.014036] fwbr100i0: port 2(tap100i0) entered blocking state
[  +0.000006] fwbr100i0: port 2(tap100i0) entered disabled state
[  +0.000019] tap100i0: entered allmulticast mode
[  +0.000116] fwbr100i0: port 2(tap100i0) entered blocking state
[  +0.000003] fwbr100i0: port 2(tap100i0) entered forwarding state

Impact · 2025-05-02T20:58:17+0200

Please share your IOMMU groups

Bash:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Where is the OS booted from? Hopefully not from a disk connected to that SATA controller.

Groovy6727 · 2025-05-02T21:49:21+0200

ThinkCentre M75q-1 has one NVME drive and one SATA drive. Proxmox is booting of an NVME drive, where a volume for VM data is located as well.

Code:

# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1                      259:0    0 238.5G  0 disk
├─nvme0n1p1                  259:1    0  1007K  0 part
├─nvme0n1p2                  259:2    0     1G  0 part /boot/efi
└─nvme0n1p3                  259:3    0 237.5G  0 part
  ├─pve-swap                 252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 252:1    0  69.4G  0 lvm  /
  ├─pve-data_tmeta           252:2    0   1.4G  0 lvm 
  │ └─pve-data-tpool         252:4    0 141.2G  0 lvm 
  │   ├─pve-data             252:5    0 141.2G  1 lvm 
  │   ├─pve-vm--100--disk--0 252:6    0     4M  0 lvm 
  │   ├─pve-vm--100--disk--1 252:7    0    64G  0 lvm 
  │   └─pve-vm--100--disk--2 252:8    0     4M  0 lvm 
  └─pve-data_tdata           252:3    0 141.2G  0 lvm 
    └─pve-data-tpool         252:4    0 141.2G  0 lvm 
      ├─pve-data             252:5    0 141.2G  1 lvm 
      ├─pve-vm--100--disk--0 252:6    0     4M  0 lvm 
      ├─pve-vm--100--disk--1 252:7    0    64G  0 lvm 
      └─pve-vm--100--disk--2 252:8    0     4M  0 lvm

I am trying to create an OMV NAS in a VM and passthrough entire SATA controller to the VM.

This is output of the bash script:

Code:

# ./debug_script.sh
IOMMU Group 0:
    00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 1:
    00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3]
IOMMU Group 2:
    00:01.5 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3]
IOMMU Group 3:
    00:01.6 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3]
IOMMU Group 4:
    00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5:
    00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A [1022:15db]
IOMMU Group 6:
    00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B [1022:15dc]
IOMMU Group 7:
    00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
    00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 8:
    00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0 [1022:15e8]
    00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1 [1022:15e9]
    00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2 [1022:15ea]
    00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3 [1022:15eb]
    00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4 [1022:15ec]
    00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5 [1022:15ed]
    00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6 [1022:15ee]
    00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7 [1022:15ef]
IOMMU Group 9:
    01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU Group 10:
    02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 11:
    03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0e)
    03:00.1 Serial controller [0700]: Realtek Semiconductor Co., Ltd. RTL8111xP UART #1 [10ec:816a] (rev 0e)
    03:00.2 Serial controller [0700]: Realtek Semiconductor Co., Ltd. RTL8111xP UART #2 [10ec:816b] (rev 0e)
    03:00.3 IPMI Interface [0c07]: Realtek Semiconductor Co., Ltd. RTL8111xP IPMI interface [10ec:816c] (rev 0e)
    03:00.4 USB controller [0c03]: Realtek Semiconductor Co., Ltd. RTL811x EHCI host controller [10ec:816d] (rev 0e)
IOMMU Group 12:
    04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Picasso/Raven 2 [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15d8] (rev da)
IOMMU Group 13:
    04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de]
    04:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
    04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e0]
    04:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e1]
    04:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2]
    04:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 14:
    05:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)

Impact · 2025-05-03T02:32:14+0200

Sorry that this is not helpful but at first glance everything related to this problem looks okay to me. I fail to see an obvious issue.

Search

Search

Onboard SATA controller PCI Passthrough

Groovy6727

New Member

Impact

Member

Groovy6727

New Member

Impact

Member

We value your privacy