PCIe Passthrough SATA Controller - Disks busy (solved)

L

lenox89

New Member
Mar 11, 2025
6
0
1
Hello everyone,
I have created a VM (Debian, more precisely Openmediavault) and passed a SATA controller through to it via PCIe passthrough. The hard disks connected to it are displayed in the VM's OS. Unfortunately, two of the three hard disks are permanently “busy”. Do you have any idea why this could be?

VM OS
Bash: 
root@homer:~# wipefs -a /dev/sdb
root@homer:~# wipefs -a /dev/sdc
wipefs: error: /dev/sdc: probing initialization failed: Device or resource busy
root@homer:~# wipefs -a /dev/sdd
wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
root@homer:~# lsof | grep sdc
root@homer:~#

I proceeded as follows to set it up:
On PVE Host
Code: 
for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:590f] (rev 06)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 06)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 06)
IOMMU Group 1 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU Group 1 02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU Group 2 00:13.0 Non-VGA unclassified device [0000]: Intel Corporation 100 Series/C230 Series Chipset Family Integrated Sensor Hub [8086:a135] (rev 31)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU Group 3 00:14.2 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem [8086:a131] (rev 31)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 [8086:a13a] (rev 31)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] [8086:a102] (rev 31)
IOMMU Group 6 00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 [8086:a114] (rev f1)
IOMMU Group 7 00:1d.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 [8086:a118] (rev f1)
IOMMU Group 8 00:1d.1 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #10 [8086:a119] (rev f1)
IOMMU Group 9 00:1d.2 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #11 [8086:a11a] (rev f1)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation C232 Chipset LPC/eSPI Controller [8086:a14a] (rev 31)
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31)
IOMMU Group 10 00:1f.4 SMBus [0c05]: Intel Corporation 100 Series/C230 Series Chipset Family SMBus [8086:a123] (rev 31)
IOMMU Group 11 03:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU Group 12 04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 13 05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 14 06:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
IOMMU Group 14 07:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)


dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.008606] ACPI: DMAR 0x000000007D80BCE0 000070 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.008642] ACPI: Reserving DMAR table memory at [mem 0x7d80bce0-0x7d80bd4f]
[    0.119628] DMAR: Host address width 39
[    0.119630] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.119638] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.119640] DMAR: RMRR base: 0x0000007d181000 end: 0x0000007d1a0fff
[    0.119646] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.119647] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.119649] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.119650] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.120079] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.369047] DMAR: No ATSR found
[    0.369048] DMAR: No SATC found
[    0.369050] DMAR: dmar0: Using Queued invalidation
[    0.370309] DMAR: Intel(R) Virtualization Technology for Directed I/O

vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT
add: "iommu=pt"
update-grub
reboot
/etc/modules
 vfio
 vfio_iommu_type1
 vfio_pci
update-initramfs -u -k all
reboot


lsmod | grep vfio
vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   65536  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                94208  1 vfio

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.008757] ACPI: DMAR 0x000000007D80BCE0 000070 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.008792] ACPI: Reserving DMAR table memory at [mem 0x7d80bce0-0x7d80bd4f]
[    0.119812] DMAR: Host address width 39
[    0.119813] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.119821] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.119823] DMAR: RMRR base: 0x0000007d181000 end: 0x0000007d1a0fff
[    0.119828] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.119829] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.119830] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.119831] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.120260] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.369278] DMAR: No ATSR found
[    0.369279] DMAR: No SATC found
[    0.369280] DMAR: dmar0: Using Queued invalidation
[    0.369747] DMAR: Intel(R) Virtualization Technology for Directed I/O

dmesg | grep 'remapping'
[    0.119436] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.119438] x2apic: IRQ remapping doesn't support X2APIC mode

pvesh get /nodes/pve/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬─────────────────────────────────────────────────────────────────────────┬──────┬──────────────────┬───────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                                             │ mdev │ subsystem_device │ subsys
╞══════════╪════════╪══════════════╪════════════╪════════╪═════════════════════════════════════════════════════════════════════════╪══════╪══════════════════╪═══════
│...
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────
│ 0x010601 │ 0xa102 │ 0000:00:17.0 │          5 │ 0x8086 │ Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] │      │ 0x089a

Proxmox WebGUI
Create VM
name: homer
machine: q35
bios: UEFI
scsi: VirtlIO SCSI single
disks: 64 GB
cpu: type: host, cores: 2
ram: 8192 GB

VM
Install Openmediavault
disable secure boot im VM Bios (https://www.thomas-krenn.com/de/wiki/UEFI_VM_-_failed_to_boot:_Access_Denied) for uefi boot
shutdown
enable qemu agent
install updates
shutdown
create snapshot
add pci hardware
raw device: 0000:00:17.0 (sata controller)
enable PCIe: yes
all functions: yes
 
Output of the kernel log

Bash: 
[    2.010121] scsi 9:0:0:0: Direct-Access     ATA      WDC WD40EFRX-68N 0A82 PQ: 0 ANSI: 5
[    2.010428] sd 9:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    2.010431] sd 9:0:0:0: [sdb] 4096-byte physical blocks
[    2.010438] sd 9:0:0:0: [sdb] Write Protect is off
[    2.010440] sd 9:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    2.010453] sd 9:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.010469] sd 9:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[    2.010572] scsi 10:0:0:0: Direct-Access     ATA      WDC WD40EFRX-68N 0A82 PQ: 0 ANSI: 5
[    2.010958] sd 10:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    2.010961] sd 10:0:0:0: [sdc] 4096-byte physical blocks
[    2.010968] sd 10:0:0:0: [sdc] Write Protect is off
[    2.010970] sd 10:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    2.010979] sd 10:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.010996] sd 10:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[    2.011240] scsi 11:0:0:0: Direct-Access     ATA      WDC WD40EFRX-68N 0A82 PQ: 0 ANSI: 5
[    2.011624] sd 11:0:0:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    2.011627] sd 11:0:0:0: [sdd] 4096-byte physical blocks
[    2.011676] sd 11:0:0:0: [sdd] Write Protect is off
[    2.011679] sd 11:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    2.011690] sd 11:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.011708] sd 11:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes
[    2.046558] sd 10:0:0:0: [sdc] Attached SCSI disk
[    2.046683] sd 11:0:0:0: [sdd] Attached SCSI disk
[    2.053443] sd 9:0:0:0: [sdb] Attached SCSI disk

Bash: 
[  269.879573] /dev/sdc: Can't open blockdev

Smartctl
Bash: 
smartctl -i /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.12.12+bpo-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red (CMR)
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-xxxxx
LU WWN Device Id: 5 0014ee 2ba6f9c02
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Mar 20 15:15:03 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
Last edited:
hm, a force wipe does the job..

Bash: 
root@homer:~# wipefs -af /dev/sdc

To complete the guide (which might be interesting for others):
install openmediavault-kernel plugin
Install proxmox Kernel: System->kernel->proxmox->install 6.11
update
reboot
install zfs plugin
reboot
System->Plugins->ZFS->Create Pool
 
You must log in or register to reply here.
Top Bottom