Trying to pass through a SATA controller on the motherboard

M

Memran

New Member
Nov 5, 2021
6
0
1
45
Hi!

I'm trying to pass through a SATA controller, so that it is available to a TrueNAS VM. I think this is more difficult than it should be due to the device being embedded on the motherboard itself.

The problem is that although the VM can see the PCI device(s) the HDDs are not recognized.

On my Proxmox node:

Code: 
root@proxmox:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0 465.8G  0 disk       <====== this disk is on the SATA controller
sdb            8:16   0 167.7G  0 disk
├─sdb1         8:17   0  1007K  0 part
├─sdb2         8:18   0   512M  0 part
└─sdb3         8:19   0 167.2G  0 part
  ├─pve-swap 253:0    0     5G  0 lvm  [SWAP]
  └─pve-root 253:1    0 162.2G  0 lvm  /

The HDD connected to the SATA controller is sda.

Code: 
root@proxmox:~# udevadm info -q path -n /dev/sda
/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda

The controller appears to be 00:1f:2

Code: 
root@proxmox:~# lspci
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1    <================ This device
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s controller (rev 11)
02:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
03:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce GTS 250] (rev a2)
05:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)
07:02.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
ff:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
ff:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
ff:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
ff:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
ff:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
ff:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
ff:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
ff:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
ff:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
ff:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
ff:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
ff:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
ff:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
ff:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
ff:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
ff:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
ff:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
ff:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
ff:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)

Therefore in the VM's hardware config, I add PCIe device 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1

Now in the VM, I can see the device:

Code: 
root@truenas[~]# pciconf -lcevV
hostb0@pci0:0:0:0:      class=0x060000 card=0x11001af4 chip=0x12378086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '440FX - 82441FX PMC [Natoma]'
    class      = bridge
    subclass   = HOST-PCI
isab0@pci0:0:1:0:       class=0x060100 card=0x11001af4 chip=0x70008086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371SB PIIX3 ISA [Natoma/Triton II]'
    class      = bridge
    subclass   = PCI-ISA
atapci0@pci0:0:1:1:     class=0x010180 card=0x11001af4 chip=0x70108086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371SB PIIX3 IDE [Natoma/Triton II]'
    class      = mass storage
    subclass   = ATA
uhci0@pci0:0:1:2:       class=0x0c0300 card=0x11001af4 chip=0x70208086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371SB PIIX3 USB [Natoma/Triton II]'
    class      = serial bus
    subclass   = USB
intsmb0@pci0:0:1:3:     class=0x068000 card=0x11001af4 chip=0x71138086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371AB/EB/MB PIIX4 ACPI'
    class      = bridge
vgapci0@pci0:0:2:0:     class=0x030000 card=0x11001af4 chip=0x11111234 rev=0x02 hdr=0x00
    class      = display
    subclass   = VGA
virtio_pci0@pci0:0:3:0: class=0x00ff00 card=0x00051af4 chip=0x10021af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio memory balloon'
    class      = old
    cap 09[84] = vendor (length 20)
    cap 09[70] = vendor (length 20)
    cap 09[60] = vendor (length 16)
    cap 09[50] = vendor (length 16)
    cap 09[40] = vendor (length 16)
virtio_pci1@pci0:0:5:0: class=0x010000 card=0x00081af4 chip=0x10041af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    cap 11[98] = MSI-X supports 7 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[84] = vendor (length 20)
    cap 09[70] = vendor (length 20)
    cap 09[60] = vendor (length 16)
    cap 09[50] = vendor (length 16)
    cap 09[40] = vendor (length 16)
atapci1@pci0:0:16:0:    class=0x01018f card=0x82d41043 chip=0x3a208086 rev=0x00 hdr=0x00    <================ Here it is
    vendor     = 'Intel Corporation'
    device     = '82801JI (ICH10 Family) 4 port SATA IDE Controller'
    class      = mass storage
    subclass   = ATA
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 13[b0] = PCI Advanced Features: FLR TP
virtio_pci2@pci0:0:18:0:        class=0x020000 card=0x00011af4 chip=0x10001af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio network device'
    class      = network
    subclass   = ethernet
    cap 11[98] = MSI-X supports 4 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[84] = vendor (length 20)
    cap 09[70] = vendor (length 20)
    cap 09[60] = vendor (length 16)
    cap 09[50] = vendor (length 16)
    cap 09[40] = vendor (length 16)
pcib1@pci0:0:30:0:      class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[4c] = MSI supports 1 message, 64 bit, vector masks
    cap 04[48] = unknown
    cap 0c[40] = unknown
pcib2@pci0:0:31:0:      class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[4c] = MSI supports 1 message, 64 bit, vector masks
    cap 04[48] = unknown
    cap 0c[40] = unknown

However, now I cannot see the attached HDD:

Code: 
root@truenas[~]# geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: QEMU QEMU HARDDISK
   ident: (null)
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 962168832 (918M)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Code: 
root@truenas[~]# camcontrol devlist
<QEMU QEMU DVD-ROM 2.5+>           at scbus1 target 0 lun 0 (cd0,pass0)
<QEMU QEMU HARDDISK 2.5+>          at scbus2 target 0 lun 0 (pass1,da0)

I'm not sure how to proceed from here, or indeed, whether it is possible with this hardware. I suspect that since there are 4 devices in the same IOMMU group, the hardware passthrough will not work.

Is there anything else I can do?
 
Last edited:
If you see the device inside the VM (and the host is working fine) then PCI passthrough is working in principle. PCI bridges in a IOMMU group are not a problem. If the device is not really usable, this is usually because it does not reset properly.
Sometimes it helps to make sure the Proxmox host does not touch the device. Either by blacklisting the driver, which might be a problem if the driver is needed for other SATA controllers. Or by adding vfio-pci.ids=xxxx:yyyy to the kernel parameters where xxxx:yyyy is the lspci -n ID of the device, which sometimes does not help because you might have multiple controllers with the same ID.
Sometimes, a device just won't reset and people buy an add-in PCIe card that is known to work with PCI passthrough. Can you find a post here or on the internet that reports this particular device as working with passthrough?
 
Thanks for your reply. I don't really understand what I am doing here. Is this what you meant?

Code: 
root@proxmox:~# lspci | grep 00:1f
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
root@proxmox:~# lspci -n -s 00:1f
00:1f.0 0601: 8086:3a16
00:1f.2 0101: 8086:3a20
00:1f.3 0c05: 8086:3a30
00:1f.5 0101: 8086:3a26
root@proxmox:~# more /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:3a20,8086:3a26

Code: 
root@pve:/# more /etc/pve/qemu-server/100.conf
boot: order=scsi0;net0
cores: 8
hostpci0: 0000:00:1f.2
hostpci1: 0000:00:1f.5
machine: q35
memory: 4096
name: TrueNAS
net0: virtio=86:84:00:75:0E:19,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=e8ccb22b-19fc-4c53-a7c1-419837e1c889
sockets: 1
vmgenid: a6f29248-613e-4d7a-826d-5ed849514e3b

I also don't understand why the VM is seeing whole list of other Intel devices, which I did not pass through...

I'm out of my depth here to be honest!
 
Last edited:
Yes, using the numeric IDs of the device and binding them to vfio-pci is indeed what you did. Did you run update-initramfs -u and reboot after making those changes?
You can check if the changes are in effect by running lspci -k -s 00:1f after rebooting the Proxmox host and before(!) starting the VM. It will show the actual drivers, but the driver in use should be vfio-pci.
If those changes don't help when you restart the Proxmox host and starting the VM for the frist time. Even if these changes help, it might be that the devices stop working when you stop the VM and start it a second time, without restarting the Proxmox host.
You have not yet shown us the VM configuration file (from the /etc/pve/qemu-server/ directory), but have you tried enabling or disabling the ROM-Bar option?

Every VM contains Intel devices because it emulates a Intel i440fx or Intel q35 motherboard for historic reasons. Maybe those are the devices you see? There are also Red Hat, Inc. devices for paravirtualized devices. That's all normal, but feel free to run spci -nn and post the devices you worry about.
 
Thanks again!

Those extra devices must be as you say from i440fx or q35 (I tried both during my attempts).

I definitely did run update-initramfs -u and then rebooted.

It looks like the driver for my SATA devices is not vfio-pci:
Code: 
root@pve:~# lspci -k -s 00:1f
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
        Subsystem: ASUSTeK Computer Inc. 82801JI (ICH10 Family) 4 port SATA IDE Controller
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel modules: i2c_i801
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
        Subsystem: ASUSTeK Computer Inc. 82801JI (ICH10 Family) 2 port SATA IDE Controller
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi

Here's my VM conf:
Code: 
root@pve:/# more /etc/pve/qemu-server/100.conf
boot: order=scsi0;net0
cores: 8
hostpci0: 0000:00:1f
memory: 4096
name: TrueNAS
net0: virtio=86:84:00:75:0E:19,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=e8ccb22b-19fc-4c53-a7c1-419837e1c889
sockets: 1
vmgenid: a6f29248-613e-4d7a-826d-5ed849514e3b

I've changed so many things, it's really hard to keep track of what was a good change and what wasn't :(

Code: 
root@pve:~# more /etc/default/grub | grep LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

root@pve:~# more /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

root@pve:~# more /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

root@pve:~# more /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

root@pve:~# more /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:3a20,8086:3a26
 
The vfio-pci options are not in effect. Try adding:
softdep ata_piix pre: vfio-pci softdep pata_acpi pre: vfio-pci
under the line options vfio-pci ids=8086:3a20,8086:3a26, and run update-initramfs -u and reboot.
Hopefully, vfio-pci will then take precedence over ata_piix and pata_acpi.
 
Thanks, but still no :(
Code: 
root@pve:~# more /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:3a20,8086:3a26
softdep ata_piix pre: vfio-pci
softdep pata_acpi pre: vfio-pci

After update-initramfs -u and reboot, its unchanged:
Code: 
root@pve:~# lspci -k -s 00:1f
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
        Subsystem: ASUSTeK Computer Inc. 82801JI (ICH10 Family) 4 port SATA IDE Controller
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel modules: i2c_i801
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
        Subsystem: ASUSTeK Computer Inc. 82801JI (ICH10 Family) 2 port SATA IDE Controller
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi
 
You could try blacklisting if lspci | grep pata_acpi and lspci | grep ata_piix don't return any other devices. This could cause the system to lose access to others disk, if there are other devices that use it! Add
blacklist pata_acpi blacklist ata_piixi and update-initramfs -u and reboot.
Otherwise, I would not know how to do it, sorry.
 
Code: 
root@pve:~# lspci | grep pata_acpi
root@pve:~# lspci | grep ata_piix
root@pve:~# nano /etc/modprobe.d/blacklist.conf
root@pve:~# more /etc/modprobe.d/blacklist.conf
blacklist pata_acpi
blacklist ata_piix
root@pve:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.11.22-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@pve:~# reboot
Connection to 192.168.1.204 closed by remote host.
Connection to 192.168.1.204 closed.

However, it's still not using the vfio driver.

I think I'm going to leave this for today and try to do more research.

Thank you so much for helping this far. Truly appreciated :)
 
I know this is a slightly old thread now, but it is one of the top results on Google at the moment - did you ever find a solution for this problem?
 
  • Like
Reactions: HydroHomie
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back