SATA controller passthrough

S

sdpve

Member
Mar 10, 2018
22
0
6
44
I've passthrough the SATA controller on my server to a VM. I was able to see that the controller is passed through and that the disks are found. I was also able to paritition the disks. But I cannot mount the disks. So something is wrong. Here is what I did:

Following this guide:https://pve.proxmox.com/wiki/Pci_passthrough
*root@pve:~# pveversion
pve-manager/5.1-41/0b958203 (running kernel: 4.13.13-2-pve)

1.Identified the controller and verified it's a root node:
root@pve:~# lspci -knn | grep -A2 SATA
00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 31)
Subsystem: Dell SATA Controller [RAID mode] [1028:07c5]
Kernel driver in use: vfio-pci
Kernel modules: ahci
root@pve:~# lspci -t
-[0000:00]-+-00.0
+-01.0-[01]--
+-02.0
+-14.0
+-14.2
+-16.0
+-16.3
+-17.0
+-1b.0-[02]----00.0
+-1c.0-[03-04]----00.0-[04]--
+-1f.0
+-1f.2
+-1f.3
+-1f.4
\-1f.6

Note that it is currently using vfio-pci driver since I have made the changes and rebooted. Previously it was using achi driver. I don't know why kernel module is listed as ahci. Is that an issue?

2. edit and updated grub
host:
root@pve:~# cat /etc/default/grub | grep iommu
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
VM:
root@omv:/srv/dev-disk-by-label-data2# cat /etc/default/grub | grep iommu
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

3. Added vfio modules on host
root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

4. Verified IOMMU lines present and that they support interrupt remapping (e an a at the end of number after ecap)
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000008B912CE0 0000A8 (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.034361] DMAR: Host address width 39
[ 0.034362] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.034366] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ 0.034367] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.034370] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.034370] DMAR: RMRR base: 0x0000008b762000 end: 0x0000008b781fff
[ 0.034371] DMAR: RMRR base: 0x0000008d800000 end: 0x0000008fffffff
[ 0.034372] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.034373] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.034373] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.034374] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.035766] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.569868] DMAR: No ATSR found
[ 0.570180] DMAR: dmar0: Using Queued invalidation
[ 0.570183] DMAR: dmar1: Using Queued invalidation
[ 0.570379] DMAR: Setting RMRR:
[ 0.570400] DMAR: Setting identity map for device 0000:00:02.0 [0x8d800000 - 0x8fffffff]
[ 0.570418] DMAR: Setting identity map for device 0000:00:14.0 [0x8b762000 - 0x8b781fff]
[ 0.570422] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.570435] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.570457] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@pve:~# dmesg | grep ecap
[ 0.034366] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ 0.034370] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da

5. Verified IOMMU processor support and isolation
root@pve:~# cat /proc/cpuinfo | grep Xeon
model name : Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz
model name : Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz
model name : Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz
model name : Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz

root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.3
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4
/sys/kernel/iommu_groups/8/devices/0000:00:1f.2
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.3
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:02:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6

6. VM config setting to pass pci
root@pve:~# cat /etc/pve/qemu-server/100.conf | grep hostpci
hostpci0: 00:17.0

7. Rebooted host

8. Verified the controller on VM
00:10.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 31)
Subsystem: Dell SATA Controller [RAID mode] [1028:07c5]
Kernel driver in use: ahci
Kernel modules: ahci
Should it be using the vfio-pci driver?

9. Verified disks on VM
Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F24C7248-7C9E-4771-A9C1-22DA6FA8XXXX

Device Start End Sectors Size Type
/dev/sdd1 2048 5860533134 5860531087 2.7T Linux filesystem


Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 543E2DAD-B1CB-4CD9-9C3D-0B5ABD80XXXX

This is what they look like after I deleted the previous partitions and setup 1 ext4 parition. Previously, these drives were in linux software raid (md..). I used parted tool and it didn't give errors

10. Mounting drives failes:
root@omv:~# mount /dev/disk/by-label/data2 /srv/dev-disk-by-label-data2
mount: mount /dev/sdc1 on /srv/dev-disk-by-label-data2 failed: Structure needs cleaning

e2fsck returns errors and I tried lettting it fix them. But at the end I still got:
elocating group 22355's block bitmap to 732431878...
Relocating group 22355's inode bitmap to 732431879...
Relocating group 22355's inode table to 732431880...
Relocating group 22356's block bitmap to 732432392...
Relocating group 22356's inode bitmap to 732432393...
Relocating group 22356's inode table to 732432394...
Recreate journal? yes

Creating journal (262144 blocks): Block bitmap checksum does not match bitmap: while trying to create journal
Restarting e2fsck from the beginning...
ext2fs_open2: Bad magic number in super-block
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/disk/by-label/data2

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>


So I am kind of stuck in that I got far enough to believe my controller passthrough worked. But accessing the drives is not working, so obviously something is not working. Anything I missed on the steps or any diagnostics I can run?

---PS. What is x2apic and do I need to enable it in the bios?
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000008B912CE0 0000A8 (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.034361] DMAR: Host address width 39
[ 0.034362] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.034366] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ 0.034367] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.034370] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.034370] DMAR: RMRR base: 0x0000008b762000 end: 0x0000008b781fff
[ 0.034371] DMAR: RMRR base: 0x0000008d800000 end: 0x0000008fffffff
[ 0.034372] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.034373] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.034373] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.034374] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.035766] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.569868] DMAR: No ATSR found
[ 0.570180] DMAR: dmar0: Using Queued invalidation
[ 0.570183] DMAR: dmar1: Using Queued invalidation
[ 0.570379] DMAR: Setting RMRR:
[ 0.570400] DMAR: Setting identity map for device 0000:00:02.0 [0x8d800000 - 0x8fffffff]
[ 0.570418] DMAR: Setting identity map for device 0000:00:14.0 [0x8b762000 - 0x8b781fff]
[ 0.570422] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.570435] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.570457] DMAR: Intel(R) Virtualization Technology for Directed I/O
 
Last edited:
sdpve said:
00:10.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 31)
Subsystem: Dell SATA Controller [RAID mode] [1028:07c5]
Click to expand...
what kind of controller is this? it seems like some kind of fakeraid? maybe this is were your issues come from?
have you tried setting that in ahci mode in bios?
 
dcsapak said:
what kind of controller is this? it seems like some kind of fakeraid? maybe this is were your issues come from?
have you tried setting that in ahci mode in bios?
Click to expand...
Yes it is fakeraid. I thought i had put it in ahci mode, but I will double check. I am definitely not using it for raid.
 
the raid controller is listed on dell website as Software RAID: Intel Rapid Storage Controller 12.0 (supports SATA 6Gb/s or SATA 3Gb/s)

I did have it on raid mode on the bios. So I changed it to achi, but the problems still remains.

https://communities.intel.com/thread/43928 seems to hint that passthrough is possible. I am trying to figure out if this applies to my case. If passthrough was not supported, would I see the drives on the VM as I am seeing now? At this point a confirmation that it is not supported would be welcomed. It just feels like I am far enough that it should work.
 
I just found out there is an error upon starting the vm. It says "cannot reset device, no reset mechanism available" See images below. I ony just saw this image because i started vm through CLI instead of gui. Where do these errors go to? I checked dmesg and /var/log/messages and it isn't in there.

Anyhow, I am looking that error up and i found this post: https://forum.proxmox.com/threads/pci-passthrough-cant-reset-pci-device.11811/

I tried to manually add the passthrough as suggested and that failed saying no iommu support. (see image below) But there is iommu support... Looking more into this now that I atleast have an error to look into.

Not sure why images aren't working, here is a link.
mZg5s

https://imgur.com/a/mZg5s
mZg5s

mZg5s
 
Ok i tried the following:

changed grup to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt intel_iommu=on intremap=no_x2apic_optout"

added a vfio.conf
root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:a102

The vfio.conf did nothing because ahci driver comes up before it.
root@pve:~# dmesg | egrep -i ahci
[ 0.783504] ahci 0000:00:17.0: version 3.0
[ 0.783774] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[ 0.783775] ahci 0000:00:17.0: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst
[ 0.824907] scsi host0: ahci
[ 0.825241] scsi host1: ahci
[ 0.825465] scsi host2: ahci
[ 0.825618] scsi host3: ahci

[ 2.333843] vfio_pci: add [8086:a102[ffff:ffff]] class 0x000000/00000000

result:
root@pve:~#search)`ls: lspci -knn | grep -A2 SATA
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
Subsystem: Dell Sunrise Point-H SATA controller [AHCI mode] [1028:07c5]
Kernel driver in use: ahci
Kernel modules: ahci

I think this is okay, because it does get added dynamically.

The grub changes cleared up some errors on iommu startup, and it now looks like this:
root@pve:~#search)`dma: dmesg | egrep -i "dmar|iommu"
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.13.13-2-pve root=/dev/mapper/pve-root ro quiet iommu=pt intel_iommu=on intremap=no_x2apic_optout
[ 0.000000] ACPI: DMAR 0x000000008B912CE0 0000A8 (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.13-2-pve root=/dev/mapper/pve-root ro quiet iommu=pt intel_iommu=on intremap=no_x2apic_optout
[ 0.000000] DMAR: IOMMU enabled
[ 0.034329] DMAR: Host address width 39
[ 0.034330] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.034333] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ 0.034334] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.034336] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.034337] DMAR: RMRR base: 0x0000008b762000 end: 0x0000008b781fff
[ 0.034338] DMAR: RMRR base: 0x0000008d800000 end: 0x0000008fffffff
[ 0.034339] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.034340] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.034340] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.035737] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.569664] DMAR: No ATSR found
[ 0.569975] DMAR: dmar0: Using Queued invalidation
[ 0.569978] DMAR: dmar1: Using Queued invalidation
[ 0.570191] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 0.570192] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 0.570196] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 0.570197] DMAR: Hardware identity mapping for device 0000:00:14.0
[ 0.570197] DMAR: Hardware identity mapping for device 0000:00:14.2
[ 0.570198] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 0.570199] DMAR: Hardware identity mapping for device 0000:00:16.3
[ 0.570200] DMAR: Hardware identity mapping for device 0000:00:17.0
[ 0.570200] DMAR: Hardware identity mapping for device 0000:00:1b.0
[ 0.570201] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 0.570202] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 0.570202] DMAR: Hardware identity mapping for device 0000:00:1f.2
[ 0.570203] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 0.570204] DMAR: Hardware identity mapping for device 0000:00:1f.4
[ 0.570205] DMAR: Hardware identity mapping for device 0000:00:1f.6
[ 0.570206] DMAR: Hardware identity mapping for device 0000:02:00.0
[ 0.570207] DMAR: Setting RMRR:
[ 0.570208] DMAR: Ignoring identity map for HW passthrough device 0000:00:02.0 [0x8d800000 - 0x8fffffff]
[ 0.570208] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x8b762000 - 0x8b781fff]
[ 0.570209] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.570209] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.570223] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 0.570257] iommu: Adding device 0000:00:00.0 to group 0
[ 0.570265] iommu: Adding device 0000:00:01.0 to group 1
[ 0.570270] iommu: Adding device 0000:00:02.0 to group 2
[ 0.570277] iommu: Adding device 0000:00:14.0 to group 3
[ 0.570281] iommu: Adding device 0000:00:14.2 to group 3
[ 0.570289] iommu: Adding device 0000:00:16.0 to group 4
[ 0.570293] iommu: Adding device 0000:00:16.3 to group 4
[ 0.570297] iommu: Adding device 0000:00:17.0 to group 5
[ 0.570304] iommu: Adding device 0000:00:1b.0 to group 6
[ 0.570310] iommu: Adding device 0000:00:1c.0 to group 7
[ 0.570321] iommu: Adding device 0000:00:1f.0 to group 8
[ 0.570326] iommu: Adding device 0000:00:1f.2 to group 8
[ 0.570330] iommu: Adding device 0000:00:1f.3 to group 8
[ 0.570335] iommu: Adding device 0000:00:1f.4 to group 8
[ 0.570339] iommu: Adding device 0000:00:1f.6 to group 9
[ 0.570346] iommu: Adding device 0000:02:00.0 to group 10
[ 0.570353] iommu: Adding device 0000:03:00.0 to group 11

still the error I noticed before persists:
root@pve:~# qm start 100
kvm: vfio: Cannot reset device 0000:00:17.0, no available reset mechanism.
kvm: vfio: Cannot reset device 0000:00:17.0, no available reset mechanism.

Maybe this error is not a problems, I still think the passthrough is successful.

but now I noticed the VM is not supporting iommu version 2, I don't know what that means yet.

root@omv:~# dmesg | egrep -i "dmar|iommu"
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0-0.bpo.1-amd64 root=UUID=86416e85-8e46-4923-ab0c-7bba409257a3 ro quiet iommu=pt intel_iommu=on
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0-0.bpo.1-amd64 root=UUID=86416e85-8e46-4923-ab0c-7bba409257a3 ro quiet iommu=pt intel_iommu=on
[ 0.000000] DMAR: IOMMU enabled
[ 0.667885] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[ 0.667885] AMD IOMMUv2 functionality not available on this system


this iommu messages seem very limited on the VM. I have to figure out if that is normal.

also have to figure out if its normal for all pci devices to have the same subsystem:
"Subsystem: Dell SATA Controller [RAID mode] [1028:07c5]" 1028:07c5 is a subsystem for all pci devices.
 
scratch what I had written on this post before, I was looking at different dev devices (one was sdb on VM and turns out it's not sdb on host)
 
Last edited:
I'm going to ask a simpler question, why can't I have the sata controller use the vfio driver on bootup.

root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:a102
root@pve:~# lspci -knn | grep -A2 SATA
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
Subsystem: Dell Sunrise Point-H SATA controller [AHCI mode] [1028:07c5]
Kernel driver in use: vfio-pci
Kernel modules: ahci

As you can see it's using the vfio driver dynamically when I start a qm with hostpci. However, I want to have it do that on bootup. Is my vfio.conf settings not sufficients?
 
Hi
Sorry to use this 2 yo thread but...
... can I PCI Passthrough my SATA controller in order to see my HDD in a FreenNAS VM if the SSD that has the Proxmox also use it?
When I start the VM, the Proxmox stops working so I suspect that is the problem, right?

Regards
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back