Prevent a sata controller to be loaded by AHCI

E

etnicor

Member
Jan 19, 2023
42
2
8
Hello,
this may be more of a Linux question then Proxmox..

I have 2 sata controller where I want one to not be loaded by AHCI driver on proxmox host, since I want to pass through one to a VM.
How do I achieve that? I am guessing with udev somehow?

Any hints appreciated.

Cheers
 
etnicor said:
I have 2 sata controller where I want one to not be loaded by AHCI driver on proxmox host, since I want to pass through one to a VM.
Click to expand...
What are the lspci -nnk outputs for both controllers? Or are they the exact same make and model? Then it's much more complicated, otherwise early binding to vfio-pci and a softdep should do it.
 
leesteken said:
What are the lspci -nnk outputs for both controllers? Or are they the exact same make and model? Then it's much more complicated, otherwise early binding to vfio-pci and a softdep should do it.
Click to expand...
One is Intel Corporation Alder Lake-S which I assume is on the CPU?
2nd one is a pcie card with a asm1166 chipset.
*So assuming they will be in different iommu groups.

Just unplugged the pcie card.

However this is the sata controller I do not want to load(currrently have ahci blacklisted)

00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
DeviceName: Onboard - SATA
Subsystem: Gigabyte Technology Co., Ltd Alder Lake-S PCH SATA Controller [AHCI Mode] [1458:1000]
Kernel driver in use: vfio-pci
Kernel modules: ahci
 
etnicor said:
One is Intel Corporation Alder Lake-S which I assume is on the CPU?
2nd one is a pcie card with a asm1166 chipset.
*So assuming they will be in different iommu groups.
Click to expand...
That depends on the motherboard and can be dissapointing.
etnicor said:
Just unplugged the pcie card.

However this is the sata controller I do not want to load(currrently have ahci blacklisted)

00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
DeviceName: Onboard - SATA
Subsystem: Gigabyte Technology Co., Ltd Alder Lake-S PCH SATA Controller [AHCI Mode] [1458:1000]
Kernel driver in use: vfio-pci
Kernel modules: ahci
Click to expand...
Please show the lspci -nnk output of the other SATA controllers (all devices that use ahci) also; I don't want to accidentally disable other controllers that are needed by Proxmox.
 
Last edited:
Here comes the other one. =)

01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
Subsystem: ZyDAS Technology Corp. ASM1166 Serial ATA Controller [2116:2116]
Kernel modules: ahci
 
etnicor said:
One is Intel Corporation Alder Lake-S which I assume is on the CPU?
2nd one is a pcie card with a asm1166 chipset.
*So assuming they will be in different iommu groups.
Click to expand...
You can check the IOMMU groups (which are different from PCI IDs) using cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done (please show the whole output).
etnicor said:
However this is the sata controller I do not want to load(currrently have ahci blacklisted)

00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
DeviceName: Onboard - SATA
Subsystem: Gigabyte Technology Co., Ltd Alder Lake-S PCH SATA Controller [AHCI Mode] [1458:1000]
Kernel driver in use: vfio-pci
Kernel modules: ahci
Click to expand...
etnicor said:
Here comes the other one. =)

01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
Subsystem: ZyDAS Technology Corp. ASM1166 Serial ATA Controller [2116:2116]
Kernel modules: ahci
Click to expand...
Adding
options vfio-pci ids=8086:7ae2 softdep ahci pre: vfio-pci
to a file in the /etc/modprobe.d/ directory should do the trick. And of course update-initframfs etc. as shown in the manual. If you already did somethign similar, please combine the vfio-pci ids etc.
 
Here comes iommu output.
Looks like they are in separate groups.

Code: 
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c)
IOMMU group 10 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
IOMMU group 11 00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
IOMMU group 12 00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11)
IOMMU group 13 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac4] (rev 11)
IOMMU group 14 00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #6 [8086:7abd] (rev 11)
IOMMU group 15 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:7abe] (rev 11)
IOMMU group 16 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a88] (rev 11)
IOMMU group 16 00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
IOMMU group 16 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
IOMMU group 16 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
IOMMU group 17 01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
IOMMU group 18 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU group 19 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a80c]
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4650] (rev 05)
IOMMU group 20 04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller 980 [144d:a809]
IOMMU group 21 05:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013] (rev 01)
IOMMU group 22 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
IOMMU group 23 07:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 06)
IOMMU group 23 08:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 52)
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 [8086:460d] (rev 05)
IOMMU group 3 00:01.1 PCI bridge [0604]: Intel Corporation Device [8086:462d] (rev 05)
IOMMU group 4 00:04.0 Signal processing controller [1180]: Intel Corporation Alder Lake Innovation Platform Framework Processor Participant [8086:461d] (rev 05)
IOMMU group 5 00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 05)
IOMMU group 6 00:08.0 System peripheral [0880]: Intel Corporation 12th Gen Core Processor Gaussian & Neural Accelerator [8086:464f] (rev 05)
IOMMU group 7 00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01)
IOMMU group 8 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
IOMMU group 8 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
IOMMU group 9 00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)
 
etnicor said:
Here comes iommu output.
Looks like they are in separate groups.
Click to expand...
Yes but you did not show the whole output, which would tell if you're using pcie_acs_override. If that's the case then it might still not work as the group information is a lie. And it might still not work because sometimes devices just don't work with passthrough. It's all trial and error...
etnicor said:
IOMMU group 17 01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
Click to expand...
I guess, you can try it. Is the one SATA controller bound to vfio-pci automatically (lspci -nnks 01:00, driver in use is vfio-pci) after a host reboot without starting any VMs? Does it work from within the VM after that?
 
Seems to have worked.
Removed blacklist of ahci and added options and softdep in /etc/modprobe.d.

Code: 
00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
        DeviceName: Onboard - SATA
        Subsystem: Gigabyte Technology Co., Ltd Alder Lake-S PCH SATA Controller [AHCI Mode] [1458:1000]
        Kernel driver in use: vfio-pci
        Kernel modules: ahci
01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
        Subsystem: ZyDAS Technology Corp. ASM1166 Serial ATA Controller [2116:2116]
        Kernel driver in use: ahci
        Kernel modules: ahci

iommu output again.
Code: 
BOOT_IMAGE=/boot/vmlinuz-6.5.11-4-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c)
IOMMU group 10 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
IOMMU group 11 00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
IOMMU group 12 00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11)
IOMMU group 13 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac4] (rev 11)
IOMMU group 14 00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #6 [8086:7abd] (rev 11)
IOMMU group 15 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:7abe] (rev 11)
IOMMU group 16 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a88] (rev 11)
IOMMU group 16 00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
IOMMU group 16 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
IOMMU group 16 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
IOMMU group 17 01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
IOMMU group 18 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU group 19 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a80c]
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4650] (rev 05)
IOMMU group 20 04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller 980 [144d:a809]
IOMMU group 21 05:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013] (rev 01)
IOMMU group 22 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
IOMMU group 23 07:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 06)
IOMMU group 23 08:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 52)
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 [8086:460d] (rev 05)
IOMMU group 3 00:01.1 PCI bridge [0604]: Intel Corporation Device [8086:462d] (rev 05)
IOMMU group 4 00:04.0 Signal processing controller [1180]: Intel Corporation Alder Lake Innovation Platform Framework Processor Participant [8086:461d] (rev 05)
IOMMU group 5 00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 05)
IOMMU group 6 00:08.0 System peripheral [0880]: Intel Corporation 12th Gen Core Processor Gaussian & Neural Accelerator [8086:464f] (rev 05)
IOMMU group 7 00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01)
IOMMU group 8 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
IOMMU group 8 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
IOMMU group 9 00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)

Now the hope is that ASM1166 sata controller will work, it did not work using pci passthrough. So I'll use do single disk passthrough of disks on this card.
 
  • Like
Reactions: leesteken
I just wanted to let everyone know that I got the ASM1166 SATA controller working in passthrough after I blocked it from the host using the instructions above. Thanks!
 
I can't understand steps in this guide.
I have ASM1166 m2 to sata 6 port card. MB Minisforum BD770i, AMD Ryzen 7745hx. I passthrough iGPU after several attempts. Controller assigned to separate IOMMU group 16. I connected 3 disks from old NAS.

1) I use same softdep way for my asm1166 (added ,1b21:1166,2116:2116 to ids) on /etc/modprobe.d/vfio.conf:
Code: 
options vfio-pci ids=1002:164e,1002:1640,1b21:1166,2116:2116 disable_vga=1
softdep ahci pre:vfio-pci
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci

2) update initramfs, reboot

3) get same issue with ahci driver. without VM starting it is
Code: 
03:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
    Subsystem: ZyDAS Technology Corp. ASM1166 Serial ATA Controller
    Flags: bus master, fast devsel, latency 0, IRQ 98, IOMMU group 16
    Memory at fce82000 (32-bit, non-prefetchable) [size=8K]
    Memory at fce80000 (32-bit, non-prefetchable) [size=8K]
    Expansion ROM at fce00000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [80] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [130] Secondary PCI Express
    Kernel driver in use: ahci
    Kernel modules: ahci
after VM with passthrough started changing kernel driver in use:
Code: 
03:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
    Subsystem: ZyDAS Technology Corp. ASM1166 Serial ATA Controller
    Flags: bus master, fast devsel, latency 0, IRQ 98, IOMMU group 16
    Memory at fce82000 (32-bit, non-prefetchable) [size=8K]
    Memory at fce80000 (32-bit, non-prefetchable) [size=8K]
    Expansion ROM at fce00000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [80] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [130] Secondary PCI Express
    Kernel driver in use: vfio-pci
    Kernel modules: ahci

But after several initramfs and checking of vfio.conf found missing space in line with softdep ahci. Add space and Kernel driver in use: vfio-pci after load and without VM start.
Code: 
options vfio-pci ids=1002:164e,1002:1640,1b21:1166,2116:2116 disable_vga=1
softdep ahci pre: vfio-pci
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci

But still no luck trueNas don't see disks.

My VM conf:
Code: 
agent: 1
bios: ovmf
boot: order=scsi0
cores: 2
cpu: host
efidisk0: local-lvm:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00.0,pcie=1
ide2: local:iso/TrueNAS-13.1-MASTER-202402240814-1ca0010.iso,media=cdrom,size=960740K
machine: q35
memory: 8192
meta: creation-qemu=8.1.5,ctime=1711216835
name: truenas
net0: virtio=BC:24:11:F0:F1:EA,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-103-disk-1,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=01a2d10f-b8f3-4741-8b95-31b47e63e3bd
sockets: 1
vmgenid: 22f53f4b-cfc4-49dc-9d7b-8b0255379ac0

4) I looked at TrueNas core lspci and I see my controller as a device. Looks like passthrough working good.
Screenshot 2024-03-23 at 22.14.30.png
but still no disks
Screenshot 2024-03-23 at 22.12.41.png

So my quest isn't completed yet. and I'm open to suggestions.
 
Today I tried TrueNas Scale. Same result. OS sees controller but no disks attached. I ordered pci-e to the slot. But I'm sceptical on success, because looks like it has similar chip and next step is corporate oriented hardware (this is overkill for small home lab project).
 
FancyBee said:
Today I tried TrueNas Scale. Same result. OS sees controller but no disks attached. I ordered pci-e to the slot. But I'm sceptical on success, because looks like it has similar chip and next step is corporate oriented hardware (this is overkill for small home lab project).
Click to expand...

I used a PCIe card when I got it working.
 
MB have only one pcie 5.0 16x slot empty because it is ITX. I tried pcie 1x version of controller in 16x slots. with UEFI bios it is just halt Proxmox and Windows in the middle of the boot. One of side effects - network adapter don't init properly, or switch off. Link and activity leds light up in the begin of boot, then switched off. I can put card in the m2 a-e card slot, but adapter delivery will be several week.

I do original tests with TrueNas on UEFI bios. After switch to Seabios, same result. VM detects adapter but not connected drive.

I write to ASMedia service for driver, or any fresh firmware or some kind of recipe to solve it.

@smefa @leesteken @etnicor can we try to compile instruction?
As I undestand your instructions from messages above:
Check kernel module check IOMMU groupping and in case controller have same group with other device
to check it execute:
Code: 
cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

In case of IOMMU grouping issue (controller share IOMMU group with something else):
1) edit /etc/default/grub:
add to parameter GRUB_CMDLINE_LINUX_DEFAULT options to end of the string " iommu=pt pcie_acs_override=downstream,multifunction"
I don't have grouping issue so my version is:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"
If you have problems with IOMMU groups it should be:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream,multifunction"
2)update boot images with command: sudo update-grub

If grouping issue is fixed and controller have uniq group, than continue with drivers switch from ahci to vfio-pci
1) get vendor and device ids from lspci -nnk for controller
2) make a softdep in /etc/modprobe.d/vfio.conf to switch for vfio-pci drivers
Code: 
options vfio-pci ids=1b21:1166 
softdep ahci pre: vfio-pci
if you have GPU passthrough than like I do you get something like:
Code: 
options vfio-pci ids=1002:164e,1002:1640,1b21:1166,2116:2116 disable_vga=1
softdep ahci pre: vfio-pci
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
3) update boot with: update-initramfs -u -k all
4) reboot server
5) switch bios SeaBIOS, and passthrough raw pci device.
5) in new VM you should detect controller and drives.

Did I miss something?
 
FancyBee said:
In case of IOMMU grouping issue (controller share IOMMU group with something else):
1) edit /etc/default/grub:
add to parameter GRUB_CMDLINE_LINUX_DEFAULT options to end of the string " iommu=pt pcie_acs_override=downstream,multifunction"
Click to expand...
I don't think it's a good idea to use pcie_acs_override and I would suggest to use different PCIe slot and/or motherboard instead. Also note that not every Proxmox uses GRUB.
FancyBee said:
5) switch bios SeaBIOS,
Click to expand...
Why? Using SeaBIOS or OVMF is mostly independent of passthrough. One or the other is sometimes needed when the device ROM does not support BIOS or UEFI.
FancyBee said:
and passthrough raw pci device.
Click to expand...
Note that raw or mapped passthrough are the same, except the latter uses a global name mapping to raw devices defined by the Proxmox user.
FancyBee said:
5) in new VM you should detect controller and drives.
Click to expand...
Only if the device works with passthrough, which is not always the case. Sometimes devices need more quirks or work-arounds or (patched) ROM-files.
 
leesteken said:
Why? Using SeaBIOS or OVMF is mostly independent of passthrough. One or the other is sometimes needed when the device ROM does not support BIOS or UEFI.
Click to expand...
In several guides on Debian and Ubuntu, ASM1166 was not working on UEFI bios images. I tried both, this trick don't work for me.
I forget about bios dump.... How to dump bios from sata controller?
leesteken said:
I don't think it's a good idea to use pcie_acs_override and I would suggest to use different PCIe slot and/or motherboard instead. Also note that not every Proxmox uses GRUB.
Click to expand...
I tried additional grub kernel options, it don't work for me. I get a separate iommu grouping without it, and passthrough of igpu and gpu in 16x was working.

Most ITX motherboards used for small homelab NAS usually have few m2 slots and one 16x. Most itx boards have 2 or 4 sata ports, but some new boards on mobile cpus don't have any sata ports. If I build it with bigger factor like matx(in some there are one 16x and one 1x or 4x slot), or atx (one 16x and 1-2 of 1x or 4x),... than I will have an option to move card between slots.
My mb don't have any onboard sata. In my case I have 2 m2 m-key slots, one m2 e-key (wifi card) slot and one pcie 16x for gpu. I tried m2 to sata 6 port controller on ASM1166 in different m2 m-key slots. Each time it get separate iommu group, no issue here. I don't try to put it to m2 e-key slot with adapter. I don't have adapter for this and I need this wifi card to work.

I buy ASM1166 on pcie 1x card with 6 ports and put it into 16x slot. It halt the boot of Proxmox and mess with network card.

My mb bios setting is ascetic at best, so I can't try several instructions based on bios turning or altering, flashing different firmware versions.

My passthrough is working. Card have 6 ports but report 32 devices in dmesg and all 32 don't response to requests. system detects 38 sata devices. ACHI one slot, but no devices. my dmesg|grep SATA
[ 0.824993] ahci 0000:00:1f.2: AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode
[ 0.840726] ata1: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b100 irq 33
[ 0.840939] ata2: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b180 irq 33
[ 0.841286] ata3: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b200 irq 33
[ 0.841638] ata4: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b280 irq 33
[ 0.842067] ata5: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b300 irq 33
[ 0.842491] ata6: SATA max UDMA/133 abar m4096@0xfea1b000 port 0xfea1b380 irq 33
[ 0.870953] ahci 0000:06:10.0: AHCI 0001.0301 32 slots 32 ports 6 Gbps 0xffffff3f impl SATA mode
[ 0.884560] ata7: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2100 irq 34
[ 0.884856] ata8: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2180 irq 34
[ 0.885180] ata9: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2200 irq 34
[ 0.885482] ata10: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2280 irq 34
[ 0.885825] ata11: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2300 irq 34
[ 0.886235] ata12: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2380 irq 34
[ 0.887152] ata15: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2500 irq 34
[ 0.887559] ata16: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2580 irq 34
[ 0.887897] ata17: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2600 irq 34
[ 0.888271] ata18: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2680 irq 34
[ 0.888612] ata19: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2700 irq 34
[ 0.888825] ata20: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2780 irq 34
[ 0.889035] ata21: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2800 irq 34
[ 0.889368] ata22: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2880 irq 34
[ 0.889699] ata23: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2900 irq 34
[ 0.889998] ata24: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2980 irq 34
[ 0.890342] ata25: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2a00 irq 34
[ 0.890753] ata26: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2a80 irq 34
[ 0.891093] ata27: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2b00 irq 34
[ 0.891452] ata28: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2b80 irq 34
[ 0.891874] ata29: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2c00 irq 34
[ 0.892289] ata30: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2c80 irq 34
[ 0.892638] ata31: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2d00 irq 34
[ 0.893010] ata32: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2d80 irq 34
[ 0.899958] ata33: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2e00 irq 34
[ 0.900302] ata34: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2e80 irq 34
[ 0.900632] ata35: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2f00 irq 34
[ 0.900990] ata36: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec2f80 irq 34
[ 0.901349] ata37: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec3000 irq 34
[ 0.901760] ata38: SATA max UDMA/133 abar m8192@0xfdec2000 port 0xfdec3080 irq 34
[ 1.156354] ata4: SATA link down (SStatus 0 SControl 300)
[ 1.176553] ata3: SATA link down (SStatus 0 SControl 300)
[ 1.176791] ata6: SATA link down (SStatus 0 SControl 300)
[ 1.176997] ata5: SATA link down (SStatus 0 SControl 300)
[ 1.177206] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 1.177463] ata1: SATA link down (SStatus 0 SControl 300)
[ 1.216919] ata7: SATA link down (SStatus 0 SControl 300)
[ 1.531030] ata8: SATA link down (SStatus 0 SControl 300)
[ 1.846770] ata9: SATA link down (SStatus 0 SControl 300)
[ 2.161195] ata10: SATA link down (SStatus 0 SControl 300)
[ 2.477323] ata11: SATA link down (SStatus 0 SControl 300)
[ 2.792968] ata12: SATA link down (SStatus 0 SControl 300)
[ 3.108320] ata15: SATA link down (SStatus 0 SControl 300)
[ 3.420633] ata16: SATA link down (SStatus 0 SControl 300)
[ 3.736984] ata17: SATA link down (SStatus 0 SControl 300)
[ 4.050748] ata18: SATA link down (SStatus 0 SControl 300)
[ 4.369588] ata19: SATA link down (SStatus 0 SControl 300)
[ 4.680351] ata20: SATA link down (SStatus 0 SControl 300)
[ 4.992361] ata21: SATA link down (SStatus 0 SControl 300)
[ 5.309484] ata22: SATA link down (SStatus 0 SControl 300)
[ 5.625304] ata23: SATA link down (SStatus 0 SControl 300)
[ 5.936465] ata24: SATA link down (SStatus 0 SControl 300)
[ 6.249448] ata25: SATA link down (SStatus 0 SControl 300)
[ 6.561104] ata26: SATA link down (SStatus 0 SControl 300)
[ 6.872357] ata27: SATA link down (SStatus 0 SControl 300)
[ 7.188467] ata28: SATA link down (SStatus 0 SControl 300)
[ 7.505461] ata29: SATA link down (SStatus 0 SControl 300)
[ 7.821654] ata30: SATA link down (SStatus 0 SControl 300)
[ 8.137477] ata31: SATA link down (SStatus 0 SControl 300)
[ 8.451182] ata32: SATA link down (SStatus 0 SControl 300)
[ 8.764879] ata33: SATA link down (SStatus 0 SControl 300)
[ 9.077463] ata34: SATA link down (SStatus 0 SControl 300)
[ 9.392788] ata35: SATA link down (SStatus 0 SControl 300)
[ 9.704721] ata36: SATA link down (SStatus 0 SControl 300)
[ 10.016476] ata37: SATA link down (SStatus 0 SControl 300)
[ 10.333502] ata38: SATA link down (SStatus 0 SControl 300)

Grep AHCI related
[ 0.824993] ahci 0000:00:1f.2: AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode
[ 0.870953] ahci 0000:06:10.0: AHCI 0001.0301 32 slots 32 ports 6 Gbps 0xffffff3f impl SATA mode
 
These are my notes from my install..
I guess you need some reboots as well

# Check which boot config to edit
https://forum.proxmox.com/threads/p...x-ve-8-installation-and-configuration.130218/

Add parameters below to bootloader according to link above
intel_iommu=on iommu=pt

Update with the correct command in link above

Add these
nano /etc/modules
# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci

#Prevent sata card to be used by system
Add the two rows below for ASM1166 ahcpi in:
nano /etc/modprobe.d/ahci-blacklist.conf
options vfio-pci ids=1b21:1166
softdep ahci pre: vfio-pci

update with
update-initramfs -u -k all

Check driver used: lspci -nnk
Should be this for the controller
Kernel driver in use: vfio-pci
Kernel modules: ahci
 
I have a Asm1166 M2 card with 6 sata slots, tried
options vfio-pci ids=1002:164e,1002:1640,1b21:1166,2116:2116 disable_vga=1
softdep ahci pre: vfio-pci

followed by update initramfs, and executed lspci -nnk, says vfio-pci in use.

But when I create a new vm, passthrough the device and started the vm, it just hung in there, and I had to restart the vm.

I guess I'll just passthrough the disks for now...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom