PCIe Passthrough Sata Controllers ASM1062

R

Rehs

New Member
Dec 21, 2019
1
0
1
37
Hello All,

I'm new to VM's and Proxmox and I'm looking for some help to setup my home server via passing through 7 drives to a VM. I'm using this PCIe card which uses a sata controller (ASM1062 x4). However, it only seems to recognize when I'm not running with intel_iommu=on. I was able to pass through four drives successfully with a (Marvel 9215 x1) in the same machine earlier and had no issues with intel_iommu=on.

I'm trying to pass through the controllers in my IOMMU group 1. (The Marvel 9215 card was also group 1)
Code: 
root@pve:/tmp# bash iommu.sh
IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Workstation Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e18] (rev 07)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
        01:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1806] (rev 01)
        02:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1806] (rev ff)
        02:02.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1806] (rev ff)
        02:06.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1806] (rev ff)
        02:0e.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1806] (rev ff)
        03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev ff)
        04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev ff)
        05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev ff)
        06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev ff)
IOMMU Group 10:
        00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a309] (rev 10)
        00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
        00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
        00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
IOMMU Group 11:
        08:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 2:
        00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3e96]
IOMMU Group 3:
        00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
IOMMU Group 4:
        00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 5:
        00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
        00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 6:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller [8086:a368] (rev 10)
IOMMU Group 7:
        00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 8:
        00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a330] (rev f0)
IOMMU Group 9:
        00:1d.2 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a332] (rev f0)

The first error codes received on boot:
Code: 
[    3.888075]  sdh: sdh1 sdh9
[    3.888828] sd 12:0:0:0: [sdh] Attached SCSI disk
[    4.278617] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[    5.289349] pcieport 0000:00:01.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[    5.289389] pcieport 0000:00:01.0: AER:   device [8086:1901] error status/mask=00000001/00002000
[    5.289390] pcieport 0000:00:01.0: AER:    [ 0] RxErr              
[    5.289391] pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: 0000:00:01.0
[    5.289432] pcieport 0000:00:01.0: AER: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Requester ID)
[    5.289463] pcieport 0000:00:01.0: AER:   device [8086:1901] error status/mask=00004000/00000000
[    5.289478] pcieport 0000:00:01.0: AER:    [14] CmpltTO                (First)
[    5.301508] ata11.00: exception Emask 0x73 SAct 0x3800 SErr 0xffffffff action 0xe frozen
[    5.301581] ata11.00: irq_stat 0xffffffff, unknown FIS 00000000 00000000 00000000 00000000, host bus
[    5.301609] ata11: SError: { RecovData RecovComm UnrecovData Persist Proto HostInt PHYRdyChg PHYInt CommWake 10B8B Dispar BadCRC Handshk LinkSeq TrStaTrns UnrecFIS DevExch }
[    5.301634] ata11.00: failed command: READ FPDMA QUEUED
[    5.301645] ata11.00: cmd 60/00:58:00:00:00/01:00:00:00:00/40 tag 11 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301670] ata11.00: status: { DRDY }
[    5.301678] ata11.00: failed command: READ FPDMA QUEUED
[    5.301689] ata11.00: cmd 60/00:60:00:08:00/01:00:00:00:00/40 tag 12 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301714] ata11.00: status: { DRDY }
[    5.301721] ata11.00: failed command: READ FPDMA QUEUED
[    5.301731] ata11.00: cmd 60/00:68:00:b8:ff/01:00:74:05:00/40 tag 13 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301756] ata11.00: status: { DRDY }
[    5.301765] ata11: hard resetting link
[    5.301766] ahci 0000:05:00.0: AHCI controller unavailable!
[    5.301779] ata10.00: exception Emask 0x73 SAct 0x38 SErr 0xffffffff action 0xe frozen
[    5.301807] ata10.00: irq_stat 0xffffffff, unknown FIS 00000000 00000000 00000000 00000000, host bus
[    5.301822] ata10: SError: { RecovData RecovComm UnrecovData Persist Proto HostInt PHYRdyChg PHYInt CommWake 10B8B Dispar BadCRC Handshk LinkSeq TrStaTrns UnrecFIS DevExch }
[    5.301846] ata10.00: failed command: READ FPDMA QUEUED
[    5.301857] ata10.00: cmd 60/00:18:00:00:00/01:00:00:00:00/40 tag 3 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301881] ata10.00: status: { DRDY }
[    5.301889] ata10.00: failed command: READ FPDMA QUEUED
[    5.301899] ata10.00: cmd 60/00:20:00:08:00/01:00:00:00:00/40 tag 4 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301923] ata10.00: status: { DRDY }
[    5.301934] ata10.00: failed command: READ FPDMA QUEUED
[    5.301944] ata10.00: cmd 60/00:28:00:b8:ff/01:00:74:05:00/40 tag 5 ncq dma 131072 in
                        res 40/00:00:00:b8:ff/00:00:74:05:00/40 Emask 0x72 (host bus error)
[    5.301968] ata10.00: status: { DRDY }
[    5.301977] ata10: hard resetting link
[    5.301978] ahci 0000:05:00.0: AHCI controller unavailable!
[    6.341731] ata11: failed to resume link (SControl FFFFFFFF)
[    6.341738] ata11: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[    6.349731] pcieport 0000:00:01.0: AER: Root Port link has been reset
[    6.349745] pcieport 0000:00:01.0: AER: Device recovery failed
[   11.373745] ata11: hard resetting link
[   11.373748] ahci 0000:05:00.0: AHCI controller unavailable!
[   11.373789] ata10: failed to resume link (SControl FFFFFFFF)
[   11.373808] ata10: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[   16.493747] ata10: hard resetting link
[   16.493749] ahci 0000:05:00.0: AHCI controller unavailable!
[   17.533749] ata11: failed to resume link (SControl FFFFFFFF)
[   17.533756] ata11: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[   17.533760] ata11: limiting SATA link speed to <unknown>
[   22.637752] ata11: hard resetting link
[   22.637755] ahci 0000:05:00.0: AHCI controller unavailable!
[   22.637798] ata10: failed to resume link (SControl FFFFFFFF)
[   22.637817] ata10: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[   22.637821] ata10: limiting SATA link speed to <unknown>
[   27.757698] ata10: hard resetting link
[   27.757700] ahci 0000:05:00.0: AHCI controller unavailable!
[   28.797747] ata11: failed to resume link (SControl FFFFFFFF)
[   28.797754] ata11: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[   28.797758] ata11.00: disabled
[   28.797764] ahci 0000:05:00.0: AHCI controller unavailable!
[   28.797779] ata10: failed to resume link (SControl FFFFFFFF)

What gets me is that most reboots proxmox sees only my onboard NVME (sda), but occasionally some of the drives will show up under my pve disks (e.g. 4 of the 7 disks appear). I've never seen all 7 successfully show up with iommu=on. I've also tried allowing interrupts, but this didn't seem to do much.
(iommu_unsafe_interrupts.conf / kvm_iommu_map_guest.conf). And sometimes proxmox won't even boot up all the way and hangs at: pcieport 0000:00:01.0: AER: [ 0] RxErr.

Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
(I've also tried adding iommu=pt w/o any luck here)

Also tried adding the below to a file in /etc/modprobe.d to use vfio-pci to no avail:
Code: 
options vfio-pci ids=1b21:0612

If I remove iommu from grub the disks and controllers seem to function correctly within proxmox. I only seem to be having issues when trying to passthrough. Any ideas or help here would be appreciated!!
 
Last edited:
What does lspci -tv show? It may need to be blacklisted, before it can be passed through. Also, is the bios up-to-date? And last but not least, you can add the device path of the disk to the config. This may be a workaround if passthrough isn't possible
 
Alwin said:
What does lspci -tv show? It may need to be blacklisted, before it can be passed through.
Click to expand...
Hi,

although this is not my thread I have a similar issue. How can I blacklist my ASM1062 controller?
Code: 
root@proxmox:~# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Gemini Lake Host Bridge
           +-00.1  Intel Corporation Celeron/Pentium Silver Processor Dynamic Platform and Thermal Framework Processor Participant
           +-02.0  Intel Corporation GeminiLake [UHD Graphics 605]
           +-0e.0  Intel Corporation Celeron/Pentium Silver Processor High Definition Audio
           +-0f.0  Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface
           +-12.0  Intel Corporation Celeron/Pentium Silver Processor SATA Controller
           +-13.0-[01]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
           |            \-00.1  Intel Corporation 82576 Gigabit Network Connection
           +-13.1-[02]--
           +-13.2-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           +-13.3-[04]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           +-15.0  Intel Corporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller
           +-1f.0  Intel Corporation Celeron/Pentium Silver Processor LPC Controller
           \-1f.1  Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model
 
options vfio_pci ids=1b21:0612 in a .conf-file in /etc/modprobe.d/ should do the trick (don't forget to run update-initramfs -u).
However, maybe the driver (which one you can find out with lspci -k) might be loaded before vfio_pci. You might need to add softdep ahci pre: vfio_pci. I don't think you want to blacklist ahci in general for all devices.

EDIT: Somewhere else on this forum, I noticed that the driver is ahci and updated this post accordingly.
 
Last edited:
Both SATA controllers use the same kernel module "ahci", so I guess blacklisting the module is not an option.
Which IDs should I use for options vfio_pci ids=1b21:0612 in my system? 0000:0400?
 
npu said:
Both SATA controllers use the same kernel module "ahci", so I guess blacklisting the module is not an option.
Which IDs should I use for options vfio_pci ids=1b21:0612 in my system? 0000:0400?
Click to expand...
Can we please move this you your own thread?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back