issue:
PCIe to SATA card (ASM1061 inside), when add this card as PCIe Passthrough in VM, PVE Host crashed
hardware summary:
CPU: E3-1225 V2
MB: Dell 9010mt Q77(modified BIOS, support start OS from the PCIe to NVME card).
PVE: 7.3-6 is installed in the PCIe to NVME card.
Memory: 4 * 4GB.
PCIe: PCIe 16x(no card), PCIe 4x(PCIe to NVME), PCIe 1x(PCIe to SATA). also tried plug this card into PCIe 16x, also got crashing,the PCIe to SATA card is:
PCIe to SATA card (ASM1061 inside), when add this card as PCIe Passthrough in VM, PVE Host crashed
hardware summary:
CPU: E3-1225 V2
MB: Dell 9010mt Q77(modified BIOS, support start OS from the PCIe to NVME card).
PVE: 7.3-6 is installed in the PCIe to NVME card.
Memory: 4 * 4GB.
PCIe: PCIe 16x(no card), PCIe 4x(PCIe to NVME), PCIe 1x(PCIe to SATA). also tried plug this card into PCIe 16x, also got crashing,the PCIe to SATA card is:
Code:
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 29
IOMMU group: 13
Region 0: I/O ports at e050 [size=8]
Region 1: I/O ports at e040 [size=4]
Region 2: I/O ports at e030 [size=8]
Region 3: I/O ports at e020 [size=4]
Region 4: I/O ports at e000 [size=32]
Region 5: Memory at f7d10000 (32-bit, non-prefetchable) [size=512]
Expansion ROM at f7d00000 [disabled] [size=64K]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002d8 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Kernel driver in use: ahci
Kernel modules: ahci
Code:
Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction
Code:
root@Q77:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.013432] ACPI: DMAR 0x00000000D7FFEDB8 0000B8 (v01 INTEL SNB 00000001 INTL 00000001)
[ 0.013459] ACPI: Reserving DMAR table memory at [mem 0xd7ffedb8-0xd7ffee6f]
[ 0.042044] DMAR: IOMMU enabled
[ 0.118620] DMAR: Host address width 36
[ 0.118621] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.118626] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[ 0.118628] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.118631] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[ 0.118632] DMAR: RMRR base: 0x000000daf77000 end: 0x000000daf9dfff
[ 0.118634] DMAR: RMRR base: 0x000000db800000 end: 0x000000df9fffff
[ 0.118636] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.118637] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.118638] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.119129] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.266769] DMAR: No ATSR found
[ 0.266770] DMAR: No SATC found
[ 0.266771] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.266773] DMAR: IOMMU feature pass_through inconsistent
[ 0.266774] DMAR: dmar0: Using Queued invalidation
[ 0.266779] DMAR: dmar1: Using Queued invalidation
[ 0.333148] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 3.911191] i915 0000:00:02.0: [drm] DMAR active, disabling use of stolen memory
root@Q77:~#
Code:
root@Q77:~# dmesg | grep 'remapping'
[ 0.118638] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.119129] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@Q77:~#
Code:
root@Q77:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:1a.0
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.2
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:19.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
root@Q77:~#
Code:
root@Q77:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation Q77 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
03:00.0 Non-Volatile memory controller: Solid State Storage Technology Corporation Device 9100 (rev 03)
Code:
root@Q77:~# lspci -vvt
-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
+-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
+-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
+-16.0 Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1
+-19.0 Intel Corporation 82579LM Gigabit Network Connection (Lewisville)
+-1a.0 Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2
+-1b.0 Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller
+-1c.0-[01]--
+-1c.2-[02]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller
+-1c.4-[03]----00.0 Solid State Storage Technology Corporation Device 9100
+-1d.0 Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1
+-1e.0-[04]--
+-1f.0 Intel Corporation Q77 Express Chipset LPC Controller
+-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
-1f.3 Intel Corporation 7 Series/C216 Chipset Family SMBus Controller
Code:
root@Q77:~# qm config 104
boot: order=sata0;ide2;net0
cores: 2
description: wqf%0A123456
hostpci0: 0000:02:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=7.1.0,ctime=1670485787
name: Ubuntu
net0: virtio=22:82:F8:E1:6C:83,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
sata0: local-lvm:vm-104-disk-0,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=4c074828-b68c-4b12-95b7-74ddff8374e8
sockets: 1
vmgenid: 237f068b-8811-4486-9de7-284eac69bb6e
root@Q77:~#
Code:
root@Q77:~# ./iommu.sh
Group 0: [8086:0158] 00:00.0 Host bridge Xeon E3-1200 v2/Ivy Bridge DRAM Controller
Group 1: [8086:016a] [R] 00:02.0 VGA compatible controller Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
Group 2: [8086:1e31] 00:14.0 USB controller 7 Series/C210 Series Chipset Family USB xHCI Host Controller
USB: [1d6b:0002] Bus 001 Device 001 Linux Foundation 2.0 root hub
USB: [1d6b:0003] Bus 002 Device 001 Linux Foundation 3.0 root hub
Group 3: [8086:1e3a] 00:16.0 Communication controller 7 Series/C216 Chipset Family MEI Controller #1
Group 4: [8086:1502] [R] 00:19.0 Ethernet controller 82579LM Gigabit Network Connection (Lewisville)
Group 5: [8086:1e2d] [R] 00:1a.0 USB controller 7 Series/C216 Chipset Family USB Enhanced Host Controller #2
USB: [8087:0024] Bus 003 Device 002 Intel Corp. Integrated Rate Matching Hub
USB: [1d6b:0002] Bus 003 Device 001 Linux Foundation 2.0 root hub
Group 6: [8086:1e20] [R] 00:1b.0 Audio device 7 Series/C216 Chipset Family High Definition Audio Controller
Group 7: [8086:1e10] [R] 00:1c.0 PCI bridge 7 Series/C216 Chipset Family PCI Express Root Port 1
Group 8: [8086:1e14] [R] 00:1c.2 PCI bridge 7 Series/C210 Series Chipset Family PCI Express Root Port 3
Group 9: [8086:1e18] [R] 00:1c.4 PCI bridge 7 Series/C210 Series Chipset Family PCI Express Root Port 5
Group 10: [8086:1e26] [R] 00:1d.0 USB controller 7 Series/C216 Chipset Family USB Enhanced Host Controller #1
USB: [8087:0024] Bus 004 Device 002 Intel Corp. Integrated Rate Matching Hub
USB: [1d6b:0002] Bus 004 Device 001 Linux Foundation 2.0 root hub
Group 11: [8086:244e] 00:1e.0 PCI bridge 82801 PCI Bridge
Group 12: [8086:1e47] 00:1f.0 ISA bridge Q77 Express Chipset LPC Controller
[8086:1e02] [R] 00:1f.2 SATA controller 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
[8086:1e22] 00:1f.3 SMBus 7 Series/C216 Chipset Family SMBus Controller
Group 13: [1b21:0612] [R] 02:00.0 SATA controller ASM1062 Serial ATA Controller
Group 14: [1e95:9100] [R] 03:00.0 Non-Volatile memory controller Device 9100
root@Q77:~#
Last edited: