Prox4 - Pcie passthrough - loosing pcie devices on boot

Q

Q-wulf

Renowned Member
Mar 3, 2013
613
39
93
my test location
I followed the following guide:

https://pve.proxmox.com/wiki/Pci_passthrough

once i execute the following part of the guide:

edit:#vi /etc/default/grub

changeGRUB_CMDLINE_LINUX_DEFAULT="quiet"

toGRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

then# update-grub
# reboot
Click to expand...

i get the following error on reboot:

Screen Shot 2015-10-28 at 22.32.45.jpg

when i execute

Code: 
dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000B9DC9310 0000B4 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.076102] DMAR: Host address width 46
[    0.076104] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.076112] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0462 ecap f020ff
[    0.076113] DMAR: RMRR base: 0x000000b9a77000 end: 0x000000b9a83fff
[    0.076114] DMAR: ATSR flags: 0x0
[    0.076115] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.076117] DMAR-IR: IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.076118] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.076119] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.076324] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.646476] DMAR: dmar0: Using Queued invalidation
[    0.646484] DMAR: Setting RMRR:
[    0.646493] DMAR: Setting identity map for device 0000:00:1a.0 [0xb9a77000 - 0xb9a83fff]
[    0.646504] DMAR: Setting identity map for device 0000:00:1d.0 [0xb9a77000 - 0xb9a83fff]
[    0.646508] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.646514] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.646518] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.491414] DMAR: DRHD: handling fault status reg 2
[    1.491430] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[    1.706450] DMAR: DRHD: handling fault status reg 102
[    1.706465] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[    1.882715] DMAR: DRHD: handling fault status reg 202
[    1.882729] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[    2.085657] DMAR: DRHD: handling fault status reg 302
[    2.085671] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[    5.052440] DMAR: DRHD: handling fault status reg 402
[    5.052456] DMAR: DMAR:[DMA Read] Request device [03:00.0] fault addr b9221000 
DMAR:[fault reason 01] Present bit in root entry is clear
[    7.040400] DMAR: DRHD: handling fault status reg 502
[    7.040417] DMAR: DMAR:[DMA Read] Request device [03:00.0] fault addr b9221000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   12.611242] DMAR: DRHD: handling fault status reg 602
[   12.611258] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   12.831820] DMAR: DRHD: handling fault status reg 702
[   12.831837] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   13.003815] DMAR: DRHD: handling fault status reg 2
[   13.003831] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   13.205852] DMAR: DRHD: handling fault status reg 102
[   13.205869] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   23.787045] DMAR: DRHD: handling fault status reg 202
[   23.787062] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   24.007131] DMAR: DRHD: handling fault status reg 302
[   24.007148] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   24.179927] DMAR: DRHD: handling fault status reg 402
[   24.179944] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   24.395113] DMAR: DRHD: handling fault status reg 502
[   24.395130] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   35.020072] DMAR: DRHD: handling fault status reg 602
[   35.020088] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   35.240777] DMAR: DRHD: handling fault status reg 702
[   35.241425] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   35.408119] DMAR: DRHD: handling fault status reg 2
[   35.408776] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   35.621399] DMAR: DRHD: handling fault status reg 102
[   35.622056] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   46.380216] DMAR: DRHD: handling fault status reg 202
[   46.380873] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   46.599426] DMAR: DRHD: handling fault status reg 302
[   46.600079] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   46.772392] DMAR: DRHD: handling fault status reg 402
[   46.773048] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   47.034842] DMAR: DRHD: handling fault status reg 502
[   47.035498] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   57.584096] DMAR: DRHD: handling fault status reg 602
[   57.584755] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   57.808051] DMAR: DRHD: handling fault status reg 702
[   57.808708] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   57.976532] DMAR: DRHD: handling fault status reg 2
[   57.977186] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   58.196034] DMAR: DRHD: handling fault status reg 102
[   58.196687] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   68.776021] DMAR: DRHD: handling fault status reg 202
[   68.776674] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   69.000018] DMAR: DRHD: handling fault status reg 302
[   69.000669] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   69.168642] DMAR: DRHD: handling fault status reg 402
[   69.169294] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   69.401406] DMAR: DRHD: handling fault status reg 502
[   69.402058] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   80.024008] DMAR: DRHD: handling fault status reg 602
[   80.024661] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   80.250337] DMAR: DRHD: handling fault status reg 702
[   80.250987] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   80.416839] DMAR: DRHD: handling fault status reg 2
[   80.417488] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   80.631588] DMAR: DRHD: handling fault status reg 102
[   80.632237] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   91.283990] DMAR: DRHD: handling fault status reg 202
[   91.284641] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   91.508981] DMAR: DRHD: handling fault status reg 302
[   91.509632] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   91.672052] DMAR: DRHD: handling fault status reg 402
[   91.672701] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[   91.871926] DMAR: DRHD: handling fault status reg 502
[   91.872573] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  102.575976] DMAR: DRHD: handling fault status reg 602
[  102.576626] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  102.800978] DMAR: DRHD: handling fault status reg 702
[  102.801625] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  102.964295] DMAR: DRHD: handling fault status reg 2
[  102.964943] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  103.205305] DMAR: DRHD: handling fault status reg 102
[  103.205952] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  113.772903] DMAR: DRHD: handling fault status reg 202
[  113.773551] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f0000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  113.992965] DMAR: DRHD: handling fault status reg 302
[  113.993611] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f1000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  114.160404] DMAR: DRHD: handling fault status reg 402
[  114.161052] DMAR: DMAR:[DMA Write] Request device [03:00.1] fault addr ad8f2000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  114.415543] DMAR: DRHD: handling fault status reg 502
[  114.416191] DMAR: DMAR:[DMA Write] Request device [03:00.0] fault addr ad8f3000 
DMAR:[fault reason 01] Present bit in root entry is clear
[  173.319533] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[  173.319535] AMD IOMMUv2 functionality not available on this system


lspci lists the following:
lspci -vv
03:00.0 RAID bus controller: HighPoint Technologies, Inc. RocketRAID 640L 4 Port SATA-III Controller (rev 01)
Subsystem: HighPoint Technologies, Inc. RocketRAID 640L 4 Port SATA-III Controller
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at d050
I/O ports at d040
I/O ports at d030
I/O ports at d020
I/O ports at d000
Memory at fbd10000 (32-bit, non-prefetchable) [size=2K]
Expansion ROM at fbd00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [70] Express Legacy Endpoint, MSI 00
Capabilities: [e0] SATA HBA v0.0
Capabilities: [100] Advanced Error Reporting
Click to expand...


Problem is, the Raidcontroller drops its /dev/sd*'s.

If i #vi /etc/default/grub
changeGRUB_CMDLINE_LINUX_DEFAULT="quiet"


again, there is no more issue with the Raid-controller.
It should be said, i'm not even trying to passthroug the Raidcontroller on 03:00.0



anyone know how i can fix this ?
 
i tried
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=pt"

doesn't throw the errors, but it also does not let me pass a the pcie wireless network adapter via

machine: q35
hostpci0: 05:00.0,pcie=1,driver=vfio
Click to expand...
do i need to pass the pcie the same way as with intel_iommo=on ??

Proxmox error on start of VM says:

unknown hostpci setting 'driver=vfio'
unknown hostpci setting 'driver=vfio'
TASK ERROR: Cannot open iommu_group: No such file or directory
Click to expand...


just an fyi :
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd Device b000

07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd Device b000

08:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd Device b000
Click to expand...

those are not dropping their Disks. even tho listed in the bug report.
 
Last edited:
those device listed in the bug already included in the kernel patch and made to the upstream source code already.

your dmesg | grep -e DMAR -e IOMMU output is very similiar described in the bug, you will need to find the vendor id and device id either a) submit them and wait for the code made to upstream b)download the patch, download pve kernel source, manually add your ids into the patch and compile your own kernel
 
is it this one ???
1103:0641


3:00.0 RAID bus controller [0104]: HighPoint Technologies, Inc. RocketRAID 640L 4 Port SATA-III Controller [1103:0641] (rev 01)
Subsystem: HighPoint Technologies, Inc. RocketRAID 640L 4 Port SATA-III Controller [1103:0641]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at d050
Region 1: I/O ports at d040
Region 2: I/O ports at d030
Region 3: I/O ports at d020
Region 4: I/O ports at d000
Region 5: Memory at fbd10000 (32-bit, non-prefetchable) [size=2K]
Expansion ROM at fbd00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Click to expand...
 
I think i might be way over my head trying to build my own kernel to fix that issue.
Might be better off dumping the raid controller for an more "up to date" one anyways.
 
try to avoid high point and also marvel controllers...

compiling and patching kernel can be easy on regular ubuntu, but it is very difficult on pve since it's modify/fork from the ubuntu repository already

The logic of the kernel patch is pretty simple, if 1103:0641, exclude from iommu, do not passthrough do not pass goal do not collect the $200
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back