I am passing through 4 PCIe devices into a VM running TruneNAS Scale: an HBA330 SAS HBA, a Radian RMS-200 cache device, & 2 virtual functions created with SR-IOV on 2 10 GbE SFP+ ports of a Broadcam BCM57800 NDC in a Dell r730XD server. All four are successfully made available inside the VM.
What is strange is that when I ran
(the driver for the NIC) on the VM guest, I get the following:
The PCIe root port on which the VF is made available in the guest is seen as a PCIe x1 link.
Guest PCIe root port:
It shows
The PCIe root ports for the other (non SR-IOV VF) passthroughs show the correct link width & speed. For instance, the RMS-200 in a PCIe x8 slot:
Any thoughts on a likely cause of this? Note: I don't have anything connected to the physical SFP+ ports; the switch & cabling is in the mail. Would that cause this?
Config (hostpci1 & hostpci2 are the NIC port passthroughs):
What is strange is that when I ran
Code:
dmesg | grep bnx2x
Code:
[ 2.043434] bnx2x 0000:02:00.0: msix capability found
[ 2.155499] bnx2x 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:1c.1 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 2.155526] bnx2x 0000:03:00.0: msix capability found
[ 2.263605] bnx2x 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:1c.2 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 2.450715] bnx2x 0000:02:00.0 enp2s0: renamed from eth2
[ 2.550617] bnx2x 0000:03:00.0 enp3s0: renamed from eth3
[ 49.576923] bnx2x 0000:02:00.0 enp2s0: using MSI-X IRQs: fp[0] 84 ... fp[3] 87
[ 50.832307] bnx2x 0000:03:00.0 enp3s0: using MSI-X IRQs: fp[0] 88 ... fp[3] 91
[ 58.098997] bnx2x 0000:02:00.0 enp2s0: using MSI-X IRQs: fp[0] 84 ... fp[3] 87
[ 59.669059] bnx2x 0000:03:00.0 enp3s0: using MSI-X IRQs: fp[0] 88 ... fp[3] 91
The PCIe root port on which the VF is made available in the guest is seen as a PCIe x1 link.
Guest PCIe root port:
Code:
root@truenas[~]# lspci -vv -s 0000:00:1c.1
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Subsystem: Red Hat, Inc. QEMU PCIe Root port
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 82248000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 1000-1fff [size=4K] [16-bit]
Memory behind bridge: 81e00000-81ffffff [size=2M] [32-bit]
Prefetchable memory behind bridge: 380800000000-380fffffffff [size=32G] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [54] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #0, PowerLimit 0W; Interlock+ NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCap: CRSVisible-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
Vector table: BAR=0 offset=00000000
PBA: BAR=0 offset=00000800
Capabilities: [40] Subsystem: Red Hat, Inc. QEMU PCIe Root port
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn+ NFERptEn+ FERptEn+
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [148 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pcieport
It shows
Code:
LnkSta: Speed 2.5GT/s, Width x1
The PCIe root ports for the other (non SR-IOV VF) passthroughs show the correct link width & speed. For instance, the RMS-200 in a PCIe x8 slot:
Code:
root@truenas[~]# lspci -vv -s 00:1c.0
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Subsystem: Red Hat, Inc. QEMU PCIe Root port
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 82249000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 8000-8fff [size=4K] [16-bit]
Memory behind bridge: 82000000-821fffff [size=2M] [32-bit]
Prefetchable memory behind bridge: 380000000000-3807ffffffff [size=32G] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [54] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #0, PowerLimit 0W; Interlock+ NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCap: CRSVisible-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
Vector table: BAR=0 offset=00000000
PBA: BAR=0 offset=00000800
Capabilities: [40] Subsystem: Red Hat, Inc. QEMU PCIe Root port
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn+ NFERptEn+ FERptEn+
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [148 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pcieport
Any thoughts on a likely cause of this? Note: I don't have anything connected to the physical SFP+ ports; the switch & cabling is in the mail. Would that cause this?
Config (hostpci1 & hostpci2 are the NIC port passthroughs):
Code:
root@pve:~# qm config 100
agent: 1
bios: ovmf
boot: order=scsi0
cores: 4
cpu: host,flags=+pcid;+aes
efidisk0: vm_nvme_mirror:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1
hostpci1: 0000:01:01.0,pcie=1
hostpci2: 0000:01:09.0,pcie=1
hostpci3: 0000:83:00.0,pcie=1
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
machine: q35
memory: 32768
meta: creation-qemu=9.0.2,ctime=1732447187
name: TrueNAS
net0: virtio=BC:24:11:37:70:4E,bridge=vmbr0,firewall=1,queues=8
net1: virtio=BC:24:11:C6:04:7D,bridge=vmbr_un,firewall=1,queues=8
numa: 1
onboot: 1
ostype: l26
scsi0: vm_nvme_mirror:vm-100-disk-1,discard=on,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=fb4857a0-8359-418e-a42f-d8cdcce49869
sockets: 2
vmgenid: b3cdbbfe-fe75-4ccd-8eb6-f9d3325eaa1c
vmstatestorage: vm_nvme_mirror
Last edited: