Mellanox 4 LX - PCIe

furfix · Jul 17, 2024

Hi all! Since sometime I'm trying to passtrough my Mellanox 4 card to an **sense FreeBSD VM without any luck. I followed every single thing I've found out in internet. The host has totally blacklisted the card, nobody is using it. I want to passtrough the whole card to the VM. Not looking to split the ports or doing things like that, but I just can't.

I choose the Raw Device from the dropdown list. I click "All functions" and PCI-e, but for some reason, the **sense while booting hangs and wait 100s for FW initialiation (mlx5_core0) that never happens. I'm already passingthrough my UHD GPU to another VM without any issue. The card has the lastest firmware out there and the VM is q35.

Code:

# lspci -s 01:00.0
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

# lspci -s 01:00.0 -n
01:00.0 0200: 15b3:1015

# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
intel-microcode: 3.20240514.1~deb12u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

# lspci -s 01:00.0  -vv
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 15
        Region 0: Memory at 6026000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at 96200000 [disabled] [size=1M]
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [48] Vital Product Data
                Product Name: CX4121A - ConnectX-4 LX SFP28
                Read-only fields:
                        [PN] Part number: MCX4121A-ACAT
                        [EC] Engineering changes: AF
                        [SN] Serial number: MT--------
                        [V0] Vendor specific: PCIeGen3 x8
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 04, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 2, stride: 1, Device ID: 1016
                Supported Page Size: 000007ff, System Page Size: 00000001
                Region 0: Memory at 0000006028800000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [230 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

gfngfn256 · Jul 17, 2024

Do you know that this card works correctly in **sense FreeBSD? To test; try it on bare-metal in **sense or plain FreeBSD.
To see if its a Proxmox/passthrough issue, spin up a plain Debian VM & pass it through to that VM.

Good luck.

v95klima · Jul 17, 2024

Hi
Mellanox Connectx-4 works for me on both FreeBSD and pfSense. It even works with SR-VIO virtual nics. It detected directly at boot correctly for me.

Openbsd on the other is not detecting it correctly.

What does dmesg show on this card?

Does other Ethernet cards/controllers pass through successfully and problem is specifically for Connectx-4 ?

furfix · Jul 17, 2024

This is what I can see while booting OPNsense:

Code:

mlx5_core0: WARN: wait_fw_init:733:(pid 0): Waiting for FW initialization, timeout abort in 100 s

and after 100s

Code:

mlx5_core0: ERR: Firmware over 120000 MS in pre-initializing state, aborting
mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -16
device_attach: mlx5_core0 attach returned 16

and keep looping forever and once if I wait long enough....I arrive to the point where the interfaces needs to be assigned, but it doesn't detect any.

v95klima · Jul 18, 2024

You are right, add Raw Device, All function does not work with latest opnsense in Proxmox VM.

But if you add the two Raw devices, in my case 02.00.0 and 02.00.1 separately with PCI-Express box enabled it boots, they show up as expected in working order for my Connectx-4.

v95klima · Jul 18, 2024

per above

furfix · Jul 18, 2024

v95klima said:
You are right, add Raw Device, All function does not work with latest opnsense in Proxmox VM.

But if you add the two Raw devices, in my case 02.00.0 and 02.00.1 separately with PCI-Express box enabled it boots, they show up as expected in working order for my Connectx-4.

Any specific config either in Proxmox or in OPNsense? Just tried, but unfortunately doesn't work for me

v95klima · Jul 18, 2024

No no special config in opnsense, that is a fresh install with q35.

In proxmox /etc/default/grub
I have

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Does other Ethernet cards/controllers pass through successfully and problem is specifically for Connectx-4 ?

By testing another Ethernet cards passthrough success will indicate if the problem is Proxmox /BIOS config or card itself.

v95klima · Jul 18, 2024

btw,
if you boot up FreeBSD 14.1 does that work passing through your card.

I run PF Firewall, DHCPd and Unbound directly in FreeBSD instead of opnsense or pfsense. I normally run my homemade Firewall/Router in OpenBSD but since that kernel does not support ConnectX-4 I’m settling for FreeBSD.

I find using PF directly so much more clear, crisp and powerful, than the webgui drop-down menus.
It all started when I found this guide,
https://openbsdrouterguide.net/
and having kids at home who are too young for unrestricted Internet access, both in duration and content.

Maybe this is your time to move to FreeBSD directly.

furfix · Jul 22, 2024

v95klima said:
btw,
if you boot up FreeBSD 14.1 does that work passing through your card.

I run PF Firewall, DHCPd and Unbound directly in FreeBSD instead of opnsense or pfsense. I normally run my homemade Firewall/Router in OpenBSD but since that kernel does not support ConnectX-4 I’m settling for FreeBSD.

I find using PF directly so much more clear, crisp and powerful, than the webgui drop-down menus.
It all started when I found this guide,
https://openbsdrouterguide.net/
and having kids at home who are too young for unrestricted Internet access, both in duration and content.

Maybe this is your time to move to FreeBSD directly.

If I boot a VM with the FreeBSD 14.1 iso, I found the same error: mlx5_core0: WARN: wait_fw_init:733

pid 0): Waiting for FW initialization, timeout abort in 100 s

I did all normal bios and base proxmox configs to turn on sriov/pass thru support. SR-IOV, VT-d and ROM Bar enabled in Motherboard BIOS running the last version. intel_iommu=on iommu=pt enabled in Host.

Also checked the Firmware/BIOS options in Mellanox Card, and results are OK:

Code:

NUM_OF_VFS 8
NUM_OF_PF 2
SRIOV_EN True(1)

I just ran out of ideas to make it work and looking for some help out there

gfngfn256 · Jul 22, 2024

furfix said:
I just ran out of ideas to make it work and looking for some help out there

You are given help - you just don't try it/ignore it:

gfngfn256 said:
To test; try it on bare-metal in **sense or plain FreeBSD.
To see if its a Proxmox/passthrough issue, spin up a plain Debian VM & pass it through to that VM.

v95klima · Jul 22, 2024

Seems like problem with the Mellanox card, or the card firmware/setting?
Is the card Firmware set in Ethernet mode?
Has the card worked on another platform, or new to you?

In FreeBSD these commands (VM on Proxmox), gives me:

dmesg |grep Mellanox
Mellanox Core driver 3.7.1 (Nov 2021) for me.

dmesg|grep mlx5
warn init_once 999 = no vendor specific capabilities
warn fw_dump_prep 92 = no vendor specific capabilities, error 2

but none of them locks up the system. Boots ok and works well.

furfix · Jul 22, 2024

Tried **Sense baremetal and it's working. I've also tried FreeBSD ISO in a Proxmox VM, and I'm getting the same error.

gfngfn256 said:
You are given help - you just don't try it/ignore it:

furfix · Jul 22, 2024

v95klima said:
Seems like problem with the Mellanox card, or the card firmware/setting?
Is the card Firmware set in Ethernet mode?
Has the card worked on another platform, or new to you?

In FreeBSD these commands (VM on Proxmox), gives me:

dmesg |grep Mellanox
Mellanox Core driver 3.7.1 (Nov 2021) for me.

dmesg|grep mlx5
warn init_once 999 = no vendor specific capabilities
warn fw_dump_prep 92 = no vendor specific capabilities, error 2

but none of them locks up the system. Boots ok and works well.

The card is working in Baremetal. It's also working in Proxmox Host. I'm currently using with a virtual bridge. This card has only Ethernet.

I tried both boot, OPNSense and FreeBSD 14.1 ISO in a Proxmox VM and I get the same error:

Code:

mlx5_core0: WARN: wait_fw_init:733:(pid 0): Waiting for FW initialization, timeout abort in 100 s

and after 100s

Code:

mlx5_core0: ERR: Firmware over 120000 MS in pre-initializing state, aborting
mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -16
device_attach: mlx5_core0 attach returned 16

I tried, in Proxmox using "All functions" and adding the ports individually to the VM unsuccessfully.

I did all normal bios and base proxmox configs to turn on sriov/pass thru support. SR-IOV, VT-d, IOMMU, and ROM Bar enabled in Motherboard BIOS running the last version. intel_iommu=on iommu=pt enabled in Proxmox Host.

Sharing more info about the card from the host:

Code:

~# dmesg | grep mlx5
[    1.084653] mlx5_core 0000:01:00.0: firmware version: 14.32.1010
[    1.084687] mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.376206] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[    1.380195] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged
[    1.637230] mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0 basic)
[    1.637247] mlx5_core 0000:01:00.0: is_dpll_supported:213:(pid 212): Missing SyncE capability
[    1.641791] mlx5_core 0000:01:00.1: firmware version: 14.32.1010
[    1.641822] mlx5_core 0000:01:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.965236] mlx5_core 0000:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[    1.971245] mlx5_core 0000:01:00.1: port_module:252:(pid 0): Port module event[error]: module 1, Cable error, Unknown error
[    2.257017] mlx5_core 0000:01:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0 basic)
[    2.257031] mlx5_core 0000:01:00.1: is_dpll_supported:213:(pid 212): Missing SyncE capability
[    2.273691] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0
[    2.286719] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth1
[    5.348529] mlx5_core 0000:01:00.0 enp1s0f0np0: entered allmulticast mode
[    6.717810] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up
[    6.746944] mlx5_core 0000:01:00.1 enp1s0f1np1: entered allmulticast mode
[    7.976309] mlx5_core 0000:01:00.1 enp1s0f1np1: Link up
[   12.383558] mlx5_core 0000:01:00.0 enp1s0f0np0: entered promiscuous mode
[   12.399248] mlx5_core 0000:01:00.0: mlx5e_fs_set_rx_mode_work:842:(pid 258): S-tagged traffic will be dropped while C-tag vlan stripping is enabled
[   12.823442] mlx5_core 0000:01:00.1 enp1s0f1np1: entered promiscuous mode

Code:

~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Code:

~# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4117_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:01:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

Code:

~# mlxconfig -d /dev/mst/mt4117_pciconf0 q

Device #1:
----------

Device type:        ConnectX4LX
Name:               MCX4121A-ACA_Ax
Description:        ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
Device:             /dev/mst/mt4117_pciconf0

Configurations:                                          Next Boot
        MEMIC_BAR_SIZE                              0
        MEMIC_SIZE_LIMIT                            _256KB(1)
        FLEX_PARSER_PROFILE_ENABLE                  0
        FLEX_IPV4_OVER_VXLAN_PORT                   0
        ROCE_NEXT_PROTOCOL                          254
        PF_NUM_OF_VF_VALID                          False(0)
        NON_PREFETCHABLE_PF_BAR                     False(0)
        VF_VPD_ENABLE                               False(0)
        STRICT_VF_MSIX_NUM                          False(0)
        VF_NODNIC_ENABLE                            False(0)
        NUM_PF_MSIX_VALID                           True(1)
        NUM_OF_VFS                                  8
        NUM_OF_PF                                   2
        SRIOV_EN                                    True(1)
        PF_LOG_BAR_SIZE                             5
        VF_LOG_BAR_SIZE                             0
        NUM_PF_MSIX                                 63
        NUM_VF_MSIX                                 11
        INT_LOG_MAX_PAYLOAD_SIZE                    AUTOMATIC(0)
        PCIE_CREDIT_TOKEN_TIMEOUT                   0
        ACCURATE_TX_SCHEDULER                       False(0)
        PARTIAL_RESET_EN                            False(0)
        SW_RECOVERY_ON_ERRORS                       False(0)
        RESET_WITH_HOST_ON_ERRORS                   False(0)
        PCI_BUS0_RESTRICT_SPEED                     PCI_GEN_1(0)
        PCI_BUS0_RESTRICT_ASPM                      False(0)
        PCI_BUS0_RESTRICT_WIDTH                     PCI_X1(0)
        PCI_BUS0_RESTRICT                           False(0)
        PCI_DOWNSTREAM_PORT_OWNER                   Array[0..15]
        CQE_COMPRESSION                             BALANCED(0)
        IP_OVER_VXLAN_EN                            False(0)
        MKEY_BY_NAME                                False(0)
        UCTX_EN                                     True(1)
        PCI_ATOMIC_MODE                             PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
        TUNNEL_ECN_COPY_DISABLE                     False(0)
        LRO_LOG_TIMEOUT0                            6
        LRO_LOG_TIMEOUT1                            7
        LRO_LOG_TIMEOUT2                            8
        LRO_LOG_TIMEOUT3                            13
        ICM_CACHE_MODE                              DEVICE_DEFAULT(0)
        TX_SCHEDULER_BURST                          0
        LOG_MAX_QUEUE                               17
        LOG_DCR_HASH_TABLE_SIZE                     14
        MAX_PACKET_LIFETIME                         0
        DCR_LIFO_SIZE                               16384
        ROCE_CC_PRIO_MASK_P1                        255
        ROCE_CC_CNP_MODERATION_P1                   DEVICE_DEFAULT(0)
        ROCE_CC_PRIO_MASK_P2                        255
        ROCE_CC_CNP_MODERATION_P2                   DEVICE_DEFAULT(0)
        CLAMP_TGT_RATE_AFTER_TIME_INC_P1            True(1)
        CLAMP_TGT_RATE_P1                           False(0)
        RPG_TIME_RESET_P1                           300
        RPG_BYTE_RESET_P1                           32767
        RPG_THRESHOLD_P1                            1
        RPG_MAX_RATE_P1                             0
        RPG_AI_RATE_P1                              5
        RPG_HAI_RATE_P1                             50
        RPG_GD_P1                                   11
        RPG_MIN_DEC_FAC_P1                          50
        RPG_MIN_RATE_P1                             1
        RATE_TO_SET_ON_FIRST_CNP_P1                 0
        DCE_TCP_G_P1                                1019
        DCE_TCP_RTT_P1                              1
        RATE_REDUCE_MONITOR_PERIOD_P1               4
        INITIAL_ALPHA_VALUE_P1                      1023
        MIN_TIME_BETWEEN_CNPS_P1                    4
        CNP_802P_PRIO_P1                            6
        CNP_DSCP_P1                                 48
        CLAMP_TGT_RATE_AFTER_TIME_INC_P2            True(1)
        CLAMP_TGT_RATE_P2                           False(0)
        RPG_TIME_RESET_P2                           300
        RPG_BYTE_RESET_P2                           32767
        RPG_THRESHOLD_P2                            1
        RPG_MAX_RATE_P2                             0
        RPG_AI_RATE_P2                              5
        RPG_HAI_RATE_P2                             50
        RPG_GD_P2                                   11
        RPG_MIN_DEC_FAC_P2                          50
        RPG_MIN_RATE_P2                             1
        RATE_TO_SET_ON_FIRST_CNP_P2                 0
        DCE_TCP_G_P2                                1019
        DCE_TCP_RTT_P2                              1
        RATE_REDUCE_MONITOR_PERIOD_P2               4
        INITIAL_ALPHA_VALUE_P2                      1023
        MIN_TIME_BETWEEN_CNPS_P2                    4
        CNP_802P_PRIO_P2                            6
        CNP_DSCP_P2                                 48
        LLDP_NB_DCBX_P1                             False(0)
        LLDP_NB_RX_MODE_P1                          OFF(0)
        LLDP_NB_TX_MODE_P1                          OFF(0)
        LLDP_NB_DCBX_P2                             False(0)
        LLDP_NB_RX_MODE_P2                          OFF(0)
        LLDP_NB_TX_MODE_P2                          OFF(0)
        ROCE_RTT_RESP_DSCP_P1                       0
        ROCE_RTT_RESP_DSCP_MODE_P1                  DEVICE_DEFAULT(0)
        ROCE_RTT_RESP_DSCP_P2                       0
        ROCE_RTT_RESP_DSCP_MODE_P2                  DEVICE_DEFAULT(0)
        DCBX_IEEE_P1                                True(1)
        DCBX_CEE_P1                                 True(1)
        DCBX_WILLING_P1                             True(1)
        DCBX_IEEE_P2                                True(1)
        DCBX_CEE_P2                                 True(1)
        DCBX_WILLING_P2                             True(1)
        KEEP_ETH_LINK_UP_P1                         True(1)
        KEEP_IB_LINK_UP_P1                          False(0)
        KEEP_LINK_UP_ON_BOOT_P1                     False(0)
        KEEP_LINK_UP_ON_STANDBY_P1                  False(0)
        DO_NOT_CLEAR_PORT_STATS_P1                  False(0)
        AUTO_POWER_SAVE_LINK_DOWN_P1                False(0)
        KEEP_ETH_LINK_UP_P2                         True(1)
        KEEP_IB_LINK_UP_P2                          False(0)
        KEEP_LINK_UP_ON_BOOT_P2                     False(0)
        KEEP_LINK_UP_ON_STANDBY_P2                  False(0)
        DO_NOT_CLEAR_PORT_STATS_P2                  False(0)
        AUTO_POWER_SAVE_LINK_DOWN_P2                False(0)
        NUM_OF_VL_P1                                _4_VLs(3)
        NUM_OF_TC_P1                                _8_TCs(0)
        NUM_OF_PFC_P1                               8
        VL15_BUFFER_SIZE_P1                         0
        NUM_OF_VL_P2                                _4_VLs(3)
        NUM_OF_TC_P2                                _8_TCs(0)
        NUM_OF_PFC_P2                               8
        VL15_BUFFER_SIZE_P2                         0
        DUP_MAC_ACTION_P1                           LAST_CFG(0)
        SRIOV_IB_ROUTING_MODE_P1                    LID(1)
        IB_ROUTING_MODE_P1                          LID(1)
        DUP_MAC_ACTION_P2                           LAST_CFG(0)
        SRIOV_IB_ROUTING_MODE_P2                    LID(1)
        IB_ROUTING_MODE_P2                          LID(1)
        PHY_FEC_OVERRIDE_P1                         DEVICE_DEFAULT(0)
        PHY_FEC_OVERRIDE_P2                         DEVICE_DEFAULT(0)
        PF_SD_GROUP                                 0
        ROCE_CONTROL                                ROCE_ENABLE(2)
        PCI_WR_ORDERING                             per_mkey(0)
        MULTI_PORT_VHCA_EN                          False(0)
        PORT_OWNER                                  True(1)
        ALLOW_RD_COUNTERS                           True(1)
        RENEG_ON_CHANGE                             True(1)
        TRACER_ENABLE                               True(1)
        IP_VER                                      IPv4(0)
        BOOT_UNDI_NETWORK_WAIT                      0
        UEFI_HII_EN                                 True(1)
        BOOT_DBG_LOG                                False(0)
        UEFI_LOGS                                   DISABLED(0)
        BOOT_VLAN                                   1
        LEGACY_BOOT_PROTOCOL                        PXE(1)
        BOOT_INTERRUPT_DIS                          False(0)
        BOOT_LACP_DIS                               True(1)
        BOOT_VLAN_EN                                False(0)
        BOOT_PKEY                                   0
        DYNAMIC_VF_MSIX_TABLE                       False(0)
        EXP_ROM_UEFI_ARM_ENABLE                     False(0)
        EXP_ROM_UEFI_x86_ENABLE                     False(0)
        EXP_ROM_PXE_ENABLE                          True(1)
        ADVANCED_PCI_SETTINGS                       False(0)
        SAFE_MODE_THRESHOLD                         10
        SAFE_MODE_ENABLE                            True(1)

Code:

~# lspci -nn | grep Mellanox
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

gfngfn256 · Jul 22, 2024

Try passthrough to a regular Debian VM in PVE as above. Also post qm config {VMID} of affected VM where it does not work.

furfix · Jul 22, 2024

gfngfn256 said:
Try passthrough to a regular Debian VM in PVE as above. Also post qm config {VMID} of affected VM where it does not work.

This is the current config of the VM. vmbr1 and 2 are there because I use the VM as firewall, but I remove them and I add the PCIe device while trying to passingthrough the card.

I'm going to try with a Debian VM to see the results.

Code:

~# qm config 100
agent: 1
bios: ovmf
boot: order=scsi0
cores: 4
cpu: host,flags=+aes
efidisk0: storage:vm-100-disk-0,efitype=4m,size=4M
machine: q35
memory: 8192
meta: creation-qemu=9.0.0,ctime=1721405047
name: OPNSense
net0: virtio=00:0F:94:64:32:81,bridge=vmbr1,firewall=1
net1: virtio=BC:24:11:14:03:5D,bridge=vmbr2,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: storage:vm-100-disk-1,backup=0,discard=on,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=914847d4-9b9e-43eb-87f2-a9c012889b65
sockets: 1
startup: order=1
tablet: 0
vmgenid: 17d91629-98dd-4748-aff2-bf914a73759f

furfix · Jul 22, 2024

Not working in debian VM

gfngfn256 · Jul 22, 2024

Ok. So no passthrough working with this card at all.

I'd start with changing CPU type to x86-64-v2-AES.
Then I'd try changing the Version of the q35 machine type. Lastly I'd look into vIOMMU / VirtIO.

gfngfn256 · Jul 22, 2024

Sorry forgot to ask: What CPU do you have?

furfix · Jul 22, 2024

it is an Intel i5 11600, running in a MSI MEG z490i Unify board. I'm already passing through the intel igpu to another VM, working fine but I assume it's not the same the igpu than the PCIx16

I will play around the cpu type and options there to see if I have more luck and let you know

btw, is it not possible to pass the whole socket, instead of the device? The board has only a PICex16 where the NIC is connected. Is it not possible to just passtrhough the socket? maybe has no sense, but just in case...

Mellanox 4 LX - PCIe

Member

Renowned Member

Member

Member

Member

Member

Attachments

Member

Member

Member

Member

Renowned Member

Member

Member

Member

Renowned Member

Member

Member

Renowned Member

Renowned Member

Member