Intel i350-T4(V2??) card not showing up in Proxmox VE

[toddimus]

Hi,
First post here. Been lurking a bit as I have been setting up my new server. It's my first foray into Proxmox and a proper server. Previously, I had been doing virtualization in Docker running on a Linux box.

I'm having trouble getting my newly purchased i350-T4 card up and running. I purchased it supposedly new and it was supposed to be an i350-T4V2 but I can't seem to find any indication of the "V2" part. I did have a look through the posts regarding counterfeit cards and this one does seem to be genuine when comparing the photos. But who knows. It was manufactured in 12/2014 in Malaysia, if that adds any info. In my Supermicro X11SCH-LN4F motherboard, it shows up as just i350-T. Don't know if that's relevant. Probably not, I'm guessing.

Ok, so the problem is that it does show up in a few places (like BIOS and the dmesg output) but not in the most important place which is the Proxmox GUI. My on-board NIC ports (eno1-eno4) do show up and I have been able to use them in my initial testing. The new i350 (on PCI bus ID 00:02) is conspicuously missing. It does actually show up in the Proxmox node list where I can add it via PCI device to a VM but the ports don't show up in the "regular" network information in the GUI.

Doing a bit of digging through forum posts and searching on the web I came up with a bit of diagnostic info that I've added below. I think the issue shows up at the last few lines of the "dmesg | grep 00:02" section where it has four lines like this:

"[ 22.484676] igb 0000:02:00.0: removed PHC on enp2s0f0"

for each of the four ports of the new NIC. Not sure why that's happening and I'm hoping one of you smart folks can shed some light on things. Hopefully, I'm just being a noob and missing something silly.

Thanks in advance!!
-Todd

Proxmox VE 6.1-8

root@proxmox:/etc/network# dmesg | grep 02:00
[    0.352409] pci 0000:02:00.0: [8086:1521] type 00 class 0x020000
[    0.352890] pci 0000:02:00.0: reg 0x10: [mem 0x91400000-0x914fffff]
[    0.353327] pci 0000:02:00.0: reg 0x1c: [mem 0x9170c000-0x9170ffff]
[    0.353764] pci 0000:02:00.0: reg 0x30: [mem 0x91680000-0x916fffff pref]
[    0.355271] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[    0.355899] pci 0000:02:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[    0.355901] pci 0000:02:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.356279] pci 0000:02:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[    0.356281] pci 0000:02:00.0: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs)
[    0.358004] pci 0000:02:00.1: [8086:1521] type 00 class 0x020000
[    0.358464] pci 0000:02:00.1: reg 0x10: [mem 0x91300000-0x913fffff]
[    0.358901] pci 0000:02:00.1: reg 0x1c: [mem 0x91708000-0x9170bfff]
[    0.359338] pci 0000:02:00.1: reg 0x30: [mem 0x91600000-0x9167ffff pref]
[    0.360807] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
[    0.361435] pci 0000:02:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[    0.361437] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.361815] pci 0000:02:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[    0.361817] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs)
[    0.363410] pci 0000:02:00.2: [8086:1521] type 00 class 0x020000
[    0.363869] pci 0000:02:00.2: reg 0x10: [mem 0x91200000-0x912fffff]
[    0.364305] pci 0000:02:00.2: reg 0x1c: [mem 0x91704000-0x91707fff]
[    0.364743] pci 0000:02:00.2: reg 0x30: [mem 0x91580000-0x915fffff pref]
[    0.366464] pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
[    0.367353] pci 0000:02:00.2: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[    0.367355] pci 0000:02:00.2: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.367732] pci 0000:02:00.2: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[    0.367734] pci 0000:02:00.2: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs)
[    0.369310] pci 0000:02:00.3: [8086:1521] type 00 class 0x020000
[    0.369769] pci 0000:02:00.3: reg 0x10: [mem 0x91100000-0x911fffff]
[    0.370206] pci 0000:02:00.3: reg 0x1c: [mem 0x91700000-0x91703fff]
[    0.370643] pci 0000:02:00.3: reg 0x30: [mem 0x91500000-0x9157ffff pref]
[    0.372127] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
[    0.372744] pci 0000:02:00.3: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[    0.372746] pci 0000:02:00.3: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.373124] pci 0000:02:00.3: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[    0.373126] pci 0000:02:00.3: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs)
[    0.497399] pci 0000:02:00.0: BAR 7: assigned [mem 0x92100000-0x9211ffff 64bit pref]
[    0.497463] pci 0000:02:00.0: BAR 10: assigned [mem 0x92120000-0x9213ffff 64bit pref]
[    0.497527] pci 0000:02:00.1: BAR 7: assigned [mem 0x92140000-0x9215ffff 64bit pref]
[    0.497590] pci 0000:02:00.1: BAR 10: assigned [mem 0x92160000-0x9217ffff 64bit pref]
[    0.497654] pci 0000:02:00.2: BAR 7: assigned [mem 0x92180000-0x9219ffff 64bit pref]
[    0.497717] pci 0000:02:00.2: BAR 10: assigned [mem 0x921a0000-0x921bffff 64bit pref]
[    0.497781] pci 0000:02:00.3: BAR 7: assigned [mem 0x921c0000-0x921dffff 64bit pref]
[    0.497844] pci 0000:02:00.3: BAR 10: assigned [mem 0x921e0000-0x921fffff 64bit pref]
[    0.892479] pci 0000:02:00.0: Adding to iommu group 1
[    0.892484] pci 0000:02:00.1: Adding to iommu group 1
[    0.892488] pci 0000:02:00.2: Adding to iommu group 1
[    0.892492] pci 0000:02:00.3: Adding to iommu group 1
[    1.654813] igb 0000:02:00.0: added PHC on eth0
[    1.658389] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[    1.666128] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) REDACTED MAC
[    1.670975] igb 0000:02:00.0: eth0: PBA No: H38747-000
[    1.674568] igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    1.777483] igb 0000:02:00.1: added PHC on eth1
[    1.779945] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
[    1.782412] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) REDACTED MAC
[    1.785590] igb 0000:02:00.1: eth1: PBA No: H38747-000
[    1.788064] igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    1.881685] igb 0000:02:00.2: added PHC on eth2
[    1.885078] igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
[    1.888692] igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) REDACTED MAC
[    1.893014] igb 0000:02:00.2: eth2: PBA No: H38747-000
[    1.893015] igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    1.989564] igb 0000:02:00.3: added PHC on eth3
[    1.993188] igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
[    1.996902] igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) REDACTED MAC
[    2.001673] igb 0000:02:00.3: eth3: PBA No: H38747-000
[    2.012824] igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    2.315603] igb 0000:02:00.1 enp2s0f1: renamed from eth1
[    2.356445] igb 0000:02:00.2 enp2s0f2: renamed from eth2
[    2.412322] igb 0000:02:00.0 enp2s0f0: renamed from eth0
[    2.456450] igb 0000:02:00.3 enp2s0f3: renamed from eth3
[   22.266020] igb 0000:02:00.2: removed PHC on enp2s0f2
[   22.484676] igb 0000:02:00.0: removed PHC on enp2s0f0
[   24.312857] igb 0000:02:00.3: removed PHC on enp2s0f3
[   24.516808] igb 0000:02:00.1: removed PHC on enp2s0f1
root@proxmox:/etc/network#

root@proxmox:/etc/network# cat /etc/network/interfaces
...
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
        address REDACTED
        gateway REDACTED
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address REDACTED
        gateway REDACTED
        bridge-ports eno4
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet manual
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

auto vmbr3
iface vmbr3 inet manual
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0


root@proxmox:/etc/network# networkctl
WARNING: systemd-networkd is not running, output will be incomplete.

IDX LINK             TYPE               OPERATIONAL SETUP    
  1 lo               loopback           n/a         unmanaged
  6 eno1             ether              n/a         unmanaged
  7 eno2             ether              n/a         unmanaged
  8 eno3             ether              n/a         unmanaged
  9 eno4             ether              n/a         unmanaged
10 vmbr0            bridge             n/a         unmanaged
11 vmbr1            bridge             n/a         unmanaged
12 vmbr2            bridge             n/a         unmanaged
13 vmbr3            bridge             n/a         unmanaged
14 tap102i0         ether              n/a         unmanaged
15 fwbr102i0        bridge             n/a         unmanaged
16 fwpr102p0        ether              n/a         unmanaged
17 fwln102i0        ether              n/a         unmanaged
19 veth202i0        ether              n/a         unmanaged
20 fwbr202i0        bridge             n/a         unmanaged
21 fwpr202p0        ether              n/a         unmanaged
22 fwln202i0        ether              n/a         unmanaged
24 veth203i0        ether              n/a         unmanaged

18 links listed.

 

[toddimus]

And to follow up with a bit of extra info because I just passed the 10000 character limit in the first post:

root@proxmox:~# lspci | grep Ethernet
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

[toddimus]

After more digging, I've come up with more details and a better explanation of what I'm trying to achieve. Here's a snippet of my post to Supermicro support looking for help there. Maybe someone here has an idea??

I am trying to use a PCIe passthrough for the two different PCIe slots going to two different virtual machines. In the 8x/16x PCIe slot closest to the CPU, I have an Intel i350-T4 quad port gigabit NIC. In the other 8x PCIe slot, I have an LSI 9207 SAS card. The SAS card is already in use by a virtual machine and it is being passed through successfully. The new i350 NIC card is not accessible to the host system, nor any of the other VMs I'm running. It is however available for PCIe passthrough to the same VM that's using the SAS card. I want to have them work with different VMs.

I've narrowed it down to the IOMMU group assignment as the main issue. Both cards are assigned to the IOMMU group 1, so they can't be separated logically by the system (Proxmox Linux host). I've tried every BIOS setting I can think of and read through countless forum threads. I also tried using the "ACS patch" in the linux kernel but it still keeps both cards on the same IOMMU group 1. From the block diagram in the motherboard manual, it appears that both PCIe slots are directly linked to the CPU. It seems that there must be a way to tell the CPU to assign them to different groups.

Is there a way in the BIOS settings to separate these? And ultimately, is there a way to enable the SR-IOV passthrough features of the i350 card once they are separated? Or maybe that will just work once they are separated in different IOMMU groups.

 

[bolmsted]

Did you ever figure this out?

I have a QLogic Broadcom BCM57810S dual port SFP+ card and a Quad port Intel i350-t4 card (supposedly, yes aware of the potential counterfeit but bought from a local place (via ebay) that says was a Dell version).

For some reason on my first proxmox install I could go to Hardware and add the Broadcom card (first port) and all 4 ports of the Dell i350-t4 card as hard ware pass through (I believe) as I configured the grub setting for IOMMU but when I added the 2nd port on the Broadcom card (which I don't need immediately as I don't have a 10G network yet) it was giving me some error like "Parameter Verification failed (400) property is not defined in schema and the schema does not allow additional properties" (googling to error message from memory as I didn't record it at the time).

When I had the NIC in proxmox it was identifying enp1s0f0, enp1s0f1 (Broadcom) and enp2s0f0, enp2s0f1, enp2s0f1, enp2s0f2, enps0f3 (Dell i350-t4) (I think I have the right names of the two sets of NICs) along with the onboard ethernet (em0 - at least in pfsense).

Originally I installed proxmox with the onboard NIC and it was added to a virtual bridge in proxmox and I tried to add the quad port card into the bridge but for some reason only enp2s0f0, enp2s0f1, enp2s0f2 were being added as Active (and I see auto start was turned off otherwise active wouldn't come p) but for some reason enp2s0f3 (4th port) would no show as active in proxmox for some reason so I thought the Dell Quad port i350-t4 was possibly bad and I decided to reseat it as I couldn't get the ethernet cable to click into the 4th port for some reason (and didn't seem to get both lights on that port lighting up) and it seemed to line up better.

To rule out NIC issues I decided to reinstall Windows 10 from a USB drive and verify that both cards were recognized properly at that this point at least verify that quad port card was working so I plugged ethernet into each of the 4 ports and was getting both lights on each port lighting up, IP addresses so proceeded to do speedtest.net tests, etc so everything is working fine.

About to try to flash the server again with another Proxmox install again and then proceed to put pfsense box on the box afterwards but I want to make sure the hardware is being recognized before I get into putting more ram into the workstation, etc

I'm new to Proxmox so just want to make sure I wasn't running into bug, etc Anyway will move forward with install again.