Proxmox user base seems rather thin?

In any case the Intel is better than the Realtek, but I think you know that. Doesn't help much if it won't work...I know. ;)


Yeah, exactly.


Rufus can build a win-to-go from .iso or something. But never tried that.

Well, that worked, to a degree. I was able to flash the T4, but not the onboard. FLASHENABLE says it's not supported for the hardware. Went from 1.52 to 1.59 on the T4, I think is what it was, but no change, still has the memory reservation problem. While in Windows (10 Pro), I was able to assign IP's to each NIC and test. They all worked. So it seems that there is something odd with igb/MSI-X and how it's handling memory reservations. Or, maybe it's handling them exactly as things should be handled and MS is loosely implementing a "standard?"

I also disabled the onboard NICs. No change.

I rarely give up on anything, but I'm close on this one. I say that because, if, say, I pin an older kernel, or some other thing, get everything up and running to the point where I can retire my old Windows stuff, it will always be a little itch in the back of my mind. Not that MS doesn't have its share of issues, to be sure.
 
I was able to flash the T4
Nice!
but not the onboard. FLASHENABLE says it's not supported for the hardware
Could be wrong preboot-package version or speciality package needed from supermicro. Anyway, then that's not so important.
While in Windows (10 Pro), I was able to assign IP's to each NIC and test. They all worked. So it seems that there is something odd with igb/MSI-X and how it's handling memory reservations. Or, maybe it's handling them exactly as things should be handled and MS is loosely implementing a "standard?"
Possible. https://bugzilla.kernel.org/show_bug.cgi?id=218050#c13
If this is a kernel bug, then I don't have or see it.
Are there some old configuration leftovers? Maybe https://github.com/gnif/vendor-reset/ still grabbing @pci-id 07:00:xx where previously an AMD-gpu was? Or some experiments with sr-iov? Possible...if you say the I350 works on Win and Ubuntu.

I rarely give up on anything, but I'm close on this one.
Would be too fast IMHO, but on the other hand...the I350 quad is worth ~20€ if you calculate that way. I don't. ;)
I have a few more ideas to observe the behavior:
Can you change the pci-e slot? Have you reseated it? Flaky contacts?
Flash firmware of the supermicro-board if it's not already on the latest fw
(CMOS)Reset the mainboard, make sure the board boots UEFI-only aka CSM off into proxmox
Try all combinations of the following settings: Above 4G on/off, rebar/resizeable bar on/off, sr-iov on/off, IOMMU auto/enabled (explicit) <- if your board has this option
Boot proxmox.iso, click through the wizard up to the point where you would set your initial NIC/IP. If you see all 6 there available and this differs from the regular boot behaviour, then it is either old misconfig stuff or really a bug in kernel.

After this I would be out of ideas.

Edit:
Check this: https://forum.proxmox.com/threads/p...pci-device-probing-failure.131203/post-738089
 
Last edited:
  • Like
Reactions: mike the newb
Nice!

Could be wrong preboot-package version or speciality package needed from supermicro. Anyway, then that's not so important.

Possible. https://bugzilla.kernel.org/show_bug.cgi?id=218050#c13
If this is a kernel bug, then I don't have or see it.
Are there some old configuration leftovers? Maybe https://github.com/gnif/vendor-reset/ still grabbing @pci-id 07:00:xx where previously an AMD-gpu was? Or some experiments with sr-iov? Possible...if you say the I350 works on Win and Ubuntu.


Would be too fast IMHO, but on the other hand...the I350 quad is worth ~20€ if you calculate that way. I don't. ;)
I have a few more ideas to observe the behavior:
Can you change the pci-e slot? Have you reseated it? Flaky contacts?
Flash firmware of the supermicro-board if it's not already on the latest fw
(CMOS)Reset the mainboard, make sure the board boots UEFI-only aka CSM off into proxmox
Try all combinations of the following settings: Above 4G on/off, rebar/resizeable bar on/off, sr-iov on/off, IOMMU auto/enabled (explicit) <- if your board has this option
Boot proxmox.iso, click through the wizard up to the point where you would set your initial NIC/IP. If you see all 6 there available and this differs from the regular boot behaviour, then it is either old misconfig stuff or really a bug in kernel.

After this I would be out of ideas.

Edit:
Check this: https://forum.proxmox.com/threads/p...pci-device-probing-failure.131203/post-738089

I was just able to get the package (older) from supermicro that includes the bootutil and firmware, which indicates to me that FLASHENABLE should work. No indication as to HOW. There are no jumpers or anything to enable it on the MB, and nothing in all the documentation I've read. But that one line about "pci=realloc=off" is one I hadn't seen before. I'd messed with iommu already, but not that. Somehow, I missed that particular response in that thread when I read it early on.

I already swapped PCI ports, which moved the memory space to 09:00.0 from 08:00.0, but it still failed. Setting bifurcation specifically, as opposed to "auto" moved the address space from 07:00.0 to 08:00.0. I also flashed to the latest BIOS and cleared cmos for the MB.
There is no on/off for resizable BAR, and BIOS has few options in that regard. I've tried everything, both on and off, and reconfigured. During install, I only see the two onboard NICs, which initially indicated to me that I was looking at a bios issue, but then, it also could have been the same memory reservation issue that be kernel related.

I'll know a bit more after I try the Realtek NIC. I also ordered a new T4 from a different MFR for shits and giggles.

BTW, thanks for taking time and offering ideas! I appreciate it.
 
Nice!

Could be wrong preboot-package version or speciality package needed from supermicro. Anyway, then that's not so important.

Possible. https://bugzilla.kernel.org/show_bug.cgi?id=218050#c13
If this is a kernel bug, then I don't have or see it.
Are there some old configuration leftovers? Maybe https://github.com/gnif/vendor-reset/ still grabbing @pci-id 07:00:xx where previously an AMD-gpu was? Or some experiments with sr-iov? Possible...if you say the I350 works on Win and Ubuntu.


Would be too fast IMHO, but on the other hand...the I350 quad is worth ~20€ if you calculate that way. I don't. ;)
I have a few more ideas to observe the behavior:
Can you change the pci-e slot? Have you reseated it? Flaky contacts?
Flash firmware of the supermicro-board if it's not already on the latest fw
(CMOS)Reset the mainboard, make sure the board boots UEFI-only aka CSM off into proxmox
Try all combinations of the following settings: Above 4G on/off, rebar/resizeable bar on/off, sr-iov on/off, IOMMU auto/enabled (explicit) <- if your board has this option
Boot proxmox.iso, click through the wizard up to the point where you would set your initial NIC/IP. If you see all 6 there available and this differs from the regular boot behaviour, then it is either old misconfig stuff or really a bug in kernel.

After this I would be out of ideas.

Edit:
Check this: https://forum.proxmox.com/threads/p...pci-device-probing-failure.131203/post-738089


OK, Realtek is in and working. BTW, I did try the "pci=realloc=off" grub edit, but it didn't work either. I can only see a few possibilities.
1 - the igb driver needs work.
2 - the MB firmware/igb driver are not compatible, since there are two pci devices using the same driver, but with very different parameters, which kinda goes back to the igb driver.
3 - well, that's actually all I can imagine.
I'd much rather be using Intel, but if I can't get it to work, as you said...

It is interesting that the realtek doesn't seem to be using the igb driver, or even the MB memory address space:


Code:
root@PVE:/# lspci -v -s 0a:00.0
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: fast devsel, IRQ 32, IOMMU group 30
        I/O ports at b000 [size=256]
        Memory at fb504000 (64-bit, non-prefetchable) [size=4K]
        Memory at fb500000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, IntMsgNum 1
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [170] Latency Tolerance Reporting
        Capabilities: [178] L1 PM Substates
        Kernel driver in use: r8169
        Kernel modules: r8169
 
I also flashed to the latest BIOS and cleared cmos for the MB.
If https://community.intel.com/t5/Ethe...ate-failure/m-p/1658703/highlight/true#M39875 is true, then NIC-fw should be already included in BIOS-fw, but I have no experience with supermicro to confirm this. That would also mean it is not always the latest if that is vendor-locked.

In contrary the single procedure with EFI: https://www.thomas-krenn.com/de/wiki/Supermicro_X11DPi-NT_NIC_Firmware_Update

But it could be a general problem with supermicro+intel, regardless of the specific chip:
https://www.reddit.com/r/supermicro/comments/13fdf2h/supermicro_nic_firmware_flash_borked_my_cards/
https://forum.level1techs.com/t/supermic-x12sca-5f-intel-nic-problems/192827
https://community.intel.com/t5/Ethe...Gbe-SFP-card-on-SuperMicro-H11SSL/m-p/1309038

It is interesting that the realtek doesn't seem to be using the igb driver, or even the MB memory address space:
That is normal behaviour, another chip(-gen) or even manufacturer=another driver.

I did try the "pci=realloc=off" grub edit
nano /etc/default/grub is if you boot proxmox legacy/CSM-on, revert it before you proceed. If you boot proxmox UEFI, then you need to edit nano /etc/kernel/cmdline:
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs pci=realloc=off
+ proxmox-boot-tool refresh + reboot.
or you try
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs pci=realloc=off iommu=pt
+ proxmox-boot-tool refresh + reboot.


Yup...at this point I'm fired empty with ideas. ;)
 
  • Like
Reactions: mike the newb
If https://community.intel.com/t5/Ethe...ate-failure/m-p/1658703/highlight/true#M39875 is true, then NIC-fw should be already included in BIOS-fw, but I have no experience with supermicro to confirm this. That would also mean it is not always the latest if that is vendor-locked.

In contrary the single procedure with EFI: https://www.thomas-krenn.com/de/wiki/Supermicro_X11DPi-NT_NIC_Firmware_Update

But it could be a general problem with supermicro+intel, regardless of the specific chip:
https://www.reddit.com/r/supermicro/comments/13fdf2h/supermicro_nic_firmware_flash_borked_my_cards/
https://forum.level1techs.com/t/supermic-x12sca-5f-intel-nic-problems/192827
https://community.intel.com/t5/Ethe...Gbe-SFP-card-on-SuperMicro-H11SSL/m-p/1309038


That is normal behaviour, another chip(-gen) or even manufacturer=another driver.


nano /etc/default/grub is if you boot proxmox legacy/CSM-on, revert it before you proceed. If you boot proxmox UEFI, then you need to edit nano /etc/kernel/cmdline:
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs pci=realloc=off
+ proxmox-boot-tool refresh + reboot.
or you try
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs pci=realloc=off iommu=pt
+ proxmox-boot-tool refresh + reboot.


Yup...at this point I'm fired empty with ideas. ;)

Well, you have some great ideas, and thank you!

But, get this...I just installed the i350 T4 card I ordered yesterday...and it works. It has different ICs than the other i350 T4. I'm going to look into the IC part numbers and see what I find. It would be nice to find something definitive. Maybe save someone the headache down the road.

Check it out:

Code:
root@PVE:/# dmesg | grep -i ethernet
[    2.068916] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.127168] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.184394] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[    2.242433] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.299332] igb 0000:08:00.1: Intel(R) Gigabit Ethernet Network Connection
[    2.356427] igb 0000:08:00.2: Intel(R) Gigabit Ethernet Network Connection
[    2.412551] igb 0000:08:00.3: Intel(R) Gigabit Ethernet Network Connection

root@PVE:/# lspci | grep -i ethernet
04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

root@PVE:/# dmesg | egrep -i --color 'igb'
[    2.068916] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.069216] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.126887] igb 0000:04:00.0: added PHC on eth0
[    2.127168] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.127427] igb 0000:04:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:25:90:7c:2e:ea
[    2.127733] igb 0000:04:00.0: eth0: PBA No: 104900-000
[    2.127957] igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    2.184116] igb 0000:04:00.1: added PHC on eth1
[    2.184394] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[    2.184624] igb 0000:04:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:25:90:7c:2e:eb
[    2.184928] igb 0000:04:00.1: eth1: PBA No: 104900-000
[    2.185154] igb 0000:04:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    2.242169] igb 0000:08:00.0: added PHC on eth2
[    2.242433] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.242672] igb 0000:08:00.0: eth2: (PCIe:5.0Gb/s:Width x4) 1a:4b:24:b5:ee:91
[    2.242982] igb 0000:08:00.0: eth2: PBA No: 106300-000
[    2.243216] igb 0000:08:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    2.299056] igb 0000:08:00.1: added PHC on eth3
[    2.299332] igb 0000:08:00.1: Intel(R) Gigabit Ethernet Network Connection
[    2.299582] igb 0000:08:00.1: eth3: (PCIe:5.0Gb/s:Width x4) 1a:4b:24:b5:ee:92
[    2.299899] igb 0000:08:00.1: eth3: PBA No: 106300-000
[    2.300133] igb 0000:08:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    2.356150] igb 0000:08:00.2: added PHC on eth4
[    2.356427] igb 0000:08:00.2: Intel(R) Gigabit Ethernet Network Connection
[    2.356683] igb 0000:08:00.2: eth4: (PCIe:5.0Gb/s:Width x4) 1a:4b:24:b5:ee:93
[    2.357016] igb 0000:08:00.2: eth4: PBA No: 106300-000
[    2.357276] igb 0000:08:00.2: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    2.412279] igb 0000:08:00.3: added PHC on eth5
[    2.412551] igb 0000:08:00.3: Intel(R) Gigabit Ethernet Network Connection
[    2.412797] igb 0000:08:00.3: eth5: (PCIe:5.0Gb/s:Width x4) 1a:4b:24:b5:ee:94
[    2.413118] igb 0000:08:00.3: eth5: PBA No: 106300-000
[    2.413370] igb 0000:08:00.3: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    4.327590] igb 0000:04:00.1 eno2: renamed from eth1
[    4.328742] igb 0000:04:00.0 eno1: renamed from eth0
[    4.331167] igb 0000:08:00.3 enp8s0f3: renamed from eth5
[    4.333215] igb 0000:08:00.2 enp8s0f2: renamed from eth4
[    4.334820] igb 0000:08:00.0 enp8s0f0: renamed from eth2
[    4.336511] igb 0000:08:00.1 enp8s0f1: renamed from eth3
[    9.492132] igb 0000:04:00.0 eno1: entered allmulticast mode
[    9.493291] igb 0000:04:00.0 eno1: entered promiscuous mode
[   13.526824] igb 0000:04:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

ethtool -i enp8s0f0
root@PVE:/# ethtool -i enp8s0f0
driver: igb
version: 6.14.11-1-pve
firmware-version: 1.63, 0x800009fa
expansion-rom-version:
bus-info: 0000:08:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
 
  • Like
Reactions: keeka
But, get this...I just installed the i350 T4 card I ordered yesterday...and it works.
Holy Moly :D
Either it is broken (sure, valid possible explanation but never at the first attempt, thats too easy) or maybe a fake card?

The counterfeit cards work fine but after a few months they start to fail.

https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/
https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/
 
Holy Moly :D
Either it is broken (sure, valid possible explanation but never at the first attempt, thats too easy) or maybe a fake card?



https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/
https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/

Maybe, but it worked with Windows 10. I bet every damn one of them I bought is a knockoff! Think I'm going to do some packet capturing to make sure nothing is "phoning home" :)
 
But, get this...I just installed the i350 T4 card I ordered yesterday...and it works. It has different ICs than the other i350 T4. I'm going to look into the IC part numbers and see what I find. It would be nice to find something definitive. Maybe save someone the headache down the road.
That's good news. Thanks for the updates on your progress. Can you post images of the suspect card?
 
  • Like
Reactions: mike the newb
That's good news. Thanks for the updates on your progress. Can you post images of the suspect card?


Sure can. I'll also post the new one. I didn't take the pic of the new one though.
 

Attachments

  • 71jpbkzFjeL._AC_SL1500_.jpg
    71jpbkzFjeL._AC_SL1500_.jpg
    107.9 KB · Views: 11
  • 20250905_074540.jpg
    20250905_074540.jpg
    1,016.6 KB · Views: 9
  • Like
Reactions: keeka
Let me know if you want full shots and I'll go take some.
Wouldn't hurt to warn others.

Looks like overall newer revision of the card, I'm not familiar with. My Fujitsu-branded card has the electronic components arranged differently in the right-hand edge area. But anyway this rainbowshiny QC passed sticker smells like fake. I've seen this on cheapest old single port Realteks but never on Intel.
 

Attachments

  • 2.jpg
    2.jpg
    274.9 KB · Views: 8
  • 1.jpg
    1.jpg
    169.5 KB · Views: 8