[SOLVED] Intel X540-T2 on PVE8-Ceph-Cluster without Switch (Broadcast)

[Noah0302]

Hello everyone,

first of all, I tried searching the forums on the Topic, but I have not really found anything that is relatively up-to-date.

My Question is, how a dual 10G NIC with the Intel X540-T2 Chip would fare if I used it run my Ceph-Cluster in Broadcast-Mode so I can skip buying a 10G Switch.
This will be similair to how I currently run my Corosync-Network:


My Cluster is currently running like this:
3 Nodes with seperate Networks for Corosync, Webgui/Ceph and VM-Bridge.

Ceph:
1x2.5G Network through a Switch
1x1TB Samsung 980 Pro OSD per Node

Corosync:
2x1G Network Broadcast without a Switch (Broadcast)

VM-Guests:
1x10G for Node 1 (Ryzen 9 3950X, ASUS X570 Mainboard, 128GB ECC)
1x1G for Node 2 and 3 (i7 8700k, Gigabyte Z390 Mainboard, 64GB RAM + i5 4th Gen, 16GB RAM)



I plan on buying a NIC with dual 10G Ports and an Intel X540 Chipset (Something like Supermicro AOC-STG-i2T REV 2.01 or this), so I can create a separate 10G Network only for Ceph.
I also want to use Broadcast, so I dont have to buy another Switch, since my 10G Ports are already in Use. I think it should work, since my Corosync has been rock solid ever since.



Thank you for reading!

 

[Noah0302]

bump

 

[Noah0302]

Anyone?

 

[Noah0302]

Well since no one seems to have any experience regarding the topic, I just decided to test it and bought 3 Supermicro AOC-STG-i2T REV 2.01 via eBay.

It actually works just as expected! I redid my entire Network-Config however. Now I run Corosync and the Webinterface via 1G (vmbr0), VMs via 2.5G (vmbr1) and Ceph via 10G (bond111).


I tested it in the same Windows Server 2022 VM before and after switching from 2.5G 1500 via a Switch to 10G 9000 via Broadcast:


Keep in mind that I do not use Datacenter SSDs, I use consumer NVMes and only one OSD per Node via a PCIe to NVMe Card in very much mixed performance Nodes (Ryzen 3950x, i7 8700k and i5 4670).
However the uptick in speed is really noticeable, not just in Benchmarks, Windows and my Linux VMs also feel snappier!

 

[crysty]

Hello
How did you succeeded to have the network device in the Network?
I installed X540-AT2 interface, but I can't see the network device in the Network. Probably is relate to the drivers, but I don't know how to install it.

dmesg | grep ixgbe
[ 1.012430] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 1.012432] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 1.012555] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 1.027548] ixgbe 0000:01:00.0: Adapter removed
[ 1.027612] ixgbe: probe of 0000:01:00.0 failed with error -5

lspci -k | sed -n '/Ethernet/,/Kernel/p'
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: Inspur Electronic Information Industry Co., Ltd. 10G base-T DP EP102Ti3A Adapter
Kernel modules: ixgbe

lshw -class network
*-network:0 UNCLAIMED
description: Ethernet controller
product: Ethernet Controller 10-Gigabit X540-AT2
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress cap_list
configuration: latency=0
resources: memory:df600000-df7fffff memory:dfa00000-dfa03fff memory:dfa04000-dfb03fff memory:dfb04000-dfc03fff

 

[Noah0302]

Hello
How did you succeeded to have the network device in the Network?
I installed X540-AT2 interface, but I can't see the network device in the Network. Probably is relate to the drivers, but I don't know how to install it.

dmesg | grep ixgbe
[ 1.012430] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 1.012432] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 1.012555] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 1.027548] ixgbe 0000:01:00.0: Adapter removed
[ 1.027612] ixgbe: probe of 0000:01:00.0 failed with error -5

lspci -k | sed -n '/Ethernet/,/Kernel/p'
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: Inspur Electronic Information Industry Co., Ltd. 10G base-T DP EP102Ti3A Adapter
Kernel modules: ixgbe

lshw -class network
*-network:0 UNCLAIMED
description: Ethernet controller
product: Ethernet Controller 10-Gigabit X540-AT2
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress cap_list
configuration: latency=0
resources: memory:df600000-df7fffff memory:dfa00000-dfa03fff memory:dfa04000-dfb03fff memory:dfb04000-dfc03fff

I actually did not do anything.
I just installed the NIC, booted the Servers and they already showed in the GUI!

Which Proxmox Version do you run?

 

[crysty]

I use latest Proxmox 8.
I also installed the NIC but is not show in the network list and got the above errors.