Thunderbolt as Ceph network in PVE cluster

[WoRie]

Hi,

I bought 3 identical asrock nuc boxes with 64gb ram, Intel 1240p alder lake CPUs and 2x2,5gbe nics each. I'm thinking about installing ceph but read in the docs, that at least 10gbe networking is recommended.

I have a 24 port 10gbe switch available, so I could connect 3 thunderbolt to 10gbe adapters and call it a day, but each one would be at least 190 euros.

I've discovered, that one could connect 2 apple macs through thunderbolt directly and get a link local network out of the box with at least 10gbe. Is this possible in proxmox as well? And if so, how to set it up in a 3 node cluster for ceph?

Thanks in advance for any input

 

[WoRie]

Just to update this:

in my case, since I'm not using Intel NUCs, there is no "official" thunderbolt support as far as I understand. However, the USB ports should be USB4. The most current linux kernel will implement flow control and other features to USB4NET, which would be the host interface network protocol.

So it remains to be seen, if and when this networking method will be implemented into the gui. I found a reddit comment which seems to have sucessfully implemented this, however in my case, no thunderbolt network device is being created. But as stated, this can be due to the missing certification for TB.

If anyone has any ideas, how I could manage to connect at least 2 preferably 3 hosts in a ceph cluster through the USB ports, please let me know

 

[kafisatz]

Hi. Have you made any progress on this? I am trying to set up a 3 node cluster of NUC13s using TB4

 

[Avolon42x]

I‘m not yet a proxmox user but have been running a small kubernetes cluster in my lab. I’ve been dabbling with various ways to improve ceph on non enterprise hardware and the TB4 networking idea came also to my mind. I don’t see any problem why it should not work. Especially since you can separate the networks in ceph for front-and backend : https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
So using the TB4-ring network as backend and the standard eth as frontend.
Since my current micro pcs don’t have TB I’m holding it out till I can upgrade.

 

[kafisatz]

Thanks. In the meantime I have this up and running. I use static ip v4 adresses for my 3 nodes (intel nuc13 i7). I get about 20gbps on Thunderbolt.

 

[Hillmann]

Thanks. In the meantime I have this up and running. I use static ip v4 adresses for my 3 nodes (intel nuc13 i7). I get about 20gbps on Thunderbolt.

I'm looking for a small setup with CEPH/PVE and came across this thread. NUC's seems to be a good fit for small homelabs.
So, just to be sure, you had success with setting up 3 nodes based on NUC13 I7-1360P, running the CEPH network on the TB4 ports in a mesh?

 

[kafisatz]

yes. i7-1360P, 64 GB Ram each, 980 pro 2TB NVME drives with Ceph.

 

[lytscha]

i'm thinking about this too,

what i have in mind is the minisforum UM690 Pro

why i think this could work out quite well:

2x RJ45 2.5G
2x USB4 Typ C
8c/16t at max 4.9 GHz
DDR5
2x PCIe gen 4 NVME Slots

would it be possible to connect the ceph frontend network via one of the 2.5GBs LAN Ports and the ceph backend via usb 4, there are 2 usb4 ports, so I could connect 3 Nodes to each other directly with an usb typ c cable

 

[kafisatz]

would it be possible to connect the ceph frontend network via one of the 2.5GBs LAN Ports and the ceph backend via usb 4, there are 2 usb4 ports, so I could connect 3 Nodes to each other directly with an usb typ c cable

This is exactly what I did (three cables, plugged into 6 ports (two in each computer)). But in my case, these were ThunderBolt ports, not USB.

 

[NeedsMoreRGB]

Wendell from Level 1 Tech's has a write up on thunderbolt networking in Proxmox, if I'm not mistaken.

 

[lytscha]

This is exactly what I did (three cables, plugged into 6 ports (two in each computer)). But in my case, these were ThunderBolt ports, not USB.

may i ask how its working out in terms of stability and reliability?

 

[no-usernames-left]

Wendell from Level 1 Tech's has a write up on thunderbolt networking in Proxmox, if I'm not mistaken.

https://forum.level1techs.com/t/3-node-65w-proxmox-thunderbolt-network-cluser/200207