connecting Ceph ProxmoxVE Nodes

A

AlBayrouni

New Member
Dec 22, 2021
18
1
3
24
Hello everyone, I need some advice in my local proxmoxVE cluster.

i am actually trying to set up a high availability ProxmoxVE cluster with Ceph inside of this cluster

After installing ProxmoxVE on my physical nodes, I am asking about the way to link them together.

I know that ceph Nodes should have 10GbE of bandwidth between each other.

So is the best way to bring a 10GbE Switch and link the nodes(that they must have 10GbE network cards also ?) to that switch ?, is this the only and best option ?

Is there any other option to link them through cables without having to buy the 10GbE switch ?

thank you so much
 
AlBayrouni said:
So is the best way to bring a 10GbE Switch
Click to expand...
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.
AlBayrouni said:
Is there any other option to link them through cables without having to buy the 10GbE switch ?
Click to expand...
Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
 
  • Like
Reactions: need2gcm
UdoB said:
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
Click to expand...
Hm, 100GBit/s switch...show me please switch with almost 1Tbps ports.
 
UdoB said:
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
Click to expand...
so the switch is the only option to make the three nodes communicate with each others ?
is there any networking strategy to link them by cables ?
 
You must log in or register to reply here.
Top Bottom