connecting Ceph ProxmoxVE Nodes

[AlBayrouni]

Hello everyone, I need some advice in my local proxmoxVE cluster.

i am actually trying to set up a high availability ProxmoxVE cluster with Ceph inside of this cluster

After installing ProxmoxVE on my physical nodes, I am asking about the way to link them together.

I know that ceph Nodes should have 10GbE of bandwidth between each other.

So is the best way to bring a 10GbE Switch and link the nodes(that they must have 10GbE network cards also ?) to that switch ?, is this the only and best option ?

Is there any other option to link them through cables without having to buy the 10GbE switch ?

thank you so much

 

[UdoB]

So is the best way to bring a 10GbE Switch

No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Is there any other option to link them through cables without having to buy the 10GbE switch ?

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!

 

[czechsys]

No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!

Hm, 100GBit/s switch...show me please switch with almost 1Tbps ports.

 

[AlBayrouni]

No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!

so the switch is the only option to make the three nodes communicate with each others ?
is there any networking strategy to link them by cables ?

 

[jsterr]

so the switch is the only option to make the three nodes communicate with each others ?
is there any networking strategy to link them by cables ?

Yes you can use https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server no need for switch. Does work with 3 servers or more when "SERVER-COUNT -1 = CEPH Ports needed per Node".