connecting Ceph ProxmoxVE Nodes

AlBayrouni

New Member
Dec 22, 2021
18
1
3
24
Hello everyone, I need some advice in my local proxmoxVE cluster.

i am actually trying to set up a high availability ProxmoxVE cluster with Ceph inside of this cluster

After installing ProxmoxVE on my physical nodes, I am asking about the way to link them together.

I know that ceph Nodes should have 10GbE of bandwidth between each other.

So is the best way to bring a 10GbE Switch and link the nodes(that they must have 10GbE network cards also ?) to that switch ?, is this the only and best option ?

Is there any other option to link them through cables without having to buy the 10GbE switch ?

thank you so much
 
So is the best way to bring a 10GbE Switch
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.
Is there any other option to link them through cables without having to buy the 10GbE switch ?
Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
 
  • Like
Reactions: need2gcm
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
Hm, 100GBit/s switch...show me please switch with almost 1Tbps ports.
 
No, its best to use a 20/40/100 GBit/s Switch. ;-)

Just kidding. If this is a Homelab you might very well get away with 1 GBit/s. As always: it depends on your requirements. More important is probably to have an independent physical network for this. You can build that "slow" system and upgrade to 10 GBit/s if your use-case is not capable enough with this approach.

Well..., please see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
But be aware that this does not scale. To add a fourth node requires four more NICs plus cabling.

Disclaimer: I am NOT a ceph expert!
so the switch is the only option to make the three nodes communicate with each others ?
is there any networking strategy to link them by cables ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!