Proxmox Ceph network setup

[jslanier]

I am buying equipment now for this, and I want to get the right stuff. If I get 2 stacked 10/25G switches, I would use a LAG group with 4 25G ports each per host for Ceph, and I would need those switches to have at least 4 or 5 100G stacking ports in use to support the bandwidth needed.

But what if I just bought 2 100G switches and used 2 100G ports on each of the 5 hosts in a 2-port LAG? Well, I would have no storage if I ever needed to reboot the switch being used for Ceph. Is it possible to configure a primary network for Ceph with a secondary Ceph network if the primary is not available? That would be ideal, because I could have all 5 hosts with 4 100G ports (1 LAG with 2 ports and the other with 2) and each LAG group could go to each switch. Whenever switch 1 was not available, Ceph would start using the other Ceph network. Is this doable?

Thanks,

 

[gurubert]

You could create an active/backup bond interface on the Proxmox nodes.

I would start with just 2 25G ports in LACP bond per node and see if the bandwidth is enough.