Guidance: Installing second network and re-configuring Ceph

[SAB]

So ... I'm a bit of an amateur here but have recently created my first 5-node HA Proxmox cluster using Ceph storage (all nodes having one network interface). Everything is working quite well, I'm spawning ubuntu cloud-init configured VMs and about to try my hand at using terraform. However, the network isn't partially robust, and I thought I might alleviate some traffic congestion by configuring a second network to improve redundancy and allow Ceph to do its thing with less impact to the primary network.

I can add a second interface to each node, for which all nodes will report it as vmbr1. I'll then install a second switch and connect them all up via another interface on the main router / gateway.

Unfortunately, networking is my weakest area in IT, and I'm not sure how best go about adding a second network with regards to the proxmox host(s) and ceph configuration post-change, and although my set up isn't a production thing, I would probably "sigh" real hard if I destroyed it.

I've looked for a guide on this one, but can't seem to find one - so if anyone knew of one I'd be happy to be pointed in the right direction and I'll have a stab at figuring it out myself. Of course, if anyone considered this trivial stuff, I'd be happy to take your advice directly as well!

 

[gurubert]

For the second network (the Ceph cluster network) you do not need an outside connection and in particular no default gateway (there can only be one).

Use the IP network you designated for this second network interface as Ceph cluster network in the configuration and Ceph should automatically use it.

For Proxmox you could create a second corosync ring over that network to improve cluster stability.

 

[SAB]

For Proxmox you could create a second corosync ring over that network to improve cluster stability.

Can a second network be added after a cluster has been created without one? I don't see any GUI options to add a second link; perhaps there is a cmdline option to so?

 

[mfed]

I would say the very first step would be to configure your new network and ensure hosts can talk to each other on this network. I believe that all can be done from GUI and not related to ceph at all. You will also need to enable routing between the old subnet and the new subnet. During the migration of ceph to the new subnets, there will be a time when OSDs and mons on different subnets need to talk to each other...

After that done you will need to put both old and new subnets as public_network and cluster_networkin /etc/ceph/ceph.conf as those act as a sort of access lists to allow incoming communications. Then you can play with public_addr and cluster_addr on osds and mon_addr on mons to force them use an IP address from a specific subnet (need to restart each osd/mon for the new configuration to take effect). After everything migrated to the new subnet, you can remove the old subnet from public_network and cluster_network and after that the routing between the ceph subnet and your old subnet is no longer required.

That was the approach I used when I migrated my ceph network to another subnet...