[SOLVED] Help Please - Ceph Public/Cluster Networks

[gasmanc]

Background:
4 nodes
3 nodes have 2x10gbe and 2x1gbe
1 node has 1x10gbe and 2x1gbe
1 x unfi 16xg
NEW 1x mikrotik 4 port 10gbe switch

I had everything setup and working:
10.0.90.0/28 Proxmox corosync on 1gbe interfaces
10.095.0/28 Ceph cluster AND public network on 10gbe interfaces
10.0.50.0/24 Proxmox public network on second 10gbe interfaces

Everything was working great BUT the network would grind to a halt when copying large amounts of data - Ceph rebalancing was causing the USG (unifi) to become unresponsive.

So I purchased a 4 port 10gbe mikrotik switch and put my 10.0.95.0 Ceph networks on that. Unfortunately my VMs can no longer talk to Ceph storage. I know I have to seperate my Ceph cluster and public networks.

1. Is it as simple as changing the ceph.conf file on each of the nodes?

2. Can the ceph public network share the same interface as the Proxmox public network (it only communicates on a single port right?)

3. If not, should I move the Proxmox public network to the 1gbe interface and the ceph public network to the 10gbe interface? This would mean in a drop in throughout to my clients connecting to the cluster for storage.

Thanks in advance

 

[t.lamprecht]

1. Is it as simple as changing the ceph.conf file on each of the nodes?

Yes, then restart each OSD, one after the other. It will come up, see the new network request use that and tell the ceph monitors "where it is". So this can be done without interrupting the service, or no need to change all at once. Your /etc/pve/ceph configs global section should look somewhat like this:

[global]
     ....
     public_network = 10.0.50.0/24
     cluster_network = 10.095.0/28

....

2. Can the ceph public network share the same interface as the Proxmox public network (it only communicates on a single port right?)

yes, their port usage does not conflicts. Ceph Monitors listen on ports 3300 and 6789 by default.

 

[gasmanc]

Thank you so much. Worked like a charm. Separate switch for the ceph cluster network is definitely the way to go. Combining the proxmox and ceph public networks means I can keep everything on 10gbe. Perfect.

 

[sidereus]

Yes, then restart each OSD, one after the other.

Can I just type

systemctl restart ceph.target

instead of restarting one after other?