[SOLVED] Help Please - Ceph Public/Cluster Networks

gasmanc

New Member
Jul 29, 2019
11
0
1
42
Background:
4 nodes
3 nodes have 2x10gbe and 2x1gbe
1 node has 1x10gbe and 2x1gbe
1 x unfi 16xg
NEW 1x mikrotik 4 port 10gbe switch

I had everything setup and working:
10.0.90.0/28 Proxmox corosync on 1gbe interfaces
10.095.0/28 Ceph cluster AND public network on 10gbe interfaces
10.0.50.0/24 Proxmox public network on second 10gbe interfaces

Everything was working great BUT the network would grind to a halt when copying large amounts of data - Ceph rebalancing was causing the USG (unifi) to become unresponsive.

So I purchased a 4 port 10gbe mikrotik switch and put my 10.0.95.0 Ceph networks on that. Unfortunately my VMs can no longer talk to Ceph storage. I know I have to seperate my Ceph cluster and public networks.

1. Is it as simple as changing the ceph.conf file on each of the nodes?

2. Can the ceph public network share the same interface as the Proxmox public network (it only communicates on a single port right?)

3. If not, should I move the Proxmox public network to the 1gbe interface and the ceph public network to the 10gbe interface? This would mean in a drop in throughout to my clients connecting to the cluster for storage.

Thanks in advance
 
Last edited:
1. Is it as simple as changing the ceph.conf file on each of the nodes?

Yes, then restart each OSD, one after the other. It will come up, see the new network request use that and tell the ceph monitors "where it is". So this can be done without interrupting the service, or no need to change all at once. Your /etc/pve/ceph configs global section should look somewhat like this:
Code:
[global]
     ....
     public_network = 10.0.50.0/24
     cluster_network = 10.095.0/28

....

2. Can the ceph public network share the same interface as the Proxmox public network (it only communicates on a single port right?)

yes, their port usage does not conflicts. Ceph Monitors listen on ports 3300 and 6789 by default.
 
  • Like
Reactions: gasmanc
Thank you so much. Worked like a charm. Separate switch for the ceph cluster network is definitely the way to go. Combining the proxmox and ceph public networks means I can keep everything on 10gbe. Perfect.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!