Ceph multi-public-network setup: CephFS on separate network

M

MMartinez

Renowned Member
Dec 11, 2014
56
6
73
Hi,

I'm trying to setup a secondary public network to a ceph cluster and I have problems. I've read two posts related to that but they are a bit different and that's why I start a new thread.
My scenario. I have 8 ceph nodes using a dedicated IP (192.168.50.0/24) network to Ceph public and private. The ceph network is isolated on a VLAN, no external route to it.

I've been using Ceph pools (RBD) until now, but no CephFS. Now I want to use CephFS from many VM (with docker) instead of NFS mounts on TrueNAS to achieve HA on that mounts.

That VM are on a different routed network and I want to publish Ceph to clients on that network. I've done the following:
* Define a direct connection on the proxmox hosts to that VM network (192.168.60.0/24)
* Modify the /etc/pve/ceph.conf file to add the new public network as the second one (public_network = 192.168.50.0/24,192.168.60.0/24).
* Reload all monitors, managers and metadata servers.
* Destroy and recreate two monitors. Now they listen on both networks as I see by executing "ceph mon dump".

After setting up the debian client on the VM, I'm not able to connect to ceph:

mount -t ceph 192.168.60.11:/ /mnt/cephfs -o name=admin,fs=cephfs
Click to expand...

I've seen that the Metadata servers are listening on the first public network, so I've tried to change that by setting the following configuration:
[mds.pvenode01]
host = pvenode01
mds_standby_for_name = pve
public_addr = 192.168.60.11
Click to expand...

After restarting the MDS daemon I see on the GUI that now it listens on the 192.168.60.0/24 network, but I'm not able to connect from the debian VM. I see this on the logs:

[2029655.252225] libceph: wrong peer, want (1)192.168.60.11:6789/0, got (1)192.168.50.11:6789/0
[2029655.252232] libceph: mon0 (1)192.168.60.11:6789 wrong peer at address
[2029655.763475] ceph: No mds server is up or the cluster is laggy
Click to expand...

It seems that the MON tells the client to use the address on the old network. On the proxmox hosts I see the new cephfs volume mounted after changing the MDS public address so I'm sure it is working.

I'd like to keep the old private/public address on the isolated network to be used for the RDB pools for the proxmox VM disks and setup a new public network for clients on another network. Maybe the scenario that I want to setup is not a usual one.

Please, can you give me a recomendation on how to correctly set up CephFS clients on a separate public network without affecting the RBD setup?

I'm using Proxmox 9.1 and Ceph 19.2.3.

Kind regards,

Manuel Martínez
 
Let me explane it a bit more.

What I want is not exactly to separate Ceph RBD from CephFS. I want to give access to VM as Ceph clients from a different network.

On proxmox nodes I have a 10GbE dual NIC. One port dedicated to Ceph (public and private) and the other is shared by corosync, truenas storage and VM.

I know that I can rename the Ceph interface as a VM bridge and give the VM direct access to the Ceph Network (public and private) without modifying the ceph configuration but I'd like to avoid client traffic on this dedicated network.

If I change the public ceph network to be on the VM network then all the current ceph RBD client traffic will use the second interface and it could be worst for ceph, corosync, ...

That's why I want two public networks: one for the current VM disks on Ceph RBD storage and another one for VM clients.

Thanks
 
Last edited:
Thanks. It seems that the problem is described on this "tip".
If you specify more than one IP address and subnet mask for either the public or the cluster network, the subnets within the network must be capable of routing to each other. Additionally, make sure you include each IP address/subnet in your IP tables and open ports for them as necessary.
Click to expand...

I was trying to keep ceph private network isolated so it is not routed. It looks that both public networks needs to be routed and visible between them.

In that case, I will choose the other aproach. Add a second interface into the ceph network to the VM's. It seems easy for me and all the ceph traffic will still on that 10Gbps segment, and at the same time I keep the ceph network without external routes.

Thanks again,

Manuel Martínez.
 
I dont think that note is as wide in scope as the working seem to suggest.

Public and Private (cluster) traffic dont need routability to each other to function, so you're safe on that front. Reading up further, monitors can only have one pinned IP, which means that while you can have multiple PUBLIC vlans, they must all be able to reach the monitors- which means all your public vlans must have a route. You can still have insulation by allowing traffic ONLY to the monitors instead of full.

So you can have a "master" public vlan, eg (10.10.10.0/30) with monitor addresses here (eg, .1, .2, .3). you can then have OTHER subnets (eg 10.10.11.0.24) which can route to the monitor subnet. you dont need the subnets to be able to talk to each other, just to the monitors.

edit forgot the OSDs. those nodes should also be in the "master" public vlan.

MMartinez said:
Add a second interface into the ceph network to the VM's
Click to expand...
This is best practices anyway, since all IO generates traffic on both public and private subnets; if you only have a single interface for both it would have to carry twice the packets for an i/o effectively halving its throughput potential.
 
Last edited:
  • Like
Reactions: gurubert
All Ceph daemons register in the CRUSH map wih exactly one IP and one port.
You can have multiple public networks in the config but this is for the case where different hosts of the Ceph cluster are in different IP networks.
It is not practical to have one Ceph host with IPs from multiple public networks.
And each Ceph client needs to be able to talk to each Ceph daemon (MON, OSD and MDS in case of CephFS).

In your case the easiest solution would be to add an interface to all the VMs that should mount CephFS in the VLAN of 192.168.50.0/24.
 
Yes, thanks. That's what I'm going to do.

Anyway, I'm considering the possibility to separate Ceph public and private networks first, and then add the VM interface to the public network.

This way the setup will be more secure.

Regards.
 
No, the separation of the Ceph public and cluster (not private) networks has nothing to do with security.
The cluster network is there to transport replication and recovery traffic between the OSD nodes. It can be configured if there is a separate physical network available that provides more bandwidth and speed than the public network. Think 25G cluster and 10G public.
 
You must log in or register to reply here.
Top Bottom
Back