Cephfs MDS only reachable in cluster network

[luigima]

Hey,

I am a little bit confused or maybe I am just missing a small detail.
My plan is to deploy and mount a cephfs directory on a samba VM.
Thus I created a MDS in the proxmox cephfs GUI but the address of the MDS is from my cluster network where the OSDs are communicating.
Well, I tried to mount the cephfs directory and it appears that it only works if I provide the cluster ip address of the MDS instance.
According to the docs this should not be the case shouldn't it?

The MDS instances should be reachable from the public network, see here: http://docs.ceph.com/docs/mimic/_images/ditaa-2452ee22ef7d825a489a08e0b935453f2b06b0e6.png

Which leads me to the question what I am doing wrong? Is this intended? Do the samba VM need to be in the cluster network? Can I somehow change the ip address of the MDS instance?


Best Regards
luigima

 

[Alwin]

If you used our default setup 'pveceph init' then the public and the cluster network do have the same IP range. You can separate them by editing the '/etc/pve/ceph.conf' and restart all Ceph services (may have a short disconnect).

 

[luigima]

This seems to be the right answer. I missed the configuration detail. Indeed, the cluster and public network are the same. However, I couldn't manage to migrate the network yet. Changing the ceph.conf file and then restarting all ceph services one by one does not seem to work. Results in timeouts..

 

[Alwin]

It very much depends on which network you migrated, but all ceph services on all nodes need to be restarted to get a healthy cluster.