CEPH stretched cluster : impossible to add a monitor

[ghusson]

Hello,

With Damien BROCHARD (Ozerim), we made some extensive CEPH tests in lab.
As I thought we cannot add a new CEPH monitor in proxmox after stretched cluster mode has been enabled :

root@training2-vsrv2:~# pveceph mon destroy
400 not enough arguments
pveceph mon destroy <monid>
root@training2-vsrv2:~# pveceph mon destroy training2-vsrv2
Removed '/etc/systemd/system/ceph-mon.target.wants/ceph-mon@training2-vsrv2.service'.
root@training2-vsrv2:~# vi /etc/pve/ceph.conf # remove remaining IP of the mon
root@training2-vsrv2:~# pveceph mon create
monmaptool: monmap file /tmp/monmap
epoch 45
fsid 3ee28d49-9366-463e-9296-190cfb3822d2
last_changed 2025-09-17T21:55:54.908155+0200
created 2025-08-14T17:29:58.322233+0200
min_mon_release 19 (squid)
election_strategy: 3
stretch_mode_enabled 1
tiebreaker_mon training3-vsrv1
disallowed_leaders training3-vsrv1
0: [v2:10.20.1.1:3300/0,v1:10.20.1.1:6789/0] mon.training1-vsrv1; crush_location {datacenter=optiplex1}
1: [v2:10.20.1.31:3300/0,v1:10.20.1.31:6789/0] mon.training3-vsrv1; crush_location {datacenter=optiplex3}
2: [v2:10.20.1.21:3300/0,v1:10.20.1.21:6789/0] mon.training2-vsrv1; crush_location {datacenter=optiplex2}
3: [v2:10.20.1.22:3300/0,v1:10.20.1.22:6789/0] mon.training2-vsrv2
monmaptool: writing epoch 45 to /tmp/monmap (4 monitors)
Created symlink '/etc/systemd/system/ceph-mon.target.wants/ceph-mon@training2-vsrv2.service' -> '/usr/lib/systemd/system/ceph-mon@.service'.
root@training2-vsrv2:~# ceph mon set_location training2-vsrv2 datacenter=optiplex2
Error ENOENT: mon.training2-vsrv2 does not exist

This is due to the fact that CEPH need a location for the new mon, at creation time. the solution is :

systemctl disable ceph-mon@$(hostname).service
systemctl stop ceph-mon@$(hostname).service
pveceph mon destroy $(hostname)
vi /etc/pve/ceph.conf # remove remaining IP of the mon
pveceph mon create
systemctl start ceph-mon@$(hostname).service
systemctl disable ceph-mon@$(hostname).service
systemctl stop ceph-mon@$(hostname).service
/usr/bin/ceph-mon -f --cluster ceph --id $(hostname) --setuser ceph --setgroup ceph --set-crush-location datacenter="__MY_DC_LOCATION__"
# Ctrl + C
systemctl enable ceph-mon@$(hostname).service
systemctl start ceph-mon@$(hostname).service
ceph mon dump