Proxmox 8.3 CEPH upgrade reef to squid error with monitor on 1 node

[aluisell]

Hi,

I do have a cluster with 3 nodes (pve1, pve2, pv3) with CEPH installed 18.2.4 (REEF).
Today I have decided to perform update to Squid version, following your instruction here https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid

No problem till the upgrade but once I have performed the command
systemctl restart ceph-mon.target
on node pve2 with no error, I did check the status with ceph -s and got error with monitor out of quorum.

I did perform some checks. comunication between nodes seems to work fine. no iptables rules.
I did also tried to retrieve the monmap and inject it again but seems to not work.

Any idea please?

Thanks

 

[aluisell]

I think I have identified the main root cause which is the low space on /
lower than 5% that was causing monitor alarm

I did freeup space, destroyed and re-created the monitor and after few manual stepping was able to make it working again.
Anyway I did notice that once you remove a monitor the ip of it remain in the main ceph.conf file and you can't create it again on the same node unless you manually remove the ip from the ceph.conf. Any chance you can fix it in your script from the web interface? Thanks