ceph-mon will not start start after ceph upgrade to 16.2.6 from 15.2.14

[Daniel Keller]

i have a ceph cluster with three monitors after the upgrade to 16.2.6 one mon is up and runing the other two monitors do not want to start and bring the following error

ceph-mon[811509]: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input'
ceph-mon[811509]: what(): void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version v < 7: Malformed input
ceph-mon[811509]: *** Caught signal (Aborted) **
ceph-mon[811509]: in thread 7f615e582700 thread_name:ms_dispatch

what is the best way to fix this. delete the two monitors and recreate?

if so what is the best way to do this without a quorum?

 

[Daniel Keller]

I was able to complete the upgrade by changing to version 16.2.5.

after that I tried to upgrade from version 16.2.5 to version 16.2.6 but as soon as I restart the second monitor they crash again with the same error message.

after I switched back to version 16.2.5 it works again

 

[not_a_salmon]

I came across your post while dealing with a similar issue.
In case you or anyone else is still dealing with this issue, adding the following to my my ceph.conf resolved the issue (located under /etc/pve/ceph.conf I believe):

[mon] mon_mds_skip_sanity = true

This is also mentioned on the upgrade page.
https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific#Monitor_crashes_after_upgrade

After adding this setting all I did was give my system a reboot and the ceph monitors were back in order. (although a full ceph restart probably would have sufficed)