ceph 1 daemons have recently crashed

[dbrega]

I have a datacenter with three PVEs and Ceph in a fullmesh configuration.
When I log in, I often get this error:
1 daemons have recently crashed
mon.pve2 crashed on host pve at ...............
This is the output of the last cpeh crash info

 

[longer]

Is it always mon.pve2 ?
You could attempt to destroy and create this mon in case other mon are fine.

Could you also share:
~# ceph status

 

[dbrega]

Could you also share:
~# ceph status

I confirm that is always mon.pve2

 

[longer]

So at the moment, the daemon successfully restarted, and running for more than hour.

Looking at journalctl may provide related information, so the last crash from shared crash info was at 2026-07-27 23:21, therefore take look what happend before that:
root@pve2 :# journalctl -k --since "2026-07-27 22:00:00" --until "2026-07-27 23:30:00" > /tmp/syslog.txt.gz
Attach the generated log file if its fine for you.

 

[dbrega]

as you advised me I removed the mon.pve2 and tried to recreate it on pve2 but now I get this error even though the mon.pve2 is not actually present in the gui
this is the gui that possible see mon.pve2 is not present

this is the error that appears when I try to create the new

 

[dbrega]

So at the moment, the daemon successfully restarted, and running for more than hour.

Looking at journalctl may provide related information, so the last crash from shared crash info was at 2026-07-27 23:21, therefore take look what happend before that:
root@pve2 :# journalctl -k --since "2026-07-27 22:00:00" --until "2026-07-27 23:30:00" > /tmp/syslog.txt.gz
Attach the generated log file if its fine for you.

the generated file is empty, maybe because I deleted mon.pve2 before generating it

 

[longer]

as you advised me I removed the mon.pve2 and tried to recreate it on pve2 but now I get this error even though the mon.pve2 is not actually present in the gui
this is the gui that possible see mon.pve2 is not present
View attachment 99159
this is the error that appears when I try to create the new
View attachment 99160

You can attempt again after couple of minutes, if still same issue check:
- Check the Active Mon Map
ceph mon dump

 

[longer]

the generated file is empty, maybe because I deleted mon.pve2 before generating it

Please try without the -k option:
root@pve2 :# journalctl --since "2026-07-27 22:00:00" --until "2026-07-27 23:30:00" > /tmp/syslog.txt.gz

 

[dbrega]

You can attempt again after couple of minutes, if still same issue check:
- Check the Active Mon Map
ceph mon dump

I also rebooted pve2
The problem persist.

 

[dbrega]

Please try without the -k option:
root@pve2 :# journalctl --since "2026-07-27 22:00:00" --until "2026-07-27 23:30:00" > /tmp/syslog.txt.gz

generated file attached

 

[longer]

View attachment 99161
I also rebooted pve2
The problem persist.

can you share:
~# cat /etc/pve/ceph.conf

 

[dbrega]

can you share:
~# cat /etc/pve/ceph.conf

I also tried to restart mon.pve1 and mon.pve3 without resolve the problem

 

[longer]

View attachment 99164

Okay, so 10.10.10.2 is still listed at the end of the mon_host line inside the [global] section.
You can use your preferaed editor and delete the ip: 10.10.10.2. make a copy of the file before making any change.

 

[longer]

Once finish editing and saving that file, please proceed with recreating the monitor using the Proxmox GUI.

 

[dbrega]

Now ok
Now I'm monitoring whether the mon.pve2 warning appears again
Thank you very much

 

[longer]

Always welcome. I will keep an eye on the thread.
In case it reoccur share journalctl again covering issue time:
root@pve2 :# journalctl --since "2026-07-xx xx:00" --until "2026-07-yy yy:yy" > /tmp/syslog.txt.gz