ceph errors after upgrade to 19.2.3

P

PmUserZFS

Well-Known Member
Feb 2, 2018
142
7
58
the monitor on nod pm2 failes with

ov 29 22:18:54 pm2 systemd[1]: ceph-mon@pm2.service: Scheduled restart job, restart counter is at 3.
Nov 29 22:18:54 pm2 systemd[1]: Started ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:18:54 pm2 ceph-mon[6532]: 2025-11-29T22:18:54.309+0100 76e5de0fbcc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:18:54 pm2 ceph-mon[6532]: 2025-11-29T22:18:54.309+0100 76e5de0fbcc0 -1 Processor -- bind was unable to bind. Trying again in 5 seconds
Nov 29 22:18:59 pm2 ceph-mon[6532]: 2025-11-29T22:18:59.309+0100 76e5de0fbcc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:18:59 pm2 ceph-mon[6532]: 2025-11-29T22:18:59.309+0100 76e5de0fbcc0 -1 Processor -- bind was unable to bind. Trying again in 5 seconds
Nov 29 22:19:04 pm2 ceph-mon[6532]: 2025-11-29T22:19:04.309+0100 76e5de0fbcc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:19:04 pm2 ceph-mon[6532]: 2025-11-29T22:19:04.310+0100 76e5de0fbcc0 -1 Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
Nov 29 22:19:04 pm2 ceph-mon[6532]: 2025-11-29T22:19:04.310+0100 76e5de0fbcc0 -1 unable to bind monitor to [v2:172.16.50.20:3300/0,v1:172.16.50.20:6789/0]
Nov 29 22:19:04 pm2 systemd[1]: ceph-mon@pm2.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 22:19:04 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:19:14 pm2 systemd[1]: ceph-mon@pm2.service: Scheduled restart job, restart counter is at 4.
Nov 29 22:19:14 pm2 systemd[1]: Started ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:19:14 pm2 ceph-mon[6830]: 2025-11-29T22:19:14.559+0100 7c278a9a3cc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:19:14 pm2 ceph-mon[6830]: 2025-11-29T22:19:14.559+0100 7c278a9a3cc0 -1 Processor -- bind was unable to bind. Trying again in 5 seconds
Nov 29 22:19:19 pm2 ceph-mon[6830]: 2025-11-29T22:19:19.560+0100 7c278a9a3cc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:19:19 pm2 ceph-mon[6830]: 2025-11-29T22:19:19.560+0100 7c278a9a3cc0 -1 Processor -- bind was unable to bind. Trying again in 5 seconds
Nov 29 22:19:24 pm2 ceph-mon[6830]: 2025-11-29T22:19:24.561+0100 7c278a9a3cc0 -1 Processor -- bind unable to bind to v2:172.16.50.20:3300/0: (99) Cannot assign requested address
Nov 29 22:19:24 pm2 ceph-mon[6830]: 2025-11-29T22:19:24.561+0100 7c278a9a3cc0 -1 Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
Nov 29 22:19:24 pm2 ceph-mon[6830]: 2025-11-29T22:19:24.561+0100 7c278a9a3cc0 -1 unable to bind monitor to [v2:172.16.50.20:3300/0,v1:172.16.50.20:6789/0]
Nov 29 22:19:24 pm2 systemd[1]: ceph-mon@pm2.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 22:19:24 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:19:34 pm2 systemd[1]: ceph-mon@pm2.service: Scheduled restart job, restart counter is at 5.
Nov 29 22:19:34 pm2 systemd[1]: ceph-mon@pm2.service: Start request repeated too quickly.
Nov 29 22:19:34 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:19:34 pm2 systemd[1]: Failed to start ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:22:47 pm2 systemd[1]: ceph-mon@pm2.service: Start request repeated too quickly.
Nov 29 22:22:47 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:22:47 pm2 systemd[1]: Failed to start ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:27:18 pm2 systemd[1]: ceph-mon@pm2.service: Start request repeated too quickly.
Nov 29 22:27:18 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:27:18 pm2 systemd[1]: Failed to start ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:34:21 pm2 systemd[1]: ceph-mon@pm2.service: Start request repeated too quickly.
Nov 29 22:34:21 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:34:21 pm2 systemd[1]: Failed to start ceph-mon@pm2.service - Ceph cluster monitor daemon.
Nov 29 22:38:54 pm2 systemd[1]: ceph-mon@pm2.service: Start request repeated too quickly.
Nov 29 22:38:54 pm2 systemd[1]: ceph-mon@pm2.service: Failed with result 'exit-code'.
Nov 29 22:38:54 pm2 systemd[1]: Failed to start ceph-mon@pm2.service - Ceph cluster monitor daemon.
Click to expand...
I have restarted

then I ran

root@pm2:~# ls -ld /var/lib/ceph/mon/ceph-pm2/
stat /var/lib/ceph/mon/ceph-pm2/store.db
drwxr-xr-x 3 ceph ceph 7 Nov 29 18:45 /var/lib/ceph/mon/ceph-pm2/
File: /var/lib/ceph/mon/ceph-pm2/store.db
Size: 35 Blocks: 17 IO Block: 2560 directory
Device: 0,28 Inode: 100616 Links: 2
Access: (0755/drwxr-xr-x) Uid: (64045/ ceph) Gid: (64045/ ceph)
Access: 2025-11-29 22:19:14.542863780 +0100
Modify: 2025-11-29 22:19:14.542863780 +0100
Change: 2025-11-29 22:19:14.542863780 +0100
Birth: 2024-01-10 13:04:21.029600752 +0100
root@pm2:~# ceph mon remove pm2

^CCluster connection aborted
root@pm2:~# systemctl stop ceph-mon@pm2.service ceph-mgr@pm2.service 2>/dev/null || true
rm -rf /var/lib/ceph/mon/ceph-pm2/*
Click to expand...


Nov 29 22:41:39 pm2 pvedaemon[3839]: <root@pam> starting task UPID:pm2:00005980:00023596:692B6893:srvstart:mgr.pm2:root@pam:
Nov 29 22:41:39 pm2 systemd[1]: ceph-mgr@pm2.service: Start request repeated too quickly.
Nov 29 22:41:39 pm2 systemd[1]: ceph-mgr@pm2.service: Failed with result 'start-limit-hit'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit ceph-mgr@pm2.service has entered the 'failed' state with result 'start-limit-hit'.
Nov 29 22:41:39 pm2 systemd[1]: Failed to start ceph-mgr@pm2.service - Ceph cluster manager daemon.
░░ Subject: A start job for unit ceph-mgr@pm2.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit ceph-mgr@pm2.service has finished with a failure.
░░
░░ The job identifier is 4509 and the job result is failed.
Nov 29 22:41:39 pm2 pvedaemon[22912]: command '/bin/systemctl start ceph-mgr@pm2' failed: exit code 1
Nov 29 22:41:39 pm2 pvedaemon[3839]: <root@pam> end task UPID:pm2:00005980:00023596:692B6893:srvstart:mgr.pm2:root@pam: command '/bin/systemctl start ceph-mgr@pm2' failed: exit code 1
Click to expand...
 
Last edited:
reall odd I could not in any way recreate the monitor, I was rm -rf it with

rm -rf /var/lib/ceph/mon/ceph-pm2rm -f /etc/ceph/ceph.confpveceph mon create


it was still running with the same old IP, even rebooted. I hgad to change ceph public IP and recreate the manager and monitor. really odd.
 
You must log in or register to reply here.
Top Bottom