cluster issues after upgrading to pve9

L

Liviu Sas

Well-Known Member
Jun 11, 2018
31
5
48
46
Hello,

I just updated my cluster to proxmox 9, and most nodes went well, except 2 of them that ended up in a very weird state.
Those 2 nodes hung at "Setting up pve-cluster". And I have noticed that /etc/pve was locked (causing any process that tried to access it to lock in a "D" state)
The only way to finish the upgrade was to reboot in recovery mode.

After the upgrade was finished, all looked good until I rebooted any one of those nodes. After the reboot, they would come up and /etc/pve would be stuck again.
This would cause /etc/pve to become stuck on other nodes in the cluster, causing them to go into a reboot loop.

The only way to recover these node is to boot in recovery mode, do a "apt install --reinstall pve-cluster" and press CTRL+D to continue boot and they come up and wotrk as expected.
But if any of these 2 nodes reboot again, the situation repeats (/etc/pve becomes stuck in all nodes and they enter the reboot loop).

It looks like something did not get upgraded correctly on those 2 nodes, but I can't figure out what.
I must mention that during this time, pvecm status shows a healthy cluster at all times.

Cheers,
Liviu
 
Hi!

Could you post some syslog on these nodes? Is HA setup on these nodes?
 
Hello,

I have rebooted one of the nodes yesterday to reproduce the issues.
See the syslog attached:
12:25:52 - first uninterrupted boot, cluster fail
12:28:31- second uninterrupted boot, cluster fail
12:32:55 - boot in recovery mode, performed "apt install --reinstall pve-cluster pve-firewall pve-manager pve-ha-manager" and continued boot, cluster healthy

Currently I have some HA services but they are all ignored:

Code: 
root@pvegiant:~# ha-manager status
quorum OK
master pvenew (active, Mon Oct  6 10:36:23 2025)
lrm proxmoxW540 (idle, Mon Oct  6 10:36:27 2025)
lrm pvebig (idle, Mon Oct  6 10:36:30 2025)
lrm pveduo (idle, Mon Oct  6 10:36:28 2025)
lrm pvegiant (idle, Mon Oct  6 10:36:31 2025)
lrm pvenew (idle, Mon Oct  6 10:36:31 2025)
lrm pveslow (idle, Mon Oct  6 10:36:29 2025)
service ct:100 (pvenew, ignored)
service ct:102 (---, ignored)
service ct:106 (---, ignored)
service ct:107 (pveduo, ignored)
service ct:112 (pveslow, ignored)
service ct:114 (pvebig, ignored)
service ct:116 (---, ignored)
service ct:117 (---, ignored)
service ct:200 (pvenew, ignored)
service vm:120 (---, ignored)

For both failed boots I can see the following concerning logs:
Code: 
2025-10-05T12:28:44.807917+13:00 proxmoxW540 pmxcfs[1298]: [quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
2025-10-05T12:28:44.810089+13:00 proxmoxW540 pmxcfs[1298]: [quorum] crit: can't initialize service
2025-10-05T12:28:44.810376+13:00 proxmoxW540 pmxcfs[1298]: [confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
2025-10-05T12:28:44.810582+13:00 proxmoxW540 pmxcfs[1298]: [confdb] crit: can't initialize service
2025-10-05T12:28:44.810838+13:00 proxmoxW540 pmxcfs[1298]: [dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
2025-10-05T12:28:44.810947+13:00 proxmoxW540 pmxcfs[1298]: [dcdb] crit: can't initialize service
2025-10-05T12:28:44.811034+13:00 proxmoxW540 pmxcfs[1298]: [status] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
2025-10-05T12:28:44.811101+13:00 proxmoxW540 pmxcfs[1298]: [status] crit: can't initialize service


And it looks like pve-ha-crm gets stuck in status change startup => wait_for_quorum.

But I do not understand why, as I can see corosync starting up and acquiring quorum and looking happy.
 

Attachments

Last edited:
Hello,

I did a bit more testing, and I found that the easiest way to start one of those two nodes is to follow these steps:
1. boot in recovery mode
2. systemctl start pve-cluster
3. CTRL+D to continue the boot process


So it looks like a race condition on node boot where the cluster service or corosync can take a little bit longer to start and it locks the processes that are supposed to start immediately after.

Also to note, that the nodes that have this issue are both a bit on the slower side (one running in a VM inside VirtualBox and another one a NUC running on a Intel(R) Celeron(R) CPU N3050.
 
Liviu Sas said:
lrm proxmoxW540 (idle, Mon Oct 6 10:36:27 2025)
lrm pvebig (idle, Mon Oct 6 10:36:30 2025)
lrm pveduo (idle, Mon Oct 6 10:36:28 2025)
lrm pvegiant (idle, Mon Oct 6 10:36:31 2025)
lrm pvenew (idle, Mon Oct 6 10:36:31 2025)
lrm pveslow (idle, Mon Oct 6 10:36:29 2025)
Click to expand...
Are these all nodes in the cluster? What does pvecm status output? Keep in mind that it is not recommended to have a even amount of nodes in a cluster, see e.g. [0]. This seems like a clustering issue first of all.

Liviu Sas said:
service ct:100 (pvenew, ignored)
service ct:102 (---, ignored)
service ct:106 (---, ignored)
service ct:107 (pveduo, ignored)
service ct:112 (pveslow, ignored)
service ct:114 (pvebig, ignored)
service ct:116 (---, ignored)
service ct:117 (---, ignored)
service ct:200 (pvenew, ignored)
service vm:120 (---, ignored)
Click to expand...
Are the HA resources with node "---" running somewhere unnoticed? Or were these removed already?

[0] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_supported_setups
 
From the provided log in log-pve540-1.zip it also seems like there's quite a few connection issues, e.g. to the InfluxDB right at the start. I assume that proxmoxW540 is one of the nodes that has the problems starting up? Is the network stable for that/these nodes?
 
Hello,

The network is stable. No issues with the cluster or the nodes during the normal usage.
I only see issues at startup and only for these 2 nodes.

As for the HA resources, I removed them all just for the sting, and I can still reproduce the issue.

root@proxmoxW540:~# ha-manager status

quorum OK
master pveslow (idle, Fri Nov 14 11:05:15 2025)
lrm proxmoxW540 (idle, Fri Nov 14 22:25:42 2025)
lrm pvebig (idle, Fri Nov 14 22:25:46 2025)
lrm pveduo (idle, Fri Nov 14 22:25:44 2025)
lrm pvegiant (idle, Fri Nov 14 22:25:46 2025)
lrm pvenew (idle, Fri Nov 14 22:25:43 2025)
lrm pveslow (idle, Fri Nov 14 22:25:42 2025)
 
I did a bit more debugging, and it seems like pmxcfs daemon fails to write to /var/lib/pve-cluster/config.db while the system is fully boot up, but it has no issues while in recovery mode.

If I delete that file and start pmxcfs/corosync, the file is recreated with a size of 4096.
root@proxmoxW540:~# ls -al /var/lib/pve-cluster/config.db*
-rw------- 1 root root 4096 Nov 19 14:12 /var/lib/pve-cluster/config.db
-rw------- 1 root root 32768 Nov 19 14:15 /var/lib/pve-cluster/config.db-shm
-rw------- 1 root root 152472 Nov 19 14:15 /var/lib/pve-cluster/config.db-wal



If I start pmxcfs/corosync in recovery mode, the file is created with a size of 135k
ls -al /var/lib/pve-cluster/
total 367
drwx------ 2 root root 6 Nov 19 14:57 .
drwxr-xr-x 48 root root 49 Oct 2 12:23 ..
-rw------- 1 root root 135168 Nov 19 15:08 config.db
-rw------- 1 root root 32768 Nov 19 15:08 config.db-shm
-rw------- 1 root root 4124152 Nov 19 15:09 config.db-wal
-rw------- 1 root root 0 Nov 19 14:15 .pmxcfs.lockfile


Hope this helps.

This is the output of
/usr/bin/pmxcfs -f
Code: 
/usr/bin/pmxcfs -f
[main] notice: resolved node name 'proxmoxW540' to '192.168.86.78' for default node IP address
[quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[quorum] crit: can't initialize service
[confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[confdb] crit: can't initialize service
[dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[dcdb] crit: can't initialize service
[status] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[status] crit: can't initialize service
[libqb] info: server name: pve2
[quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[status] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
[status] notice: update cluster info (cluster name  home-zfs, version = 18)
[dcdb] notice: members: 2/2114
[dcdb] notice: all data is up to date
[status] notice: members: 2/2114
[status] notice: all data is up to date
[dcdb] notice: members: 2/2114, 3/885, 4/23748, 6/1324770, 7/1425143, 8/3664738
[dcdb] notice: starting data syncronisation
[dcdb] notice: cpg_send_message retried 1 times
[status] notice: node has quorum
[status] notice: members: 2/2114, 3/885, 4/23748, 6/1324770, 7/1425143, 8/3664738
[status] notice: starting data syncronisation
[dcdb] notice: received sync request (epoch 2/2114/00000002)
[status] notice: received sync request (epoch 2/2114/00000002)
[dcdb] notice: received all states
[dcdb] notice: leader is 3/885
[dcdb] notice: synced members: 3/885, 4/23748, 6/1324770, 7/1425143, 8/3664738
[dcdb] notice: waiting for updates from leader
[status] notice: received all states
[status] notice: all data is up to date
[dcdb] notice: update complete - trying to commit (got 210 inode updates)
[dcdb] notice: all data is up to date
[main] notice: teardown filesystem


And with debugging added:
Code: 
[libqb] debug: qb_ipcc_disconnect() (ipcc.c:483:qb_ipcc_disconnect)
[quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync) (quorum.c:111:service_quorum_initialize)
[libqb] debug: qb_ipcc_disconnect() (ipcc.c:483:qb_ipcc_disconnect)
[confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync) (confdb.c:225:service_cmap_initialize)
[main] debug: dfsm_set_mode - set mode to 0 (dfsm.c:446:dfsm_set_mode)
[libqb] debug: qb_ipcc_disconnect() (ipcc.c:483:qb_ipcc_disconnect)
[dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync) (dfsm.c:1355:dfsm_initialize)
[main] debug: dfsm_set_mode - set mode to 0 (dfsm.c:446:dfsm_set_mode)
[libqb] debug: qb_ipcc_disconnect() (ipcc.c:483:qb_ipcc_disconnect)
[status] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync) (dfsm.c:1355:dfsm_initialize)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[libqb] debug: shm size:1048589; real_size:1052672; rb->word_size:263168 (ringbuffer.c:236:qb_rb_open_2)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_getattr / (pmxcfs.c:118:cfs_fuse_getattr)
[main] debug: find_plug start  (pmxcfs.c:97:find_plug)
[main] debug: cfs_plug_base_lookup_plug  (cfs-plug.c:51:cfs_plug_base_lookup_plug)
[main] debug: find_plug end  = 0x77f4ff710340 () (pmxcfs.c:104:find_plug)
[main] debug: enter cfs_plug_base_getattr  (cfs-plug.c:86:cfs_plug_base_getattr)
[main] debug: leave cfs_plug_base_getattr  (cfs-plug.c:106:cfs_plug_base_getattr)
[main] debug: leave cfs_fuse_getattr / (0) (pmxcfs.c:141:cfs_fuse_getattr)
[main] debug: enter cfs_fuse_getattr /priv (pmxcfs.c:118:cfs_fuse_getattr)
[main] debug: find_plug start priv (pmxcfs.c:97:find_plug)
[main] debug: cfs_plug_base_lookup_plug priv (cfs-plug.c:51:cfs_plug_base_lookup_plug)
[main] debug: cfs_plug_base_lookup_plug name = priv new path = (null) (cfs-plug.c:59:cfs_plug_base_lookup_plug)
[main] debug: find_plug end priv = 0x77f4ff710340 (priv) (pmxcfs.c:104:find_plug)
[main] debug: enter cfs_plug_base_getattr priv (cfs-plug.c:86:cfs_plug_base_getattr)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)



[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)
[main] debug: enter cfs_fuse_statfs / (pmxcfs.c:510:cfs_fuse_statfs)
[main] debug: enter cfs_plug_base_statfs  (cfs-plug.c:420:cfs_plug_base_statfs)


Code: 
root@proxmoxW540:~#  pidof pmxcfs
1817

root@proxmoxW540:~# strace -fp 1817
strace: Process 1817 attached with 8 threads
[pid  1857] read(8,  <unfinished ...>
[pid  1832] futex(0x62b5cd459340, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  1828] read(8,  <unfinished ...>
[pid  1823] futex(0x62b5cd459340, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  1822] futex(0x62b5cd459340, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  1818] futex(0x62b5cd459340, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  1817] futex(0x7ffc9fdd4e50, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1821] recvfrom(15,


 <unfinished ...>
[pid  1857] <... read resumed>"(\0\0\0\21\0\0\0D\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0q\0\0\0y\0\0\0"..., 135168) = 40
[pid  1857] getpid()                    = 1817
[pid  1857] sendto(3, "<31>Nov 19 16:26:38 pmxcfs[1817]"..., 102, MSG_NOSIGNAL, NULL, 0) = 102
[pid  1857] write(2, "[main] debug: enter cfs_fuse_sta"..., 69) = 69
[pid  1857] getpid()                    = 1817
[pid  1857] sendto(3, "<31>Nov 19 16:26:38 pmxcfs[1817]"..., 113, MSG_NOSIGNAL, NULL, 0) = 113
[pid  1857] write(2, "[main] debug: enter cfs_plug_bas"..., 80) = 80
[pid  1857] writev(8, [{iov_base="`\0\0\0\0\0\0\0D\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\0\200\0\0\0\0\0\0\351\177\0\0\0\0\0\0\351\177\0\0\0\0\0\0\0\0\4\0\0\0\0\0"..., iov_len=80}], 2) = 96
[pid  1857] read(8,  <unfinished ...>
[pid  1828] <... read resumed>"(\0\0\0\21\0\0\0F\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0q\0\0\0y\0\0\0"..., 135168) = 40
[pid  1828] getpid()                    = 1817
[pid  1828] sendto(3, "<31>Nov 19 16:26:53 pmxcfs[1817]"..., 102, MSG_NOSIGNAL, NULL, 0) = 102
[pid  1828] write(2, "[main] debug: enter cfs_fuse_sta"..., 69) = 69
[pid  1828] getpid()                    = 1817
[pid  1828] sendto(3, "<31>Nov 19 16:26:53 pmxcfs[1817]"..., 113, MSG_NOSIGNAL, NULL, 0) = 113
[pid  1828] write(2, "[main] debug: enter cfs_plug_bas"..., 80) = 80
[pid  1828] writev(8, [{iov_base="`\0\0\0\0\0\0\0F\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\0\200\0\0\0\0\0\0\351\177\0\0\0\0\0\0\351\177\0\0\0\0\0\0\0\0\4\0\0\0\0\0"..., iov_len=80}], 2) = 96
[pid  1828] read(8, strace: Process 1823 detached
strace: Process 1857 detached
strace: Process 1832 detached
strace: Process 1828 detached
 <detached ...>
strace: Process 1822 detached
strace: Process 1821 detached
strace: Process 1818 detached
strace: Process 1817 detached
 
Last edited:
You must log in or register to reply here.
Top Bottom