status unknown

[linhu]

Hi i have a testsetup with 2 servers and 2 testvm's.
What meeens status unknown and how can i resolv this?

The scenario was a server goes offline, after i have removed this server from the cluster all was fine.
But the next was the server comes online again, added to the cluster and now this:

prox1 and 2 are in a cluster

root@prox1:~# pvecm status
Cluster information
-------------------
Name: linuxhus1
Config Version: 2
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Thu Aug 24 16:11:31 2023
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000002
Ring ID: 1.23
Quorate: Yes

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.151
0x00000002 1 192.168.1.150 (local)
root@prox1:~#

 

[Maximiliano]

Hello, to run a cluster with two nodes (or any even number of nodes) you need to setup a QDevice, otherwise the cluster will be out of quorum as soon as a server is offline.

[1] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[linhu]

ok have setup a qdevice, but its still unknown.
the nodes are online but the storage is unknown

 

[linhu]

after 5min all is unknown again

 

[linhu]

Aug 24 16:54:06 prox1 pve-guests[1101]: mount error: received interrupt
Aug 24 16:52:41 prox1 pmxcfs[825]: [status] crit: can't initialize service
Aug 24 16:52:41 prox1 pmxcfs[825]: [status] crit: cpg_initialize failed: 2
Aug 24 16:52:41 prox1 pmxcfs[825]: [dcdb] crit: can't initialize service
Aug 24 16:52:41 prox1 pmxcfs[825]: [dcdb] crit: cpg_initialize failed: 2
Aug 24 16:52:41 prox1 pmxcfs[825]: [confdb] crit: can't initialize service
Aug 24 16:52:41 prox1 pmxcfs[825]: [confdb] crit: cmap_initialize failed: 2
Aug 24 16:52:41 prox1 pmxcfs[825]: [quorum] crit: can't initialize service
Aug 24 16:52:41 prox1 pmxcfs[825]: [quorum] crit: quorum_initialize failed: 2
-- Boot cac268d3825f413198e6db1dfdbbf28d --
Aug 24 16:51:26 prox1 kernel: watchdog: watchdog0: watchdog did not stop!
Aug 24 16:51:25 prox1 pmxcfs[2635]: [confdb] crit: cmap_finalize failed: 9
Aug 24 16:51:25 prox1 pmxcfs[2635]: [confdb] crit: cmap_track_delete version failed: 9
Aug 24 16:51:25 prox1 pmxcfs[2635]: [confdb] crit: cmap_track_delete nodelist failed: 9
Aug 24 16:51:25 prox1 pmxcfs[2635]: [quorum] crit: quorum_finalize failed: 9
Aug 24 16:51:24 prox1 pmxcfs[2635]: [status] crit: can't initialize service
Aug 24 16:51:24 prox1 pmxcfs[2635]: [status] crit: cpg_initialize failed: 2
Aug 24 16:51:24 prox1 pmxcfs[2635]: [dcdb] crit: can't initialize service
Aug 24 16:51:24 prox1 pmxcfs[2635]: [dcdb] crit: cpg_initialize failed: 2

 

[Maximiliano]

Did you try restarting the corosync and pve-cluster services? If the storage does not come back up in 2 minutes you could restart the pvestatd service and see if the situation improves.

 

[linhu]

the nodes are back but no storrage
Aug 24 17:40:08 prox1 pmxcfs[13496]: [status] crit: can't initialize service
Aug 24 17:40:08 prox1 pmxcfs[13496]: [status] crit: cpg_initialize failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [dcdb] crit: can't initialize service
Aug 24 17:40:08 prox1 pmxcfs[13496]: [dcdb] crit: cpg_initialize failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [confdb] crit: can't initialize service
Aug 24 17:40:08 prox1 pmxcfs[13496]: [confdb] crit: cmap_initialize failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [quorum] crit: can't initialize service
Aug 24 17:40:08 prox1 pmxcfs[13496]: [quorum] crit: quorum_initialize failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [status] crit: cpg_leave failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [status] crit: cpg_dispatch failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [dcdb] crit: cpg_leave failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [dcdb] crit: cpg_dispatch failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [quorum] crit: quorum_dispatch failed: 2
Aug 24 17:40:08 prox1 pmxcfs[13496]: [confdb] crit: cmap_dispatch failed: 2
Aug 24 17:40:08 prox1 corosync-qdevice[13488]: Lost connection with heuristics worker
Aug 24 17:35:08 prox1 pmxcfs[825]: [confdb] crit: cmap_finalize failed: 9
Aug 24 17:35:08 prox1 pmxcfs[825]: [confdb] crit: cmap_track_delete version failed: 9
Aug 24 17:35:08 prox1 pmxcfs[825]: [confdb] crit: cmap_track_delete nodelist failed: 9
Aug 24 17:35:08 prox1 pmxcfs[825]: [quorum] crit: quorum_finalize failed: 9
Aug 24 17:35:06 prox1 pmxcfs[825]: [status] crit: can't initialize service
Aug 24 17:35:06 prox1 pmxcfs[825]: [status] crit: cpg_initialize failed: 2

Aug 24 17:41:10 prox2 pmxcfs[12156]: [status] crit: can't initialize service
Aug 24 17:41:10 prox2 pmxcfs[12156]: [status] crit: cpg_initialize failed: 2
Aug 24 17:41:10 prox2 pmxcfs[12156]: [dcdb] crit: can't initialize service
Aug 24 17:41:10 prox2 pmxcfs[12156]: [dcdb] crit: cpg_initialize failed: 2
Aug 24 17:41:10 prox2 pmxcfs[12156]: [confdb] crit: can't initialize service
Aug 24 17:41:10 prox2 pmxcfs[12156]: [confdb] crit: cmap_initialize failed: 2
Aug 24 17:41:10 prox2 pmxcfs[12156]: [quorum] crit: can't initialize service
Aug 24 17:41:10 prox2 pmxcfs[12156]: [quorum] crit: quorum_initialize failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [status] crit: cpg_leave failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [status] crit: cpg_dispatch failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [dcdb] crit: cpg_leave failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [dcdb] crit: cpg_dispatch failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [quorum] crit: quorum_dispatch failed: 2
Aug 24 17:41:09 prox2 pmxcfs[12156]: [confdb] crit: cmap_dispatch failed: 2
Aug 24 17:41:09 prox2 corosync-qdevice[12241]: Lost connection with heuristics worker
Aug 24 17:36:25 prox2 pmxcfs[12156]: [status] crit: can't initialize service
Aug 24 17:36:25 prox2 pmxcfs[12156]: [status] crit: cpg_initialize failed: 2
Aug 24 17:36:25 prox2 pmxcfs[12156]: [dcdb] crit: can't initialize service
Aug 24 17:36:25 prox2 pmxcfs[12156]: [dcdb] crit: cpg_initialize failed: 2
Aug 24 17:36:25 prox2 pmxcfs[12156]: [confdb] crit: can't initialize service
Aug 24 17:36:25 prox2 pmxcfs[12156]: [confdb] crit: cmap_initialize failed: 2
Aug 24 17:36:25 prox2 pmxcfs[12156]: [quorum] crit: can't initialize service
Aug 24 17:36:25 prox2 pmxcfs[12156]: [quorum] crit: quorum_initialize failed: 2