Proxmox Cluster One Node greyed out

[ubu]

In webinterface one node and containers are greyed out Question marks.

pct list and lxc-ls do not return

containers are running, ssh login on them is possible.

restarting pvedaemon, pvestatd, pve-cluster does not fix the situation

pvecm status shows all hosts

 

[aaron]

anything in the logs from pve-cluster/pmxcfs?

 

[ubu]

anything in the logs from pve-cluster/pmxcfs?

Have restarted pve-cluster, pvestatd etc multiple times, log for pmxcfs looks good, i thinks

Mai 30 12:19:25 pvehe12 pmxcfs[1209739]: [main] notice: teardown filesystem
Mai 30 12:19:26 pvehe12 pmxcfs[1209739]: [main] notice: exit proxmox configuration filesystem (0)
Mai 30 12:19:26 pvehe12 pmxcfs[1236619]: [status] notice: update cluster info (cluster name pvehe, version = 17)
Mai 30 12:19:26 pvehe12 pmxcfs[1236619]: [status] notice: node has quorum
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000020)
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000020)
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: dfsm_deliver_queue: queue length 122
Mai 30 12:19:55 pvehe12 pmxcfs[1236619]: [status] notice: received log
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000021)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000021)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000022)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000022)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date




I have rebootet the host, now everything is ok, but the problem has happened several times, only fix for now is reboot, but this is not a good solution.

 

[jsterr]

Please post /etc/pve/corosync.conf of one node and
/etc/hosts /etc/network/interfaces of all nodes that are in that cluster.
Please also post your /etc/chrony/chrony.conf or make sure, you have correct time on each node.

Webinterface usually doesnt work anymore (or at least it did, when there was no quorum). I also had problems with hosts showing up correctly in webview, when time was offsync.