Proxmox Cluster One Node greyed out

ubu

Renowned Member
Nov 24, 2017
425
84
68
49
In webinterface one node and containers are greyed out Question marks.

pct list and lxc-ls do not return

containers are running, ssh login on them is possible.

restarting pvedaemon, pvestatd, pve-cluster does not fix the situation

pvecm status shows all hosts
 
Last edited:
anything in the logs from pve-cluster/pmxcfs?
 
anything in the logs from pve-cluster/pmxcfs?
Have restarted pve-cluster, pvestatd etc multiple times, log for pmxcfs looks good, i thinks

Mai 30 12:19:25 pvehe12 pmxcfs[1209739]: [main] notice: teardown filesystem
Mai 30 12:19:26 pvehe12 pmxcfs[1209739]: [main] notice: exit proxmox configuration filesystem (0)
Mai 30 12:19:26 pvehe12 pmxcfs[1236619]: [status] notice: update cluster info (cluster name pvehe, version = 17)
Mai 30 12:19:26 pvehe12 pmxcfs[1236619]: [status] notice: node has quorum
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000020)
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000020)
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 3/1576, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date
Mai 30 12:19:27 pvehe12 pmxcfs[1236619]: [status] notice: dfsm_deliver_queue: queue length 122
Mai 30 12:19:55 pvehe12 pmxcfs[1236619]: [status] notice: received log
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000021)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000021)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: starting data syncronisation
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received sync request (epoch 1/1597/00000022)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received sync request (epoch 1/1597/00000022)
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: leader is 1/1597
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: synced members: 1/1597, 2/1663, 3/3147473, 4/2412, 5/1960, 6/1985, 7/1868, 8/2041, 9/1905, 10/2189, 11/2046, 12/1236619, 13/2171
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [dcdb] notice: all data is up to date
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: received all states
Mai 30 12:20:45 pvehe12 pmxcfs[1236619]: [status] notice: all data is up to date




I have rebootet the host, now everything is ok, but the problem has happened several times, only fix for now is reboot, but this is not a good solution.
 
Please post /etc/pve/corosync.conf of one node and
/etc/hosts /etc/network/interfaces of all nodes that are in that cluster.
Please also post your /etc/chrony/chrony.conf or make sure, you have correct time on each node.

Webinterface usually doesnt work anymore (or at least it did, when there was no quorum). I also had problems with hosts showing up correctly in webview, when time was offsync.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!