All containers missing after creating cluster

[MikeC]

Hello, all.

I have two proxmox4.x servers and I decided to group them into a cluster. After running the command 'pvecm create datacenter2', which completed successfully, I no longer see any of the containers on the server! I only see the server itself, with no nodes under it except local storage. There are 13 nodes associated with this server. The configs are in /etc/pve/nodes/proxmox4/lxc/, and I can still ssh to the containers, so I figure it's just an issue with the web portal?

Running 'pvecm status' shows the following:

root@proxmox4:/etc/pve/nodes/proxmox4# pvecm status
Quorum information
------------------
Date: Tue Jul 12 15:41:47 2016
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 4
Quorate: Yes

Votequorum information
----------------------
Expected votes: 1
Highest expected: 1
Total votes: 1
Quorum: 1
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.10.109.24 (local)

How do I reset the web view so that I can see and manage the 13 VMs?
Can I undo the pvecm create in order to restore how the server looked before?

The version of this server is 4.1-15/8cd55b52

#mikec

 

[MikeC]

Additional info...

root@proxmox4:/etc/pve/lxc# service pve-cluster status
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Tue 2016-07-12 15:33:33 CDT; 29min ago
Process: 15879 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 15875 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 15877 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
└─15877 /usr/bin/pmxcfs

Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [ipcs] crit: connection from bad user 33! - rejected
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [libqb] error: Error in connection setup (15877-16872-26): Unknown err... (-1)
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [ipcs] crit: connection from bad user 33! - rejected
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [libqb] error: Error in connection setup (15877-16872-26): Unknown err... (-1)
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [ipcs] crit: connection from bad user 33! - rejected
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [libqb] error: Error in connection setup (15877-16872-26): Unknown err... (-1)
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [ipcs] crit: connection from bad user 33! - rejected
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [libqb] error: Error in connection setup (15877-16872-26): Unknown err... (-1)
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [ipcs] crit: connection from bad user 33! - rejected
Jul 12 15:55:18 proxmox4 pmxcfs[15877]: [libqb] error: Error in connection setup (15877-16872-26): Unknown err... (-1)
Hint: Some lines were ellipsized, use -l to show in full.

 

[MikeC]

In case it was pveproxy, I did a 'service pveproxy restart'. Now the web UI won't load. Status provides the following:

root@proxmox4:/etc/pve/local# service pveproxy status
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled)
Active: active (running) since Tue 2016-07-12 15:55:20 CDT; 12min ago
Process: 26531 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
Process: 26598 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 26602 (pveproxy)
CGroup: /system.slice/pveproxy.service
├─ 379 pveproxy worker
├─ 380 pveproxy worker
├─ 381 pveproxy worker
└─26602 pveproxy

Jul 12 16:07:43 proxmox4 pveproxy[379]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file...1639.
Jul 12 16:07:43 proxmox4 pveproxy[26602]: worker 32758 finished
Jul 12 16:07:43 proxmox4 pveproxy[26602]: starting 1 worker(s)
Jul 12 16:07:43 proxmox4 pveproxy[26602]: worker 380 started
Jul 12 16:07:43 proxmox4 pveproxy[380]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file...1639.
Jul 12 16:07:43 proxmox4 pveproxy[319]: worker exit
Jul 12 16:07:43 proxmox4 pveproxy[26602]: worker 319 finished
Jul 12 16:07:43 proxmox4 pveproxy[26602]: starting 1 worker(s)
Jul 12 16:07:43 proxmox4 pveproxy[26602]: worker 381 started
Jul 12 16:07:43 proxmox4 pveproxy[381]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file...1639.
Hint: Some lines were ellipsized, use -l to show in full.


I haven't changed anything outside of issuing the initial "pvecm create cluster"

 

[MikeC]

okay, looking around online I found a number of similar issues.
What worked ultimately was a mixture of re-doing the certs (pvecm updatecerts), then doing a full stop on the cluster using 'service pve-cluster stop', then restarting it again.

No clue what happened, but it was a bit disconcerting.

Issue resolved.