Proxmox Cluster dead after addition of node

[iprigger]

Hi All,

I have a four-node cluster that I wanted to add a 5th node...

What I did:
- Setup the 5th node
- Add the subscription key
- Did all the updates
- Reboot

- pvecm add 10.81.0.10 (the IP of the first Proxmox cluster node)...
and... boing:

unable to copy ssh ID: cat: write error: Permission denied

After that: ALL NODES ARE NOT SEING THE OTHER CLUSTER NODES. The cluster
seems to be shot completely....

ANY idea?

First Cluster Node:
root@pve01-8ca:~# pveversion -v
proxmox-ve: 4.4-76 (running kernel: 4.4.35-1-pve)
pve-manager: 4.4-1 (running version: 4.4-1/eb2d6f1e)
pve-kernel-4.4.35-1-pve: 4.4.35-76
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-101
pve-firmware: 1.1-10
libpve-common-perl: 4.0-83
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-70
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-1
pve-qemu-kvm: 2.7.0-9
pve-container: 1.0-88
pve-firewall: 2.0-33
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-2
lxcfs: 2.0.5-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80

New Cluster Node:
pve-manager/4.4-13/7ea56165 (running kernel: 4.4.59-1-pve)
root@pve04-12cxs:~# pveversion -v
proxmox-ve: 4.4-87 (running kernel: 4.4.59-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.59-1-pve: 4.4.59-87
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-49
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-94
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-99
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80

checktime: 1495441303
key: PROXMOX-KEY
level: c
nextduedate: 2017-09-06
productname: Proxmox VE Community Subscription 1 CPU/year
regdate: 2015-09-06
serverid: 87DC51CA7E51B65C5305792686946E50
sockets: 1
status: Active
validdirectory: 87DC51CA7E51B65C5305792686946E50

... this is HIGHLY annoying... I can't reboot the system just at will
now (already running production stuff).

Thanks!

Tobias

 

[tom]

your first node is running a quite old packages, and maybe you have network problems (multicast)?

make always sure that all nodes have the same software level and also test your cluster network before adding nodes (multicast).

 

[iprigger]

Hi,
Just updated the systems (all).

One node keeps playing up:
May 22 13:00:32 pve05-24cxf pve-ha-lrm[2323]: unable to write lrm status file - closing file '/etc/pve/nodes/pve05-24cxf/lrm_status.tmp.2323' failed - Operation not permitted
May 22 13:00:32 pve05-24cxf pmxcfs[2256]: [dcdb] notice: start cluster connection
May 22 13:00:32 pve05-24cxf pmxcfs[2256]: [dcdb] notice: members: 4/2256
May 22 13:00:32 pve05-24cxf pmxcfs[2256]: [dcdb] notice: all data is up to date

after a restart of corosync, all is well.

and with one node (although added now), live-migration fails...

Kind Regards
Tobias

 

[iprigger]

btw... network problems are always possible, yes. but the node playing up (i.e. going out of corosync) has been added some time ago - no issues so far...

tobias

 

[iprigger]

just caught the log-event:

[.... much more.... ]
May 22 13:09:06 pve05-24cxf corosync[9403]: [TOTEM ] Retransmit List: b25 b14 b15 b16 b17 b18 b19 b1a b1b b1c b1d b1e b1f b20 b22 b23 b24 b26 b21 b27 b28 b29 b2a b2b b2c b2d b2e b2f b30 b31
May 22 13:09:06 pve05-24cxf corosync[9403]: [TOTEM ] Retransmit List: b26 b14 b15 b16 b17 b18 b19 b1a b1b b1c b1d b1e b1f b20 b22 b23 b24 b25 b21 b27 b28 b29 b2a b2b b2c b2d b2e b2f b30 b31
May 22 13:09:06 pve05-24cxf corosync[9403]: [TOTEM ] Retransmit List: b25 b14 b15 b16 b17 b18 b19 b1a b1b b1c b1d b1e b1f b20 b22 b23 b24 b26 b21 b27 b28 b29 b2a b2b b2c b2d b2e b2f b30 b31
May 22 13:09:07 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7976) was formed. Members left: 1
May 22 13:09:07 pve05-24cxf corosync[9403]: [TOTEM ] Failed to receive the leave message. failed: 1
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: members: 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: starting data syncronisation
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: members: 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: starting data syncronisation
May 22 13:09:07 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:07 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:07 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7980) was formed. Members
May 22 13:09:07 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:07 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: received sync request (epoch 2/1381/0000001C)
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: received sync request (epoch 2/1381/0000001C)
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: received all states
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: leader is 2/1381
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: synced members: 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: all data is up to date
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [dcdb] notice: dfsm_deliver_queue: queue length 2
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: received all states
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: all data is up to date
May 22 13:09:07 pve05-24cxf pmxcfs[2256]: [status] notice: dfsm_deliver_queue: queue length 17
May 22 13:09:09 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7984) was formed. Members
May 22 13:09:09 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:09 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:10 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7988) was formed. Members
May 22 13:09:10 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:10 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:12 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7992) was formed. Members
May 22 13:09:12 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:12 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:13 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:7996) was formed. Members
May 22 13:09:13 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:13 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:15 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8000) was formed. Members
May 22 13:09:15 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:15 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:16 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8004) was formed. Members
May 22 13:09:16 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:16 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:17 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8008) was formed. Members
May 22 13:09:17 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:17 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:19 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8012) was formed. Members
May 22 13:09:19 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:19 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:20 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8016) was formed. Members
May 22 13:09:20 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:20 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:22 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8020) was formed. Members
May 22 13:09:22 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:22 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:23 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8024) was formed. Members
May 22 13:09:23 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:23 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:24 pve05-24cxf pveproxy[9517]: Clearing outdated entries from certificate cache
May 22 13:09:25 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8028) was formed. Members
May 22 13:09:25 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:25 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:25 pve05-24cxf pmxcfs[2256]: [status] notice: received log
May 22 13:09:26 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8032) was formed. Members
May 22 13:09:26 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:26 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:28 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8036) was formed. Members
May 22 13:09:28 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:28 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:29 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8040) was formed. Members
May 22 13:09:29 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:29 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:30 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8044) was formed. Members
May 22 13:09:30 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:30 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:32 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8048) was formed. Members
May 22 13:09:32 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:32 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:33 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8052) was formed. Members
May 22 13:09:33 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:33 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:35 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8056) was formed. Members
May 22 13:09:36 pve05-24cxf corosync[9403]: [TOTEM ] A processor failed, forming new configuration.
May 22 13:09:36 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8060) was formed. Members
May 22 13:09:36 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:36 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:36 pve05-24cxf pmxcfs[2256]: [dcdb] notice: cpg_send_message retried 8 times
May 22 13:09:36 pve05-24cxf pmxcfs[2256]: [status] notice: cpg_send_message retried 2 times
May 22 13:09:38 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8064) was formed. Members
May 22 13:09:38 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:38 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:39 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8068) was formed. Members
May 22 13:09:39 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:39 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:41 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8072) was formed. Members
May 22 13:09:41 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:41 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:42 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8076) was formed. Members
May 22 13:09:42 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:42 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:43 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8080) was formed. Members
May 22 13:09:43 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:43 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:45 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8084) was formed. Members
May 22 13:09:45 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:45 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:46 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8088) was formed. Members
May 22 13:09:46 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:46 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:48 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.11:8092) was formed. Members
May 22 13:09:48 pve05-24cxf corosync[9403]: [QUORUM] Members[4]: 2 3 5 4
May 22 13:09:48 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:48 pve05-24cxf corosync[9403]: [TOTEM ] A new membership (10.81.0.10:8096) was formed. Members joined: 1
May 22 13:09:48 pve05-24cxf corosync[9403]: [QUORUM] Members[5]: 1 2 3 5 4
May 22 13:09:48 pve05-24cxf corosync[9403]: [MAIN ] Completed service synchronization, ready to provide service.
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: members: 1/1371, 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: starting data syncronisation
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [status] notice: members: 1/1371, 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [status] notice: starting data syncronisation
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: received sync request (epoch 1/1371/00000034)
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [status] notice: received sync request (epoch 1/1371/00000030)
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: received all states
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: leader is 2/1381
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: synced members: 2/1381, 3/1371, 4/2256, 5/2330
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [dcdb] notice: all data is up to date
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [status] notice: received all states
May 22 13:09:54 pve05-24cxf pmxcfs[2256]: [status] notice: all data is up to date


no specific action was startet at that moment... just playing up...

Tobias