Hi,
Latest update seems to broke our cluster setup. On all nodes cman stops working. Manually tried to start it on all nodes:
root@kvm45:~# /etc/init.d/cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Unfencing self... [ OK ]
But they stop in a few seconds as you see.
root@kvm45:~# pveversion -v
pve-manager: 2.1-12 (pve-manager/2.1/be112d89)
running kernel: 2.6.32-13-pve
proxmox-ve-2.6.32: 2.1-72
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-13-pve: 2.6.32-72
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-45
pve-firmware: 1.0-17
libpve-common-perl: 1.0-28
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-27
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-6
ksm-control-daemon: 1.1-1
Edit: I can open webadmins of the nodes one by one and start the VMs via each nodes webadmins. But on each node webadmin, the other nodes seem offline.
Edit2: I get these on nodes:
Jul 26 13:17:19 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:19 corosync [CMAN ] Error reloading the configuration, will retry every second
Jul 26 13:17:20 corosync [CMAN ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration
Jul 26 13:17:20 corosync [CMAN ] Can't get updated config version 6: New configuration version has to be newer than current running configuration
.
Jul 26 13:17:20 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:20 corosync [CMAN ] Error reloading the configuration, will retry every second
Jul 26 13:17:21 corosync [CMAN ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration
Jul 26 13:17:21 corosync [CMAN ] Can't get updated config version 6: New configuration version has to be newer than current running configuration
.
Jul 26 13:17:21 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:21 corosync [CMAN ] Error reloading the configuration, will retry every second
How can I fix this?
Latest update seems to broke our cluster setup. On all nodes cman stops working. Manually tried to start it on all nodes:
root@kvm45:~# /etc/init.d/cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Unfencing self... [ OK ]
But they stop in a few seconds as you see.
root@kvm45:~# pveversion -v
pve-manager: 2.1-12 (pve-manager/2.1/be112d89)
running kernel: 2.6.32-13-pve
proxmox-ve-2.6.32: 2.1-72
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-13-pve: 2.6.32-72
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-45
pve-firmware: 1.0-17
libpve-common-perl: 1.0-28
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-27
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-6
ksm-control-daemon: 1.1-1
Edit: I can open webadmins of the nodes one by one and start the VMs via each nodes webadmins. But on each node webadmin, the other nodes seem offline.
Edit2: I get these on nodes:
Jul 26 13:17:19 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:19 corosync [CMAN ] Error reloading the configuration, will retry every second
Jul 26 13:17:20 corosync [CMAN ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration
Jul 26 13:17:20 corosync [CMAN ] Can't get updated config version 6: New configuration version has to be newer than current running configuration
.
Jul 26 13:17:20 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:20 corosync [CMAN ] Error reloading the configuration, will retry every second
Jul 26 13:17:21 corosync [CMAN ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration
Jul 26 13:17:21 corosync [CMAN ] Can't get updated config version 6: New configuration version has to be newer than current running configuration
.
Jul 26 13:17:21 corosync [CMAN ] Activity suspended on this node
Jul 26 13:17:21 corosync [CMAN ] Error reloading the configuration, will retry every second
How can I fix this?