Hi,
I have three nodes. After adding an NFS datastore (FreeNAS), the status and metadata of the second node is wrong. The node is shown as offline and I got no metadata about the VM's. I only see the VM id's, but not their status (running/stopped) and other information (disk, network, ...).
I get the management interface from the second node, but I can't login. After waiting 30 seconds I got: "Login failed. Please try again.".
From the other nodes, login succeeds. Rarely, I also see that node 1 or 2 is shown as offline.
pveversion -v:
/var/log/messages on node 1:
/var/log/messages on node 2:
/var/log/messages on node 3:
The time configuration is OK.
I removed the NFS datastore, but no luck.
What can I do?
I have three nodes. After adding an NFS datastore (FreeNAS), the status and metadata of the second node is wrong. The node is shown as offline and I got no metadata about the VM's. I only see the VM id's, but not their status (running/stopped) and other information (disk, network, ...).
I get the management interface from the second node, but I can't login. After waiting 30 seconds I got: "Login failed. Please try again.".
From the other nodes, login succeeds. Rarely, I also see that node 1 or 2 is shown as offline.
pveversion -v:
Code:
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-12-pve
proxmox-ve-2.6.32: 2.1-68
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-16
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1/var/log/messages on node 1:
Code:
Aug 29 06:25:01 hause03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1859" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Aug 29 06:25:01 hause03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1859" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Code:
Aug 28 06:25:01 hause02 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1891" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Aug 28 08:16:18 hause02 kernel: device tap701i0 entered promiscuous mode
Aug 28 08:16:18 hause02 kernel: vmbr1: port 4(tap701i0) entering forwarding state
Aug 28 08:18:08 hause02 kernel: device tap702i0 entered promiscuous mode
Aug 28 08:18:08 hause02 kernel: vmbr1: port 5(tap702i0) entering forwarding state
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 08:41:27 hause02 corosync[2170]: [QUORUM] Members[2]: 1 3
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 08:41:27 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 08:41:27 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 08:41:27 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:3 left:1)
Aug 28 08:41:27 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 09:22:35 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 09:22:35 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 09:22:35 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 09:22:35 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 09:22:35 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:2 left:0)
Aug 28 09:22:35 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 11:32:42 hause02 corosync[2170]: [QUORUM] Members[2]: 1 3
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 11:32:42 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 11:32:42 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 11:32:42 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:3 left:1)
Aug 28 11:32:42 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 11:40:05 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 11:40:05 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 11:40:05 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 11:40:05 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 11:40:05 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:2 left:0)
Aug 28 11:40:05 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 12:35:38 hause02 corosync[2170]: [QUORUM] Members[2]: 1 3
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 12:35:38 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 12:35:38 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 12:35:38 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:3 left:1)
Aug 28 12:35:38 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 12:40:58 hause02 kernel: svc: failed to register lockdv1 RPC service (errno 97).
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 12:55:16 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 12:55:16 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 12:55:16 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 12:55:16 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 12:55:16 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.11) ; members(old:2 left:0)
Aug 28 12:55:16 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 12:55:49 hause02 kernel: ct0 nfs: server 10.20.64.111 not responding, still trying
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:03:53 hause02 corosync[2170]: [QUORUM] Members[2]: 2 3
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:03:53 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:03:53 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 13:03:53 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:3 left:1)
Aug 28 13:03:53 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:08:38 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:08:38 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 13:08:38 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 13:08:38 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 13:08:38 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:2 left:0)
Aug 28 13:08:38 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:49:33 hause02 corosync[2170]: [QUORUM] Members[2]: 2 3
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:49:33 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:49:33 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 13:49:33 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:3 left:1)
Aug 28 13:49:33 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 13:52:09 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 13:52:09 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 13:52:09 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 13:52:09 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 13:52:09 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:2 left:0)
Aug 28 13:52:09 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:15:40 hause02 corosync[2170]: [QUORUM] Members[2]: 2 3
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:15:40 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:15:40 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 14:15:40 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:3 left:1)
Aug 28 14:15:40 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:18:10 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:18:10 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 14:18:10 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 14:18:10 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 14:18:10 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:2 left:0)
Aug 28 14:18:10 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:29:58 hause02 corosync[2170]: [QUORUM] Members[2]: 2 3
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:29:58 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:29:58 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 14:29:58 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:3 left:1)
Aug 28 14:29:58 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] CLM CONFIGURATION CHANGE
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] New Configuration:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.12)
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.13)
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] Members Left:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] Members Joined:
Aug 28 14:32:03 hause02 corosync[2170]: [CLM ] #011r(0) ip(10.20.64.11)
Aug 28 14:32:03 hause02 corosync[2170]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 28 14:32:03 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 14:32:03 hause02 corosync[2170]: [QUORUM] Members[3]: 1 2 3
Aug 28 14:32:03 hause02 corosync[2170]: [CPG ] chosen downlist: sender r(0) ip(10.20.64.13) ; members(old:2 left:0)
Aug 28 14:32:03 hause02 corosync[2170]: [MAIN ] Completed service synchronization, ready to provide service.
Code:
Aug 29 06:25:01 hause03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1859" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Aug 29 06:25:01 hause03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1859" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.I removed the NFS datastore, but no luck.
What can I do?