loosing qorum during backup

F

felipe

Well-Known Member
Oct 28, 2013
222
6
58
we are loosing qorum one onde node during backup. another node which is the default management node shows no statistics for all node machines (and all other nodes red). i have to restart the pvemanager on that node to get statistics and green signs again.

1) question one is why is the node loosing qorum?
2) whay does the other node (which is not loosing the qorum) not showing any more statistics and marking all other nodes red?
 
this node is loosing the qorum:

pveversion --verbose
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-31-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-31-pve: 2.6.32-132
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
----------------------

Dec 11 03:17:01 node6 /USR/SBIN/CRON[215395]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 11 03:23:19 node6 pveproxy[737259]: worker 198309 finished
Dec 11 03:23:19 node6 pveproxy[737259]: starting 1 worker(s)
Dec 11 03:23:19 node6 pveproxy[737259]: worker 217794 started
Dec 11 03:31:34 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 03:45:31 node6 pvestatd[737374]: status update time (8.486 seconds)
Dec 11 03:46:34 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 03:47:29 node6 rrdcached[3247]: flushing old values
Dec 11 03:47:29 node6 rrdcached[3247]: rotating journals
Dec 11 03:47:29 node6 rrdcached[3247]: started new journal /var/lib/rrdcached/journal/rrd.journal.1418266049.035533
Dec 11 03:47:29 node6 rrdcached[3247]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1418258849.035429
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 03:48:23 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 03:48:23 node6 corosync[3418]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 03:48:23 node6 corosync[3418]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:0)
Dec 11 03:48:23 node6 corosync[3418]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 03:48:28 node6 pvestatd[737374]: status update time (6.173 seconds)
Dec 11 03:49:57 node6 pvestatd[737374]: status update time (5.502 seconds)
Dec 11 03:54:18 node6 vzdump[144761]: INFO: Finished Backup of VM 143 (00:48:07)
Dec 11 03:54:18 node6 vzdump[144761]: INFO: Starting Backup of VM 146 (qemu)
Dec 11 03:54:18 node6 qm[229879]: <root@pam> update VM 146: -lock backup
Dec 11 03:56:20 node6 pmxcfs[734255]: [dcdb] notice: data verification successful
Dec 11 04:01:35 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 04:12:49 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 04:12:49 node6 corosync[3418]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:12:49 node6 corosync[3418]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:0)
Dec 11 04:12:49 node6 corosync[3418]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:16:35 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 04:17:01 node6 /USR/SBIN/CRON[238658]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 11 04:24:00 node6 vzdump[144761]: INFO: Finished Backup of VM 146 (00:29:42)
Dec 11 04:24:00 node6 vzdump[144761]: INFO: Starting Backup of VM 148 (qemu)
Dec 11 04:24:01 node6 qm[241365]: <root@pam> update VM 148: -lock backup
Dec 11 04:24:13 node6 kernel: device tap148i0 entered promiscuous mode
Dec 11 04:24:13 node6 kernel: vmbr0: port 34(tap148i0) entering forwarding state
Dec 11 04:24:13 node6 kernel: device tap148i1 entered promiscuous mode
Dec 11 04:24:13 node6 kernel: vmbr101: port 1(tap148i1) entering forwarding state
Dec 11 04:24:23 node6 kernel: tap148i0: no IPv6 routers present
Dec 11 04:24:24 node6 kernel: tap148i1: no IPv6 routers present
Dec 11 04:25:42 node6 ntpd[3223]: Listen normally on 208 tap148i1 fe80::1418:a1ff:fec6:815 UDP 123
Dec 11 04:25:42 node6 ntpd[3223]: Listen normally on 209 tap148i0 fe80::f0fc:1ff:fed7:53e4 UDP 123
Dec 11 04:25:42 node6 ntpd[3223]: peers refreshed
Dec 11 04:26:48 node6 pvestatd[737374]: status update time (5.476 seconds)
Dec 11 04:31:36 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 04:34:27 node6 pvestatd[737374]: status update time (5.273 seconds)
Dec 11 04:36:21 node6 pvestatd[737374]: status update time (9.310 seconds)
Dec 11 04:36:30 node6 pvestatd[737374]: status update time (7.798 seconds)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:32 node6 pmxcfs[734255]: [status] notice: node lost quorum
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[10]: 2 3 4 5 6 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[9]: 2 4 5 6 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[8]: 2 4 5 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[7]: 2 4 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[6]: 2 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [CMAN ] quorum lost, blocking activity
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[5]: 7 8 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[4]: 7 9 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[3]: 7 10 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[2]: 7 11
Dec 11 04:36:32 node6 corosync[3418]: [QUORUM] Members[1]: 7
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:32 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:32 node6 corosync[3418]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:36:32 node6 pmxcfs[734255]: [status] notice: node has quorum
Dec 11 04:36:32 node6 kernel: dlm: closing connection to node 1
Dec 11 04:36:32 node6 kernel: dlm: closing connection to node 2
Dec 11 04:36:32 node6 kernel: dlm: closing connection to node 3
......
......
Dec 11 04:40:14 node6 corosync[3418]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:40:14 node6 corosync[3418]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:0)
Dec 11 04:40:14 node6 corosync[3418]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:40:14 node6 pmxcfs[734255]: [status] notice: cpg_send_message retried 1 times
Dec 11 04:42:46 node6 pve-firewall[737112]: firewall update time (5.477 seconds)
Dec 11 04:46:36 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 04:47:29 node6 rrdcached[3247]: flushing old values
Dec 11 04:47:29 node6 rrdcached[3247]: rotating journals
Dec 11 04:47:29 node6 rrdcached[3247]: started new journal /var/lib/rrdcached/journal/rrd.journal.1418269649.035461
Dec 11 04:47:29 node6 rrdcached[3247]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1418262449.035423
Dec 11 04:48:39 node6 pvestatd[737374]: status update time (5.414 seconds)
Dec 11 04:48:49 node6 pvestatd[737374]: status update time (5.221 seconds)
Dec 11 04:49:00 node6 pvestatd[737374]: status update time (5.668 seconds)
Dec 11 04:49:10 node6 pvestatd[737374]: status update time (5.408 seconds)
Dec 11 04:58:22 node6 kernel: vmbr0: port 34(tap148i0) entering disabled state
Dec 11 04:58:22 node6 kernel: vmbr101: port 1(tap148i1) entering disabled state
Dec 11 04:58:28 node6 vzdump[144761]: INFO: Finished Backup of VM 148 (00:34:28)
Dec 11 04:58:29 node6 vzdump[144761]: INFO: Starting Backup of VM 149 (qemu)
Dec 11 04:58:29 node6 qm[254593]: <root@pam> update VM 149: -lock backup
Dec 11 04:58:31 node6 kernel: device tap149i0 entered promiscuous mode
Dec 11 04:58:31 node6 kernel: HTB: quantum of class 10001 is big. Consider r2q change.
Dec 11 04:58:31 node6 kernel: vmbr0: port 34(tap149i0) entering forwarding state
Dec 11 04:58:32 node6 kernel: device tap149i1 entered promiscuous mode
Dec 11 04:58:32 node6 kernel: vmbr101: port 1(tap149i1) entering forwarding state
Dec 11 04:58:42 node6 kernel: tap149i1: no IPv6 routers present
Dec 11 04:58:42 node6 kernel: tap149i0: no IPv6 routers present
Dec 11 04:58:50 node6 pvestatd[737374]: status update time (6.068 seconds)
Dec 11 05:00:01 node6 /USR/SBIN/CRON[255248]: (root) CMD (vzdump 151 --quiet 1 --mailto backup@velartis.at --mode snapshot --node node2 --compress lzo --storage backups)
Dec 11 05:00:02 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 05:00:03 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 05:01:03 node6 ntpd[3223]: Listen normally on 210 tap149i0 fe80::4e4:d9ff:febb:1a75 UDP 123
Dec 11 05:01:03 node6 ntpd[3223]: Listen normally on 211 tap149i1 fe80::e026:6aff:fe9f:5d67 UDP 123
Dec 11 05:01:03 node6 ntpd[3223]: Deleting interface #209 tap148i0, fe80::f0fc:1ff:fed7:53e4#123, interface stats: received=0, sent=0, dropped=0, active_time=2100 secs
Dec 11 05:01:03 node6 ntpd[3223]: Deleting interface #208 tap148i1, fe80::1418:a1ff:fec6:815#123, interface stats: received=0, sent=0, dropped=0, active_time=2100 secs
Dec 11 05:01:03 node6 ntpd[3223]: peers refreshed
Dec 11 05:01:36 node6 pmxcfs[734255]: [status] notice: received log
Dec 11 05:01:50 node6 pvestatd[737374]: status update time (5.887 seconds)
Dec 11 05:14:06 node6 pvestatd[737374]: WARNING: command 'df -P -B 1 /mnt/pve/images' failed: got timeout
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] Members Left:
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] Members Joined:
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] New Configuration:
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 05:15:52 node6 corosync[3418]: [CLM ] #011r(0) ip(192.168.11.33)
 
this is the default management node (uses ngnix as reverse proxy to the outer world)

pveversion --verbose
proxmox-ve-2.6.32: 3.3-138 (running kernel: 3.2.0-4-amd64)
pve-manager: 3.3-2 (running version: 3.3-2/995e687e)
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-35
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-9
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
--------------------------------------

Dec 11 04:01:35 node7 pvedaemon[4259]: <root@pam> successful auth for user 'root@pam'
Dec 11 04:03:26 node7 pveproxy[1997]: worker 3453 finished
Dec 11 04:03:26 node7 pveproxy[1997]: starting 1 worker(s)
Dec 11 04:03:26 node7 pveproxy[1997]: worker 21023 started
Dec 11 04:05:01 node7 /USR/SBIN/CRON[21602]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 04:11:47 node7 corosync[3471]: [TOTEM ] Retransmit List: 5ee6
Dec 11 04:12:49 node7 corosync[3471]: [TOTEM ] A processor failed, forming new configuration.
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:12:49 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:12:49 node7 corosync[3471]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:12:49 node7 corosync[3471]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:0)
Dec 11 04:12:49 node7 corosync[3471]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:14:57 node7 corosync[3471]: [TOTEM ] Retransmit List: 9c6
Dec 11 04:15:01 node7 /USR/SBIN/CRON[25325]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 04:16:35 node7 pvedaemon[4259]: <root@pam> successful auth for user 'root@pam'
Dec 11 04:17:01 node7 /USR/SBIN/CRON[26078]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 11 04:24:01 node7 pmxcfs[32713]: [status] notice: received log
Dec 11 04:25:01 node7 /USR/SBIN/CRON[29106]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 04:31:36 node7 pvedaemon[4259]: <root@pam> successful auth for user 'root@pam'
Dec 11 04:35:01 node7 /USR/SBIN/CRON[386]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 04:36:16 node7 corosync[3471]: [TOTEM ] A processor failed, forming new configuration.
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:36:28 node7 corosync[3471]: [QUORUM] Members[10]: 1 2 3 4 5 6 8 9 10 11
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:36:28 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:36:28 node7 corosync[3471]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:36:28 node7 kernel: [9533947.375655] dlm: closing connection to node 7
Dec 11 04:36:28 node7 corosync[3471]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:1)
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: starting data syncronisation
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: starting data syncronisation
Dec 11 04:36:28 node7 corosync[3471]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006D)
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006D)
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: received all states
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: leader is 1/401980
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: synced members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:36:28 node7 pmxcfs[32713]: [dcdb] notice: all data is up to date
Dec 11 04:36:30 node7 pmxcfs[32713]: [dcdb] notice: received all states
Dec 11 04:36:30 node7 pmxcfs[32713]: [dcdb] notice: all data is up to date
Dec 11 04:36:30 node7 pmxcfs[32713]: [status] notice: dfsm_deliver_queue: queue length 427
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:36:32 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:36:32 node7 corosync[3471]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:36:32 node7 corosync[3471]: [QUORUM] Members[11]: 1 2 3 4 5 6 7 8 9 10 11
Dec 11 04:36:32 node7 corosync[3471]: [QUORUM] Members[11]: 1 2 3 4 5 6 7 8 9 10 11
Dec 11 04:36:32 node7 corosync[3471]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:10 left:0)
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 7/734255, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: starting data syncronisation
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 7/734255, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: starting data syncronisation
Dec 11 04:36:32 node7 corosync[3471]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006E)
Dec 11 04:36:32 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006E)
Dec 11 04:38:33 node7 corosync[3471]: [TOTEM ] A processor failed, forming new configuration.
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:38:45 node7 corosync[3471]: [QUORUM] Members[10]: 1 2 3 4 5 6 8 9 10 11
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.6)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:38:45 node7 corosync[3471]: [QUORUM] Members[10]: 1 2 3 4 5 6 8 9 10 11
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] CLM CONFIGURATION CHANGE
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] New Configuration:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.1)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.2)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.3)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.4)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.5)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.7)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.8)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.31)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.32)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] #011r(0) ip(192.168.11.33)
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Left:
Dec 11 04:38:45 node7 corosync[3471]: [CLM ] Members Joined:
Dec 11 04:38:45 node7 corosync[3471]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 11 04:38:45 node7 kernel: [9534084.301446] dlm: closing connection to node 7
Dec 11 04:38:45 node7 corosync[3471]: [CPG ] chosen downlist: sender r(0) ip(192.168.11.1) ; members(old:11 left:1)
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:38:45 node7 corosync[3471]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006F)
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: received sync request (epoch 1/401980/0000006F)
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: received all states
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: leader is 1/401980
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: synced members: 1/401980, 2/32713, 3/446279, 4/782678, 5/861566, 6/840974, 8/806921, 9/15905, 10/3545, 11/3884
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: all data is up to date
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: received all states
Dec 11 04:38:45 node7 pmxcfs[32713]: [dcdb] notice: all data is up to date
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: dfsm_deliver_queue: queue length 2962
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
Dec 11 04:38:45 node7 pmxcfs[32713]: [status] notice: remove message from non-member 7/734255
 
and yes we have a dedicatet network for the backups. the cluster communication is on the other network. and this nework was more or less idle during night.
 
happened again this. night. it is just one specific node. all other nodes make no problem. the load was not very high during the night. also the network which is used for the cluster communication was idle....

i found this in log:

Dec 18 01:29:12 node6 pvestatd[737374]: WARNING: unable to connect to VM 111 socket - timeout after 31 retries
Dec 18 01:29:15 node6 pvestatd[737374]: status update time (6.349 seconds)
Dec 18 01:29:21 node6 pvestatd[737374]: WARNING: unable to connect to VM 152 socket - timeout after 31 retries
Dec 18 01:29:26 node6 pvestatd[737374]: status update time (7.957 seconds)
Dec 18 01:33:09 node6 pmxcfs[734255]: [status] notice: received log
Dec 18 01:36:49 node6 pmxcfs[734255]: [status] notice: received log
Dec 18 01:36:52 node6 pvestatd[737374]: status update time (13.951 seconds)
Dec 18 01:38:16 node6 pvedaemon[943851]: WARNING: unable to connect to VM 111 socket - timeout after 31 retries
Dec 18 01:38:16 node6 pvedaemon[942451]: WARNING: unable to connect to VM 111 socket - timeout after 31 retries
Dec 18 01:38:26 node6 pvestatd[737374]: WARNING: unable to connect to VM 111 socket - timeout after 31 retries
and lots of this

Dec 18 02:34:39 node6 pvestatd[737374]: status update time (24.717 seconds)
Dec 18 02:34:40 node6 pmxcfs[734255]: [status] notice: node lost quorum
Dec 18 02:34:40 node6 pmxcfs[734255]: [status] notice: node has quorum
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/node5: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/124: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/123: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/120: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/122: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/121: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/121: /var/lib/rrdcached/db/pve2-vm/121: illegal attempt to update using time 1418866455 when last update time is 1418866485 (minimum
one second step)
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/119: -1
Dec 18 02:34:52 node6 pmxcfs[734255]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/119: /var/lib/rrdcached/db/pve2-vm/119: illegal attempt to update using time 1418866455 when last update time is 1418866485 (minimum
one second step)

can it be the cause for the problem?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back