HA

frantek

Renowned Member
May 30, 2009
176
7
83
Hi,

I've a HA test setup with NFS storage and 3 nodes. Everyting works as expected. VMs could be migrated from one node to an other. Fencing is set up as described in the wiki. But when I remove power from a node to simulate a failure the VMs running on this node are not automatically started on an other node. Why?

TIA
 
post your cluster.conf file

> cat /etc/pve/cluster.conf

and check logs.
 
cat /etc/cluster/cluster.conf
Code:
<?xml version="1.0"?>
<cluster config_version="9" name="CL-SSB">
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey"/>
  <fencedevices>
    <fencedevice agent="fence_bladecenter" ipaddr="172.27.1.250" login="test" name="blade_mmc" passwd="test1"/>
  </fencedevices>
  <clusternodes>
    <clusternode name="sbapm01" nodeid="1" votes="1">
      <fence>
        <method name="1">
          <device port="1" name="blade_mmc"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="sbapm02" nodeid="2" votes="1">
      <fence>
        <method name="1">
          <device port="2" name="blade_mmc"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="sbapm03" nodeid="3" votes="1">
      <fence>
        <method name="1">
          <device port="3" name="blade_mmc"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <rm>
    <pvevm autostart="1" vmid="100"/>
  </rm>
</cluster>

fence_tool ls
Code:
fence domain
member count  3
victim count  0
victim now    0
master nodeid 2
wait state    none
members       1 2 3

pvecm status
Code:
Version: 6.2.0
Config Version: 9
Cluster Name: CL-SSB
Cluster Id: 4284
Cluster Member: Yes
Cluster Generation: 88
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 6
Flags: 
Ports Bound: 0  
Node name: sbapm01
Node ID: 1
Multicast addresses: 239.192.16.204 
Node addresses: 10.1.6.51

fence_bladecenter -h
Code:
Usage:
    fence_bladecenter [options]
Options:
   -a, --ip=[ip]                  IP address or hostname of fencing device
   -l, --username=[name]          Login name
   -p, --password=[password]      Login password or passphrase
   -c, --command-prompt=[prompt]  Force command prompt
   -x, --ssh                      Use ssh connection
   -n, --plug=[id]                Physical plug number on device, UUID or
                                        identification of machine
   -u, --ipport=[port]            TCP/UDP port to use
                                        (default 23, 22 if --ssh option is used)
   -4, --inet4-only               Forces agent to use IPv4 addresses only
   -6, --inet6-only               Forces agent to use IPv6 addresses only
   -S, --password-script=[script] Script to run to retrieve password
   -k, --identity-file=[filename] Identity file (private key) for ssh 
   --ssh-options=[options]      SSH options to use
   -o, --action=[action]          Action: status, reboot (default), off or on
   -v, --verbose                  Verbose mode
   -D, --debug-file=[debugfile]   Debugging to output file
   -V, --version                  Output version information and exit
   -h, --help                     Display this help and exit
   -C, --separator=[char]         Separator for CSV created by 'list' operation
   --missing-as-off               Missing port returns OFF instead of failure
   --power-timeout=[seconds]      Test X seconds for status change after ON/OFF
   --shell-timeout=[seconds]      Wait X seconds for cmd prompt after issuing command
   --login-timeout=[seconds]      Wait X seconds for cmd prompt after login
   --power-wait=[seconds]         Wait X seconds after issuing ON/OFF
   --delay=[seconds]              Wait X seconds before fencing is started
   --retry-on=[attempts]          Count of attempts to retry power on

fence_bladecenter -a 172.27.1.250 --username="test" --password="test1" -o status -n 2 -v
Code:
telnet> set binary
Negotiating binary mode with remote host.
telnet> open 172.27.1.250 -23
Trying 172.27.1.250...
Connected to 172.27.1.250.
Escape character is '^]'.

username: password: 

Hostname:              Blade4
Static IP address:     172.27.1.250
Burned-in MAC address: 00:11:25:C3:44:D0
DHCP:                  Disabled - Use static IP configuration.
Last login: Unable to connect/login to fencing device

fence_bladecenter -a 172.27.1.250 --username="test" --password="test1" -o status -n 2 -v -x
Code:
password:

Hostname:              Blade4
Static IP address:     172.27.1.250
Burned-in MAC address: 00:11:25:C3:44:D0
DHCP:                  Disabled - Use static IP configuration.
Last login: Tuesday November 5 2013 11:39 from 10.1.6.52 (Telnet)

system> env -T system:blade[2]
OK
system:blade[2]> power -state
On
system:blade[2]> env -T system
OK
system>Status: ON
 
Finaly found some time to assemble data:

rgmanager.log
Code:
Oct 28 08:31:33 rgmanager I am node #1
Oct 28 08:31:33 rgmanager Resource Group Manager Starting
Oct 28 08:31:33 rgmanager Loading Service Data
Oct 28 08:31:35 rgmanager Initializing Services
Oct 28 08:31:36 rgmanager [pvevm] VM 100 is already stopped
Oct 28 08:31:36 rgmanager Services Initialized
Oct 28 08:31:36 rgmanager State change: Local UP
Oct 28 08:31:36 rgmanager State change: sbapm02 UP
Oct 28 08:31:36 rgmanager State change: sbapm03 UP
Oct 28 08:31:37 rgmanager Migration: pvevm:100 is running on 3
Oct 28 08:32:48 rgmanager [pvevm] VM 100 is not running
Oct 28 08:33:03 rgmanager [pvevm] VM 100 is not running
Oct 28 08:33:05 rgmanager [pvevm] VM 100 is not running
Oct 28 08:34:32 rgmanager State change: sbapm03 DOWN
Oct 28 08:37:46 rgmanager Shutting down
Oct 28 08:37:56 rgmanager Shutting down
Oct 28 08:40:05 rgmanager Disconnecting from CMAN
Oct 28 08:40:05 rgmanager Exiting
Oct 28 08:43:45 rgmanager Waiting for quorum to form
Oct 28 08:44:23 rgmanager Quorum formed

fenced.log
Code:
Oct 18 12:54:51 fenced fenced 1364188437 started
Oct 28 08:34:32 fenced fencing node sbapm03
Oct 28 08:34:32 fenced fence sbapm03 dev 0.0 agent none result: error no method
Oct 28 08:34:32 fenced fence sbapm03 failed
Oct 28 08:34:35 fenced fencing node sbapm03
Oct 28 08:34:35 fenced fence sbapm03 dev 0.0 agent none result: error no method
Oct 28 08:34:35 fenced fence sbapm03 failed
Oct 28 08:34:38 fenced fencing node sbapm03
Oct 28 08:34:38 fenced fence sbapm03 dev 0.0 agent none result: error no method
Oct 28 08:34:38 fenced fence sbapm03 failed
Oct 28 09:34:40 fenced fenced 1364188437 started
Nov 05 11:43:52 fenced fenced 1364188437 started
dlm_controld.log
Code:
Oct 18 12:54:51 dlm_controld dlm_controld 1364188437 started
Oct 28 09:34:40 dlm_controld dlm_controld 1364188437 started
Nov 05 11:43:52 dlm_controld dlm_controld 1364188437 started
cpglockd.log
Code:
Oct 28 08:37:30 cpglockd cpglockd entering normal operation
Oct 28 08:40:08 cpglockd cman requested shutdown. Exiting.
 
corosync.log
Code:
Oct 18 12:54:47 corosync [MAIN  ] Corosync Cluster Engine ('1.4.5'): started and ready to provide service.
Oct 18 12:54:47 corosync [MAIN  ] Corosync built-in features: nss
Oct 18 12:54:47 corosync [MAIN  ] Successfully read config from /etc/cluster/cluster.conf
Oct 18 12:54:47 corosync [MAIN  ] Successfully parsed cman config
Oct 18 12:54:47 corosync [MAIN  ] Successfully configured openais services to load
Oct 18 12:54:47 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 18 12:54:47 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 18 12:54:47 corosync [TOTEM ] The network interface [10.1.6.51] is now up.
Oct 18 12:54:47 corosync [QUORUM] Using quorum provider quorum_cman
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 18 12:54:47 corosync [CMAN  ] CMAN 1364188437 (built Mar 25 2013 06:14:01) started
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync CMAN membership service 2.90
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais cluster membership service B.01.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais event service B.01.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais message service B.03.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais distributed locking service B.03.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: openais timer service A.01.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync configuration service
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync profile loading service
Oct 18 12:54:47 corosync [QUORUM] Using quorum provider quorum_cman
Oct 18 12:54:47 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 18 12:54:47 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Oct 18 12:54:47 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 12:54:47 corosync [CLM   ] New Configuration:
Oct 18 12:54:47 corosync [CLM   ] Members Left:
Oct 18 12:54:47 corosync [CLM   ] Members Joined:
Oct 18 12:54:47 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 12:54:47 corosync [CLM   ] New Configuration:
Oct 18 12:54:47 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 12:54:47 corosync [CLM   ] Members Left:
Oct 18 12:54:47 corosync [CLM   ] Members Joined:
Oct 18 12:54:47 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 12:54:47 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 18 12:54:47 corosync [CMAN  ] quorum regained, resuming activity
Oct 18 12:54:47 corosync [QUORUM] This node is within the primary component and will provide service.
Oct 18 12:54:47 corosync [QUORUM] Members[1]: 1
Oct 18 12:54:47 corosync [QUORUM] Members[1]: 1
Oct 18 12:54:47 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:0 left:0)
Oct 18 12:54:47 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 18 12:58:03 corosync [QUORUM] Members[1]: 1
Oct 18 12:58:17 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 12:58:17 corosync [CLM   ] New Configuration:
Oct 18 12:58:17 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 12:58:17 corosync [CLM   ] Members Left:
Oct 18 12:58:17 corosync [CLM   ] Members Joined:
Oct 18 12:58:17 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 12:58:17 corosync [CLM   ] New Configuration:
Oct 18 12:58:17 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 12:58:17 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 12:58:17 corosync [CLM   ] Members Left:
Oct 18 12:58:17 corosync [CLM   ] Members Joined:
Oct 18 12:58:17 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 12:58:17 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 18 12:58:17 corosync [QUORUM] Members[2]: 1 2
Oct 18 12:58:17 corosync [QUORUM] Members[2]: 1 2
Oct 18 12:58:17 corosync [QUORUM] Members[2]: 1 2
Oct 18 12:58:17 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:1 left:0)
Oct 18 12:58:17 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 18 11:26:32 corosync [QUORUM] Members[2]: 1 2
Oct 18 11:26:46 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:26:46 corosync [CLM   ] New Configuration:
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 11:26:46 corosync [CLM   ] Members Left:
Oct 18 11:26:46 corosync [CLM   ] Members Joined:
Oct 18 11:26:46 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:26:46 corosync [CLM   ] New Configuration:
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:26:46 corosync [CLM   ] Members Left:
Oct 18 11:26:46 corosync [CLM   ] Members Joined:
Oct 18 11:26:46 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:26:46 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 18 11:26:46 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:26:46 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:26:46 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:26:46 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 18 11:26:46 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 18 11:28:47 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:28:47 corosync [CLM   ] New Configuration:
Oct 18 11:28:47 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:28:47 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:28:47 corosync [CLM   ] Members Left:
Oct 18 11:28:47 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 11:28:47 corosync [CLM   ] Members Joined:
Oct 18 11:28:47 corosync [QUORUM] Members[2]: 1 3
Oct 18 11:28:47 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:28:47 corosync [CLM   ] New Configuration:
Oct 18 11:28:47 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:28:47 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:28:47 corosync [CLM   ] Members Left:
Oct 18 11:28:47 corosync [CLM   ] Members Joined:
Oct 18 11:28:47 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 18 11:28:47 corosync [QUORUM] Members[2]: 1 3
Oct 18 11:28:47 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 18 11:28:47 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 18 11:38:59 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:38:59 corosync [CLM   ] New Configuration:
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:38:59 corosync [CLM   ] Members Left:
Oct 18 11:38:59 corosync [CLM   ] Members Joined:
Oct 18 11:38:59 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 18 11:38:59 corosync [CLM   ] New Configuration:
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 18 11:38:59 corosync [CLM   ] Members Left:
Oct 18 11:38:59 corosync [CLM   ] Members Joined:
Oct 18 11:38:59 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 18 11:38:59 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 18 11:38:59 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:38:59 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:38:59 corosync [QUORUM] Members[3]: 1 2 3
Oct 18 11:38:59 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 18 11:38:59 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 22 10:27:29 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 22 10:27:31 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:27:31 corosync [CLM   ] New Configuration:
Oct 22 10:27:31 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:27:31 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:27:31 corosync [CLM   ] Members Left:
Oct 22 10:27:31 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 22 10:27:31 corosync [CLM   ] Members Joined:
Oct 22 10:27:31 corosync [QUORUM] Members[2]: 1 2
Oct 22 10:27:31 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:27:31 corosync [CLM   ] New Configuration:
Oct 22 10:27:31 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:27:31 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:27:31 corosync [CLM   ] Members Left:
Oct 22 10:27:31 corosync [CLM   ] Members Joined:
Oct 22 10:27:31 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 22 10:27:31 corosync [QUORUM] Members[2]: 1 2
Oct 22 10:27:31 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 22 10:27:31 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 22 10:34:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:34:23 corosync [CLM   ] New Configuration:
Oct 22 10:34:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:34:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:34:23 corosync [CLM   ] Members Left:
Oct 22 10:34:23 corosync [CLM   ] Members Joined:
Oct 22 10:34:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:34:23 corosync [CLM   ] New Configuration:
Oct 22 10:34:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:34:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:34:23 corosync [CLM   ] Members Left:
Oct 22 10:34:23 corosync [CLM   ] Members Joined:
Oct 22 10:34:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 22 10:34:23 corosync [QUORUM] Members[2]: 1 2
Oct 22 10:34:23 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 22 10:34:23 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 22 10:34:25 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:34:25 corosync [CLM   ] New Configuration:
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:34:25 corosync [CLM   ] Members Left:
Oct 22 10:34:25 corosync [CLM   ] Members Joined:
Oct 22 10:34:25 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 22 10:34:25 corosync [CLM   ] New Configuration:
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 22 10:34:25 corosync [CLM   ] Members Left:
Oct 22 10:34:25 corosync [CLM   ] Members Joined:
Oct 22 10:34:25 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 22 10:34:25 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 22 10:34:25 corosync [QUORUM] Members[3]: 1 2 3
Oct 22 10:34:25 corosync [QUORUM] Members[3]: 1 2 3
Oct 22 10:34:25 corosync [QUORUM] Members[3]: 1 2 3
Oct 22 10:34:25 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 22 10:34:25 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:13:40 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:18:57 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:18:57 corosync [CLM   ] New Configuration:
Oct 28 08:18:57 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:18:57 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:18:57 corosync [CLM   ] Members Left:
Oct 28 08:18:57 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:18:57 corosync [CLM   ] Members Joined:
Oct 28 08:18:57 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:18:57 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:18:57 corosync [CLM   ] New Configuration:
Oct 28 08:18:57 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:18:57 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:18:57 corosync [CLM   ] Members Left:
Oct 28 08:18:57 corosync [CLM   ] Members Joined:
Oct 28 08:18:57 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:18:57 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:18:57 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 28 08:18:57 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:22:39 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:22:39 corosync [CLM   ] New Configuration:
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:22:39 corosync [CLM   ] Members Left:
Oct 28 08:22:39 corosync [CLM   ] Members Joined:
Oct 28 08:22:39 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:22:39 corosync [CLM   ] New Configuration:
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:22:39 corosync [CLM   ] Members Left:
Oct 28 08:22:39 corosync [CLM   ] Members Joined:
Oct 28 08:22:39 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:22:39 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:22:39 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:22:39 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:22:39 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:22:39 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 28 08:22:39 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:34:30 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 28 08:34:32 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:34:32 corosync [CLM   ] New Configuration:
Oct 28 08:34:32 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:34:32 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:34:32 corosync [CLM   ] Members Left:
Oct 28 08:34:32 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:34:32 corosync [CLM   ] Members Joined:
Oct 28 08:34:32 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:34:32 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:34:32 corosync [CLM   ] New Configuration:
Oct 28 08:34:32 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:34:32 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:34:32 corosync [CLM   ] Members Left:
Oct 28 08:34:32 corosync [CLM   ] Members Joined:
Oct 28 08:34:32 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:34:32 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:34:32 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 28 08:34:32 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:40:01 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:40:01 corosync [CLM   ] New Configuration:
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:40:01 corosync [CLM   ] Members Left:
Oct 28 08:40:01 corosync [CLM   ] Members Joined:
Oct 28 08:40:01 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:40:01 corosync [CLM   ] New Configuration:
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:40:01 corosync [CLM   ] Members Left:
Oct 28 08:40:01 corosync [CLM   ] Members Joined:
Oct 28 08:40:01 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 08:40:01 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:40:01 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:40:01 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:40:01 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 08:40:01 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 28 08:40:01 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:40:08 corosync [SERV  ] Unloading all Corosync service engines.
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync extended virtual synchrony service
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync configuration service
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync cluster config database access v1.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync profile loading service
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais cluster membership service B.01.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais checkpoint service B.01.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais event service B.01.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais distributed locking service B.03.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais message service B.03.01
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync CMAN membership service 2.90
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Oct 28 08:40:08 corosync [SERV  ] Service engine unloaded: openais timer service A.01.01
Oct 28 08:40:08 corosync [MAIN  ] Corosync Cluster Engine exiting with status 0 at main.c:1893.
Oct 28 08:42:55 corosync [MAIN  ] Corosync Cluster Engine ('1.4.5'): started and ready to provide service.
Oct 28 08:42:55 corosync [MAIN  ] Corosync built-in features: nss
Oct 28 08:42:55 corosync [MAIN  ] Successfully read config from /etc/cluster/cluster.conf
Oct 28 08:42:55 corosync [MAIN  ] Successfully parsed cman config
Oct 28 08:42:55 corosync [MAIN  ] Successfully configured openais services to load
Oct 28 08:42:55 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 28 08:42:55 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 28 08:42:55 corosync [TOTEM ] The network interface [10.1.6.51] is now up.
Oct 28 08:42:55 corosync [QUORUM] Using quorum provider quorum_cman
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 28 08:42:55 corosync [CMAN  ] CMAN 1364188437 (built Mar 25 2013 06:14:01) started
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync CMAN membership service 2.90
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais cluster membership service B.01.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais event service B.01.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais message service B.03.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais distributed locking service B.03.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: openais timer service A.01.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync configuration service
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync profile loading service
Oct 28 08:42:55 corosync [QUORUM] Using quorum provider quorum_cman
Oct 28 08:42:55 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 28 08:42:55 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Oct 28 08:42:55 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:42:55 corosync [CLM   ] New Configuration:
Oct 28 08:42:55 corosync [CLM   ] Members Left:
Oct 28 08:42:55 corosync [CLM   ] Members Joined:
Oct 28 08:42:55 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:42:55 corosync [CLM   ] New Configuration:
Oct 28 08:42:55 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:42:55 corosync [CLM   ] Members Left:
Oct 28 08:42:55 corosync [CLM   ] Members Joined:
Oct 28 08:42:55 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:42:55 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:42:55 corosync [QUORUM] Members[1]: 1
Oct 28 08:42:55 corosync [QUORUM] Members[1]: 1
Oct 28 08:42:55 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:0 left:0)
Oct 28 08:42:55 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 08:44:22 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:44:22 corosync [CLM   ] New Configuration:
Oct 28 08:44:22 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:44:22 corosync [CLM   ] Members Left:
Oct 28 08:44:22 corosync [CLM   ] Members Joined:
Oct 28 08:44:22 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 08:44:22 corosync [CLM   ] New Configuration:
Oct 28 08:44:22 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 08:44:22 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:44:22 corosync [CLM   ] Members Left:
Oct 28 08:44:22 corosync [CLM   ] Members Joined:
Oct 28 08:44:22 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 08:44:22 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 08:44:22 corosync [CMAN  ] quorum regained, resuming activity
Oct 28 08:44:22 corosync [QUORUM] This node is within the primary component and will provide service.
Oct 28 08:44:22 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:44:22 corosync [QUORUM] Members[2]: 1 2
Oct 28 08:44:22 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:1 left:0)
Oct 28 08:44:22 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:14:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:14:23 corosync [CLM   ] New Configuration:
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:14:23 corosync [CLM   ] Members Left:
Oct 28 09:14:23 corosync [CLM   ] Members Joined:
Oct 28 09:14:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:14:23 corosync [CLM   ] New Configuration:
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:14:23 corosync [CLM   ] Members Left:
Oct 28 09:14:23 corosync [CLM   ] Members Joined:
Oct 28 09:14:23 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:14:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:14:23 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:14:23 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:14:23 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 28 09:14:23 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:16:25 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 28 09:16:37 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:16:37 corosync [CLM   ] New Configuration:
Oct 28 09:16:37 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:16:37 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:16:37 corosync [CLM   ] Members Left:
Oct 28 09:16:37 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:16:37 corosync [CLM   ] Members Joined:
Oct 28 09:16:37 corosync [QUORUM] Members[2]: 1 2
Oct 28 09:16:37 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:16:37 corosync [CLM   ] New Configuration:
Oct 28 09:16:37 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:16:37 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:16:37 corosync [CLM   ] Members Left:
Oct 28 09:16:37 corosync [CLM   ] Members Joined:
Oct 28 09:16:37 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:16:37 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 28 09:16:37 corosync [MAIN  ] Completed service synchronization, ready to provide service.
 
Code:
Oct 28 09:19:41 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:19:41 corosync [CLM   ] New Configuration:
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:19:41 corosync [CLM   ] Members Left:
Oct 28 09:19:41 corosync [CLM   ] Members Joined:
Oct 28 09:19:41 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:19:41 corosync [CLM   ] New Configuration:
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:19:41 corosync [CLM   ] Members Left:
Oct 28 09:19:41 corosync [CLM   ] Members Joined:
Oct 28 09:19:41 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:19:41 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:19:41 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:19:41 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:19:41 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 28 09:19:41 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:20:55 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 28 09:21:07 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:21:07 corosync [CLM   ] New Configuration:
Oct 28 09:21:07 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:21:07 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:21:07 corosync [CLM   ] Members Left:
Oct 28 09:21:07 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:21:07 corosync [CLM   ] Members Joined:
Oct 28 09:21:07 corosync [QUORUM] Members[2]: 1 3
Oct 28 09:21:07 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:21:07 corosync [CLM   ] New Configuration:
Oct 28 09:21:07 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:21:07 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:21:07 corosync [CLM   ] Members Left:
Oct 28 09:21:07 corosync [CLM   ] Members Joined:
Oct 28 09:21:07 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:21:07 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:3 left:1)
Oct 28 09:21:07 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:28:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:28:23 corosync [CLM   ] New Configuration:
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:28:23 corosync [CLM   ] Members Left:
Oct 28 09:28:23 corosync [CLM   ] Members Joined:
Oct 28 09:28:23 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:28:23 corosync [CLM   ] New Configuration:
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:28:23 corosync [CLM   ] Members Left:
Oct 28 09:28:23 corosync [CLM   ] Members Joined:
Oct 28 09:28:23 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:28:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:28:23 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:28:23 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:28:23 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:2 left:0)
Oct 28 09:28:23 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:31:31 corosync [SERV  ] Unloading all Corosync service engines.
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync extended virtual synchrony service
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync configuration service
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync cluster config database access v1.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync profile loading service
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais cluster membership service B.01.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais checkpoint service B.01.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais event service B.01.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais distributed locking service B.03.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais message service B.03.01
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync CMAN membership service 2.90
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Oct 28 09:31:31 corosync [SERV  ] Service engine unloaded: openais timer service A.01.01
Oct 28 09:31:31 corosync [MAIN  ] Corosync Cluster Engine exiting with status 0 at main.c:1893.
Oct 28 09:34:17 corosync [MAIN  ] Corosync Cluster Engine ('1.4.5'): started and ready to provide service.
Oct 28 09:34:17 corosync [MAIN  ] Corosync built-in features: nss
Oct 28 09:34:17 corosync [MAIN  ] Successfully read config from /etc/cluster/cluster.conf
Oct 28 09:34:17 corosync [MAIN  ] Successfully parsed cman config
Oct 28 09:34:17 corosync [MAIN  ] Successfully configured openais services to load
Oct 28 09:34:17 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 28 09:34:17 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 28 09:34:17 corosync [TOTEM ] The network interface [10.1.6.51] is now up.
Oct 28 09:34:17 corosync [QUORUM] Using quorum provider quorum_cman
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 28 09:34:17 corosync [CMAN  ] CMAN 1364188437 (built Mar 25 2013 06:14:01) started
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync CMAN membership service 2.90
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais cluster membership service B.01.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais event service B.01.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais message service B.03.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais distributed locking service B.03.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: openais timer service A.01.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync configuration service
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync profile loading service
Oct 28 09:34:17 corosync [QUORUM] Using quorum provider quorum_cman
Oct 28 09:34:17 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Oct 28 09:34:17 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Oct 28 09:34:17 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:34:17 corosync [CLM   ] New Configuration:
Oct 28 09:34:17 corosync [CLM   ] Members Left:
Oct 28 09:34:17 corosync [CLM   ] Members Joined:
Oct 28 09:34:17 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:34:17 corosync [CLM   ] New Configuration:
Oct 28 09:34:17 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:34:17 corosync [CLM   ] Members Left:
Oct 28 09:34:17 corosync [CLM   ] Members Joined:
Oct 28 09:34:17 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:34:17 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:34:17 corosync [QUORUM] Members[1]: 1
Oct 28 09:34:17 corosync [QUORUM] Members[1]: 1
Oct 28 09:34:17 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.51) ; members(old:0 left:0)
Oct 28 09:34:17 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:34:39 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:34:39 corosync [CLM   ] New Configuration:
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:34:39 corosync [CLM   ] Members Left:
Oct 28 09:34:39 corosync [CLM   ] Members Joined:
Oct 28 09:34:39 corosync [CLM   ] CLM CONFIGURATION CHANGE
Oct 28 09:34:39 corosync [CLM   ] New Configuration:
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.51) 
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:34:39 corosync [CLM   ] Members Left:
Oct 28 09:34:39 corosync [CLM   ] Members Joined:
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.52) 
Oct 28 09:34:39 corosync [CLM   ]     r(0) ip(10.1.6.53) 
Oct 28 09:34:39 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 28 09:34:39 corosync [CMAN  ] quorum regained, resuming activity
Oct 28 09:34:39 corosync [QUORUM] This node is within the primary component and will provide service.
Oct 28 09:34:39 corosync [QUORUM] Members[2]: 1 2
Oct 28 09:34:39 corosync [QUORUM] Members[2]: 1 2
Oct 28 09:34:39 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:34:39 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 09:34:39 corosync [CPG   ] chosen downlist: sender r(0) ip(10.1.6.52) ; members(old:2 left:0)
Oct 28 09:34:39 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Oct 28 09:50:00 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 10:24:12 corosync [QUORUM] Members[3]: 1 2 3
Oct 28 10:55:04 corosync [QUORUM] Members[3]: 1 2 3