We have a three node cluster withProxmox 3.3-1/a06c9f73 with Ceph up and running.
After each reboot of any of the clusters nodes rgmanager won’t starting automatically on the rebooted node.
Also we are not able to start rgmanager via web interface.
To start rgmanager we have to restart cman and pve-cluster first. After this erverything is running fine.
Each node has three bonded interfaces. Cluster communication is established via bridge connected to one of the bonds.
I’ve tested multicast transmissions (omping / asmping) between all nodes already with the result that the transmission seems to be fine (no losses, constant ping times)
Fencing is configured and tested.
All three nodes are also acting as CEPH server nodes. CEPH communictaion takes place on a dedicated bond interface.
Here are some information I collected directly after reboot:
Nothing in fenced.log
Nothing in dlm_control.log
Syslog directly after reboot:
Any ideas how to solve this problem?
After each reboot of any of the clusters nodes rgmanager won’t starting automatically on the rebooted node.
Also we are not able to start rgmanager via web interface.
To start rgmanager we have to restart cman and pve-cluster first. After this erverything is running fine.
Each node has three bonded interfaces. Cluster communication is established via bridge connected to one of the bonds.
I’ve tested multicast transmissions (omping / asmping) between all nodes already with the result that the transmission seems to be fine (no losses, constant ping times)
Fencing is configured and tested.
All three nodes are also acting as CEPH server nodes. CEPH communictaion takes place on a dedicated bond interface.
Here are some information I collected directly after reboot:
Code:
:~# clustat
Cluster Status for dmc-cluster-ni @ Fri Oct 10 08:56:12 2014
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
lx-vmhost-ni1 1 Online
lx-vmhost-ni0 2 Online
lx-vmhost-ni2 3 Online, Local
Code:
:~# /etc/init.d/rgmanager restart
Stopping Cluster Service Manager: [ OK ]
Starting Cluster Service Manager: [FAILED]
Code:
#clusvcadm -e testservice
Local machine trying to enable service:testservice...Could not connect to resource group manager
Code:
#pvecm status
Version: 6.2.0
Config Version: 16
Cluster Name: dmc-cluster-ni
Cluster Id: 49049
Cluster Member: Yes
Cluster Generation: 304
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 1
Flags:
Ports Bound: 0
Node name: lx-vmhost-ni2
Node ID: 3
Multicast addresses: 239.192.191.89
Node addresses: 172.18.0.37
Code:
#cat /var/log/cluster/rgmanager.log
Oct 10 08:52:29 rgmanager Waiting for quorum to form
Oct 10 08:52:52 rgmanager Quorum formedNothing in fenced.log
Nothing in dlm_control.log
Code:
#cat /var/log/cluster/corosync.log
Oct 10 08:51:40 corosync [MAIN ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Oct 10 08:51:40 corosync [MAIN ] Corosync built-in features: nss
Oct 10 08:51:40 corosync [MAIN ] Successfully read config from /etc/cluster/cluster.conf
Oct 10 08:51:40 corosync [MAIN ] Successfully parsed cman config
Oct 10 08:51:40 corosync [MAIN ] Successfully configured openais services to load
Oct 10 08:51:40 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 10 08:51:40 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 10 08:51:40 corosync [TOTEM ] The network interface [172.18.0.37] is now up.
Oct 10 08:51:40 corosync [QUORUM] Using quorum provider quorum_cman
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Oct 10 08:51:40 corosync [CMAN ] CMAN 1364188437 (built Mar 25 2013 06:14:01) started
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync CMAN membership service 2.90
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais cluster membership service B.01.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais event service B.01.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais checkpoint service B.01.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais message service B.03.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais distributed locking service B.03.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: openais timer service A.01.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync configuration service
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync profile loading service
Oct 10 08:51:40 corosync [QUORUM] Using quorum provider quorum_cman
Oct 10 08:51:40 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Oct 10 08:51:40 corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
Oct 10 08:51:40 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:51:40 corosync [CLM ] New Configuration:
Oct 10 08:51:40 corosync [CLM ] Members Left:
Oct 10 08:51:40 corosync [CLM ] Members Joined:
Oct 10 08:51:40 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:51:40 corosync [CLM ] New Configuration:
Oct 10 08:51:40 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:51:40 corosync [CLM ] Members Left:
Oct 10 08:51:40 corosync [CLM ] Members Joined:
Oct 10 08:51:40 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:51:40 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:51:40 corosync [QUORUM] Members[1]: 3
Oct 10 08:51:40 corosync [QUORUM] Members[1]: 3
Oct 10 08:51:40 corosync [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:0 left:0)
Oct 10 08:51:40 corosync [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:24 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:24 corosync [CLM ] New Configuration:
Oct 10 08:52:24 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:24 corosync [CLM ] Members Left:
Oct 10 08:52:24 corosync [CLM ] Members Joined:
Oct 10 08:52:24 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:24 corosync [CLM ] New Configuration:
Oct 10 08:52:24 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:24 corosync [CLM ] Members Left:
Oct 10 08:52:24 corosync [CLM ] Members Joined:
Oct 10 08:52:24 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:24 corosync [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:1 left:0)
Oct 10 08:52:24 corosync [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:44 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:44 corosync [CLM ] New Configuration:
Oct 10 08:52:44 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:44 corosync [CLM ] Members Left:
Oct 10 08:52:44 corosync [CLM ] Members Joined:
Oct 10 08:52:44 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:44 corosync [CLM ] New Configuration:
Oct 10 08:52:44 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:44 corosync [CLM ] Members Left:
Oct 10 08:52:44 corosync [CLM ] Members Joined:
Oct 10 08:52:44 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:44 corosync [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:1 left:0)
Oct 10 08:52:44 corosync [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:52 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:52 corosync [CLM ] New Configuration:
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:52 corosync [CLM ] Members Left:
Oct 10 08:52:52 corosync [CLM ] Members Joined:
Oct 10 08:52:52 corosync [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:52 corosync [CLM ] New Configuration:
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.32)
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.35)
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.37)
Oct 10 08:52:52 corosync [CLM ] Members Left:
Oct 10 08:52:52 corosync [CLM ] Members Joined:
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.32)
Oct 10 08:52:52 corosync [CLM ] r(0) ip(172.18.0.35)
Oct 10 08:52:52 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:52 corosync [CMAN ] quorum regained, resuming activity
Oct 10 08:52:52 corosync [QUORUM] This node is within the primary component and will provide service.
Oct 10 08:52:52 corosync [QUORUM] Members[2]: 1 3
Oct 10 08:52:52 corosync [QUORUM] Members[2]: 1 3
Oct 10 08:52:52 corosync [QUORUM] Members[3]: 1 2 3
Oct 10 08:52:52 corosync [QUORUM] Members[3]: 1 2 3
Oct 10 08:52:52 corosync [CPG ] chosen downlist: sender r(0) ip(172.18.0.35) ; members(old:2 left:0)
Oct 10 08:52:52 corosync [MAIN ] Completed service synchronization, ready to provide service.Syslog directly after reboot:
Code:
Oct 10 08:51:39 lx-vmhost-ni2 kernel: igb 0000:81:00.1: eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 10 08:51:39 lx-vmhost-ni2 kernel: igb 0000:02:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 10 08:51:39 lx-vmhost-ni2 kernel: igb 0000:02:00.1: eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 10 08:51:39 lx-vmhost-ni2 kernel: bond0: link status definitely up for interface eth0, 1000 Mbps full duplex.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: bond0: link status definitely up for interface eth1, 1000 Mbps full duplex.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: vmbr0: port 1(bond0) entering forwarding state
Oct 10 08:51:39 lx-vmhost-ni2 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Oct 10 08:51:39 lx-vmhost-ni2 kernel: device bond2 entered promiscuous mode
Oct 10 08:51:39 lx-vmhost-ni2 kernel: device eth4 entered promiscuous mode
Oct 10 08:51:39 lx-vmhost-ni2 kernel: device eth5 entered promiscuous mode
Oct 10 08:51:39 lx-vmhost-ni2 kernel: ADDRCONF(NETDEV_UP): bond2: link is not ready
Oct 10 08:51:39 lx-vmhost-ni2 kernel: 8021q: adding VLAN 0 to HW filter on device bond2
Oct 10 08:51:39 lx-vmhost-ni2 kernel: 8021q: adding VLAN 0 to HW filter on device vmbr2
Oct 10 08:51:39 lx-vmhost-ni2 kernel: bond1: link status definitely up for interface eth3, 1000 Mbps full duplex.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: igb 0000:83:00.0: eth4: igb: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 10 08:51:39 lx-vmhost-ni2 kernel: bond2: link status definitely up for interface eth4, 1000 Mbps full duplex.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: vmbr2: port 1(bond2) entering forwarding state
Oct 10 08:51:39 lx-vmhost-ni2 kernel: ADDRCONF(NETDEV_CHANGE): bond2: link becomes ready
Oct 10 08:51:39 lx-vmhost-ni2 kernel: igb 0000:83:00.1: eth5: igb: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 10 08:51:39 lx-vmhost-ni2 kernel: bond2: link status definitely up for interface eth5, 1000 Mbps full duplex.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: RPC: Registered named UNIX socket transport module.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: RPC: Registered udp transport module.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: RPC: Registered tcp transport module.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: Slow work thread pool: Starting up
Oct 10 08:51:39 lx-vmhost-ni2 kernel: Slow work thread pool: Ready
Oct 10 08:51:39 lx-vmhost-ni2 kernel: FS-Cache: Loaded
Oct 10 08:51:39 lx-vmhost-ni2 kernel: NFS: Registering the id_resolver key type
Oct 10 08:51:39 lx-vmhost-ni2 kernel: FS-Cache: Netfs 'nfs' registered for caching
Oct 10 08:51:39 lx-vmhost-ni2 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Oct 10 08:51:39 lx-vmhost-ni2 kernel: Loading iSCSI transport class v2.0-870.
Oct 10 08:51:39 lx-vmhost-ni2 kernel: iscsi: registered transport (tcp)
Oct 10 08:51:39 lx-vmhost-ni2 kernel: iscsi: registered transport (iser)
Oct 10 08:51:39 lx-vmhost-ni2 rrdcached[3594]: starting up
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3589]: ntpd 4.2.6p5@1.2349-o Sat May 12 09:54:55 UTC 2012 (1)
Oct 10 08:51:39 lx-vmhost-ni2 rrdcached[3594]: checking for journal files
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: proto: precision = 0.159 usec
Oct 10 08:51:39 lx-vmhost-ni2 rrdcached[3594]: started new journal /var/lib/rrdcached/journal/rrd.journal.1412923899.648033
Oct 10 08:51:39 lx-vmhost-ni2 rrdcached[3594]: journal processing complete
Oct 10 08:51:39 lx-vmhost-ni2 rrdcached[3594]: listening for connections
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen and drop on 1 v6wildcard :: UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 3 vmbr0 172.18.0.37 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 4 vmbr1 192.168.151.4 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 5 vmbr2 192.168.152.4 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 6 lo ::1 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 7 vmbr1 fe80::225:90ff:fee8:2be8 UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listen normally on 8 vmbr0 fe80::225:90ff:feef:9aea UDP 123
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: peers refreshed
Oct 10 08:51:39 lx-vmhost-ni2 ntpd[3595]: Listening on routing socket on fd #25 for interface updates
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [quorum] crit: quorum_initialize failed: 6
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [quorum] crit: can't initialize service
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [confdb] crit: confdb_initialize failed: 6
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [quorum] crit: can't initialize service
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] crit: cpg_initialize failed: 6
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [quorum] crit: can't initialize service
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] crit: cpg_initialize failed: 6
Oct 10 08:51:39 lx-vmhost-ni2 pmxcfs[3629]: [quorum] crit: can't initialize service
Oct 10 08:51:39 lx-vmhost-ni2 postfix/master[3713]: daemon started -- version 2.9.6, configuration /etc/postfix
Oct 10 08:51:40 lx-vmhost-ni2 /usr/sbin/cron[3764]: (CRON) INFO (pidfile fd = 3)
Oct 10 08:51:40 lx-vmhost-ni2 /usr/sbin/cron[3766]: (CRON) STARTUP (fork ok)
Oct 10 08:51:40 lx-vmhost-ni2 /usr/sbin/cron[3766]: (CRON) INFO (Running @reboot jobs)
Oct 10 08:51:40 lx-vmhost-ni2 kernel: DLM (built Aug 21 2014 08:36:35) installed
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Corosync built-in features: nss
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Successfully read config from /etc/cluster/cluster.conf
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Successfully parsed cman config
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Successfully configured openais services to load
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [TOTEM ] The network interface [172.18.0.37] is now up.
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [QUORUM] Using quorum provider quorum_cman
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CMAN ] CMAN 1364188437 (built Mar 25 2013 06:14:01) started
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync CMAN membership service 2.90
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais cluster membership service B.01.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais event service B.01.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais checkpoint service B.01.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais message service B.03.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais distributed locking service B.03.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: openais timer service A.01.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync extended virtual synchrony service
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync configuration service
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
Oct 10 08:51:40 lx-vmhost-ni2 kernel: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync cluster config database access v1.01
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync profile loading service
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [QUORUM] Using quorum provider quorum_cman
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:51:40 lx-vmhost-ni2 kernel: SGI XFS Quota Management subsystem
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[1]: 3
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[1]: 3
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:0 left:0)
Oct 10 08:51:40 lx-vmhost-ni2 corosync[3915]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:51:40 lx-vmhost-ni2 kernel: XFS (sde1): Mounting Filesystem
Oct 10 08:51:40 lx-vmhost-ni2 kernel: XFS (sde1): Ending clean mount
Oct 10 08:51:40 lx-vmhost-ni2 iscsid: iSCSI daemon with pid=3387 started!
Oct 10 08:51:41 lx-vmhost-ni2 cimserver[3648]: Listening on HTTP port 5988.
Oct 10 08:51:41 lx-vmhost-ni2 cimserver[3648]: Listening on HTTPS port 5989.
Oct 10 08:51:41 lx-vmhost-ni2 cimserver[3648]: Listening on local connection socket.
Oct 10 08:51:41 lx-vmhost-ni2 cimserver[3648]: Started CIM Server version 2.11.1.
Oct 10 08:51:41 lx-vmhost-ni2 cimserver[3648]: CIM Server registration with Internal SLP Failed. Exception: CIM_ERR_METHOD_NOT_AVAILABLE: register
Oct 10 08:51:45 lx-vmhost-ni2 pmxcfs[3629]: [status] notice: update cluster info (cluster name dmc-cluster-ni, version = 16)
Oct 10 08:51:45 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 3/3629
Oct 10 08:51:45 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: all data is up to date
Oct 10 08:51:45 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 3/3629
Oct 10 08:51:45 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: all data is up to date
Oct 10 08:51:46 lx-vmhost-ni2 kernel: vmbr0: no IPv6 routers present
Oct 10 08:51:48 lx-vmhost-ni2 kernel: vmbr1: no IPv6 routers present
Oct 10 08:51:49 lx-vmhost-ni2 kernel: vmbr2: no IPv6 routers present
Oct 10 08:51:59 lx-vmhost-ni2 maxView Storage Manager Agent: [752] Flush and fetch rate set to Medium: controller 1 ( Adaptec ASR7805Q #4B21136D03F Physical Slot: 3 )
Oct 10 08:51:59 lx-vmhost-ni2 ntpd[3595]: Deferring DNS for 0.debian.pool.ntp.org 1
Oct 10 08:52:10 lx-vmhost-ni2 kernel: XFS (sdc1): Mounting Filesystem
Oct 10 08:52:10 lx-vmhost-ni2 kernel: XFS (sdc1): Ending clean mount
Oct 10 08:52:11 lx-vmhost-ni2 ntpd[3595]: Deferring DNS for 1.debian.pool.ntp.org 1
Oct 10 08:52:11 lx-vmhost-ni2 ntpd[4780]: signal_no_reset: signal 17 had flags 4000000
Oct 10 08:52:11 lx-vmhost-ni2 kernel: XFS (sdf1): Mounting Filesystem
Oct 10 08:52:11 lx-vmhost-ni2 kernel: XFS (sdf1): Ending clean mount
Oct 10 08:52:12 lx-vmhost-ni2 kernel: XFS (sdd1): Mounting Filesystem
Oct 10 08:52:12 lx-vmhost-ni2 kernel: XFS (sdd1): Ending clean mount
Oct 10 08:52:13 lx-vmhost-ni2 ntpd[3595]: Listen normally on 9 vmbr2 fe80::225:90ff:fee8:2e20 UDP 123
Oct 10 08:52:13 lx-vmhost-ni2 ntpd[3595]: peers refreshed
Oct 10 08:52:13 lx-vmhost-ni2 ntpd_intres[4780]: DNS 0.debian.pool.ntp.org -> 144.76.118.85
Oct 10 08:52:13 lx-vmhost-ni2 ntpd_intres[4780]: DNS 1.debian.pool.ntp.org -> 148.251.9.60
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:1 left:0)
Oct 10 08:52:24 lx-vmhost-ni2 corosync[3915]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:29 lx-vmhost-ni2 kernel: Netfilter messages via NETLINK v0.30.
Oct 10 08:52:29 lx-vmhost-ni2 pvepw-logger[5725]: starting pvefw logger
Oct 10 08:52:29 lx-vmhost-ni2 kernel: tun: Universal TUN/TAP device driver, 1.6
Oct 10 08:52:29 lx-vmhost-ni2 kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Oct 10 08:52:29 lx-vmhost-ni2 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Oct 10 08:52:29 lx-vmhost-ni2 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Oct 10 08:52:29 lx-vmhost-ni2 kernel: Enabling conntracks and NAT for ve0
Oct 10 08:52:29 lx-vmhost-ni2 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Oct 10 08:52:29 lx-vmhost-ni2 kernel: ploop_dev: module loaded
Oct 10 08:52:29 lx-vmhost-ni2 rgmanager[5876]: Waiting for quorum to form
Oct 10 08:52:29 lx-vmhost-ni2 pve-firewall[5881]: starting server
Oct 10 08:52:29 lx-vmhost-ni2 kernel: ip_set: protocol 6
Oct 10 08:52:30 lx-vmhost-ni2 pvedaemon[5894]: starting server
Oct 10 08:52:30 lx-vmhost-ni2 pvedaemon[5894]: starting 3 worker(s)
Oct 10 08:52:30 lx-vmhost-ni2 pvedaemon[5894]: worker 5896 started
Oct 10 08:52:30 lx-vmhost-ni2 pvedaemon[5894]: worker 5897 started
Oct 10 08:52:30 lx-vmhost-ni2 pvedaemon[5894]: worker 5898 started
Oct 10 08:52:30 lx-vmhost-ni2 pvestatd[5918]: starting server
Oct 10 08:52:30 lx-vmhost-ni2 pveproxy[5923]: starting server
Oct 10 08:52:30 lx-vmhost-ni2 pveproxy[5923]: starting 3 worker(s)
Oct 10 08:52:30 lx-vmhost-ni2 pveproxy[5923]: worker 5924 started
Oct 10 08:52:30 lx-vmhost-ni2 pveproxy[5923]: worker 5925 started
Oct 10 08:52:30 lx-vmhost-ni2 pveproxy[5923]: worker 5926 started
Oct 10 08:52:30 lx-vmhost-ni2 ntpd[3595]: Listen normally on 10 venet0 fe80::1 UDP 123
Oct 10 08:52:30 lx-vmhost-ni2 ntpd[3595]: peers refreshed
Oct 10 08:52:31 lx-vmhost-ni2 pvesh: <root@pam> starting task UPID:lx-vmhost-ni2:00001738:0000190C:5437822F:startall::root@pam:
Oct 10 08:52:31 lx-vmhost-ni2 spiceproxy[5945]: starting server
Oct 10 08:52:31 lx-vmhost-ni2 spiceproxy[5945]: starting 1 worker(s)
Oct 10 08:52:31 lx-vmhost-ni2 spiceproxy[5945]: worker 5946 started
Oct 10 08:52:38 lx-vmhost-ni2 kernel: venet0: no IPv6 routers present
Oct 10 08:52:41 lx-vmhost-ni2 task UPID:lx-vmhost-ni2:00001738:0000190C:5437822F:startall::root@pam:: cluster not ready - no quorum?
Oct 10 08:52:41 lx-vmhost-ni2 pvesh: <root@pam> end task UPID:lx-vmhost-ni2:00001738:0000190C:5437822F:startall::root@pam: cluster not ready - no quorum?
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [CPG ] chosen downlist: sender r(0) ip(172.18.0.37) ; members(old:1 left:0)
Oct 10 08:52:44 lx-vmhost-ni2 corosync[3915]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] CLM CONFIGURATION CHANGE
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] New Configuration:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.32)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.35)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.37)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Left:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] Members Joined:
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.32)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CLM ] #011r(0) ip(172.18.0.35)
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CMAN ] quorum regained, resuming activity
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [QUORUM] This node is within the primary component and will provide service.
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[2]: 1 3
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [status] notice: node has quorum
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[2]: 1 3
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[3]: 1 2 3
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [QUORUM] Members[3]: 1 2 3
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [CPG ] chosen downlist: sender r(0) ip(172.18.0.35) ; members(old:2 left:0)
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 2/5675, 3/3629
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: starting data syncronisation
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 2/5675, 3/3629
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: starting data syncronisation
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 1/5063, 2/5675, 3/3629
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: members: 1/5063, 2/5675, 3/3629
Oct 10 08:52:52 lx-vmhost-ni2 corosync[3915]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: received sync request (epoch 1/5063/0000000F)
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: received sync request (epoch 1/5063/0000000F)
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: received all states
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: leader is 1/5063
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: synced members: 1/5063, 2/5675, 3/3629
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: all data is up to date
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: received all states
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [dcdb] notice: all data is up to date
Oct 10 08:52:52 lx-vmhost-ni2 pmxcfs[3629]: [status] notice: dfsm_deliver_queue: queue length 9
Oct 10 08:52:52 lx-vmhost-ni2 rgmanager[5876]: Quorum formed
Oct 10 08:52:52 lx-vmhost-ni2 kernel: dlm: no local IP address has been set
Oct 10 08:52:52 lx-vmhost-ni2 kernel: dlm: cannot start dlm lowcomms -107
Oct 10 08:53:32 lx-vmhost-ni2 pmxcfs[3629]: [status] notice: received logAny ideas how to solve this problem?