after power lost -> TASK ERROR: cluster not ready - no quorum?

I

informant

Renowned Member
Jan 31, 2012
824
11
83
hi, after power lost on cluster i have following problem on cluster: TASK ERROR: cluster not ready - no quorum?
what can i do here, after reboot its the same issue.

pvecm status
Cannot initialize CMAP service

manual start show the same error.

what can i do here, plwase help, thanks and best regards.
last version of proxmox 4.* is installed.

the cluster works fine before, since more as 4 years...
the error comes on all startet nodes, why? works before without problems. what can i do here?

systemctl status corosync.service
* corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: failed (Result: exit-code) since Fr 2016-06-17 15:09:30 CEST; 7min ago
Process: 4727 ExecStart=/usr/share/corosync/corosync start (code=exited, status=1/FAILURE)

Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Unloading all Corosync service engines.
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync configuration map access
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:09:30 pegasus corosync[4727]: Starting Corosync Cluster Engine (corosync): [FAILED]
Jun 17 15:09:30 pegasus systemd[1]: corosync.service: control process exited, code=exited status=1
Jun 17 15:09:30 pegasus systemd[1]: Failed to start Corosync Cluster Engine.
Jun 17 15:09:30 pegasus systemd[1]: Unit corosync.service entered failed state.
Click to expand...

journalctl -xn
-- Logs begin at Fr 2016-06-17 15:02:27 CEST, end at Fr 2016-06-17 15:17:22 CEST. --
Jun 17 15:17:10 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:17:10 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:17:16 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:17:16 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:17:16 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:17:16 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:17:22 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:17:22 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:17:22 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:17:22 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Click to expand...

thanks and regards
 
Last edited:
Jun 17 15:08:27 pegasus pmxcfs[1202]: [main] notice: exit proxmox configuration filesystem (0)
Jun 17 15:08:27 pegasus systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Jun 17 15:08:27 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:27 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:27 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:27 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:08:28 pegasus pmxcfs[4717]: [quorum] crit: can't initialize service
Jun 17 15:08:28 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:08:28 pegasus pmxcfs[4717]: [confdb] crit: can't initialize service
Jun 17 15:08:28 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:08:28 pegasus pmxcfs[4717]: [dcdb] crit: can't initialize service
Jun 17 15:08:28 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:08:28 pegasus pmxcfs[4717]: [status] crit: can't initialize service
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pve-ha-crm[1647]: ipcc_send_rec failed: Der Socket ist nicht verbunden
Jun 17 15:08:28 pegasus pve-ha-crm[1647]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pve-ha-crm[1647]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:28 pegasus pveproxy[1653]: ipcc_send_rec failed: Verbindungsaufbau abgelehnt
Jun 17 15:08:29 pegasus corosync[4733]: [MAIN ] Corosync Cluster Engine ('2.3.5.15-e2b6b'): started and ready to provide service.
Jun 17 15:08:29 pegasus corosync[4733]: [MAIN ] Corosync built-in features: augeas systemd pie relro bindnow
Jun 17 15:08:29 pegasus pveproxy[1651]: ipcc_send_rec failed: Der Socket ist nicht verbunden
Jun 17 15:08:29 pegasus pvedaemon[1640]: ipcc_send_rec failed: Der Socket ist nicht verbunden
Jun 17 15:08:29 pegasus corosync[4738]: [TOTEM ] Initializing transport (UDP/IP Multicast).
Jun 17 15:08:29 pegasus corosync[4738]: [TOTEM ] Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Jun 17 15:08:29 pegasus corosync[4738]: [TOTEM ] The network interface [217.69.254.67] is now up.
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync configuration map access [0]
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] server name: cmap
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync configuration service [1]
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] server name: cfg
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] server name: cpg
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync profile loading service [4]
Jun 17 15:08:29 pegasus corosync[4738]: [QUORUM] Using quorum provider corosync_votequorum
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] server name: votequorum
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] server name: quorum
Jun 17 15:08:29 pegasus corosync[4738]: [TOTEM ] A new membership (217.69.254.67:26588) was formed. Members joined: 1
Jun 17 15:08:29 pegasus corosync[4738]: [QUORUM] Members[1]: 1
Jun 17 15:08:29 pegasus corosync[4738]: [MAIN ] Completed service synchronization, ready to provide service.
Jun 17 15:08:29 pegasus corosync[4738]: [TOTEM ] A new membership (217.69.254.64:26592) was formed. Members joined: 3 6 5
Jun 17 15:08:29 pegasus corosync[4738]: [CMAP ] Received config version (13) is different than my config version (14)! Exiting
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Unloading all Corosync service engines.
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync configuration map access
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync configuration service
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync cluster closed process group service v1.01
Jun 17 15:08:29 pegasus corosync[4738]: [QB ] withdrawing server sockets
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync cluster quorum service v0.1
Jun 17 15:08:29 pegasus corosync[4738]: [SERV ] Service engine unloaded: corosync profile loading service
Jun 17 15:08:29 pegasus corosync[4738]: [MAIN ] Corosync Cluster Engine exiting normally
Jun 17 15:08:29 pegasus pve-ha-lrm[1657]: ipcc_send_rec failed: Der Socket ist nicht verbunden
Jun 17 15:08:34 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
...
Jun 17 15:08:46 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:08:46 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:08:46 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:08:49 pegasus systemd-sysv-generator[9854]: Ignoring creation of an alias umountiscsi.service for itself
Jun 17 15:08:49 pegasus systemd[1]: [/run/systemd/generator.late/rgmanager.service:7] Failed to add dependency on +vz.service, ignoring: Invalid argument
Jun 17 15:08:49 pegasus systemd[1]: [/run/systemd/generator.late/rgmanager.service:7] Failed to add dependency on +qemu-server.service, ignoring: Invalid argument
Jun 17 15:08:49 pegasus systemd[1]: [/run/systemd/generator.late/qemu-server.service:7] Failed to add dependency on +iscsi.service, ignoring: Invalid argument
Jun 17 15:08:52 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:08:52 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:08:52 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:08:52 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:08:58 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:08:58 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:08:58 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:08:58 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:09:01 pegasus cron[1283]: (*system*pveupdate) RELOAD (/etc/cron.d/pveupdate)
Jun 17 15:09:04 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Click to expand...
 
Jun 17 15:09:28 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:09:28 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:09:28 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:09:28 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:09:30 pegasus corosync[4727]: Starting Corosync Cluster Engine (corosync): [FAILED]
Jun 17 15:09:30 pegasus systemd[1]: corosync.service: control process exited, code=exited status=1
Jun 17 15:09:30 pegasus systemd[1]: Failed to start Corosync Cluster Engine.
Jun 17 15:09:30 pegasus systemd[1]: Unit corosync.service entered failed state.
Jun 17 15:09:30 pegasus systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Jun 17 15:09:32 pegasus pvedaemon[10409]: send HUP to 1639
Jun 17 15:09:32 pegasus pvedaemon[1639]: received signal HUP
Jun 17 15:09:32 pegasus pvedaemon[1639]: server closing
Jun 17 15:09:32 pegasus pvedaemon[1639]: server shutdown (restart)
Jun 17 15:09:32 pegasus pvedaemon[1640]: worker exit
Jun 17 15:09:33 pegasus pvedaemon[10428]: worker exit
Jun 17 15:09:34 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:09:34 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:09:34 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:09:34 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:09:34 pegasus pvedaemon[1639]: restarting server
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 1640 finished
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 1642 finished
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 1641 finished
Jun 17 15:09:34 pegasus pvedaemon[1639]: starting 3 worker(s)
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 10450 started
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 10451 started
Jun 17 15:09:34 pegasus pvedaemon[1639]: worker 10452 started
Jun 17 15:09:34 pegasus pveproxy[10431]: send HUP to 1650
Jun 17 15:09:34 pegasus pveproxy[1650]: received signal HUP
Jun 17 15:09:34 pegasus pveproxy[1650]: server closing
Jun 17 15:09:34 pegasus pveproxy[1650]: server shutdown (restart)
Jun 17 15:09:34 pegasus pveproxy[1653]: worker exit
Jun 17 15:09:34 pegasus pveproxy[1651]: worker exit
Jun 17 15:09:35 pegasus pveproxy[10453]: worker exit
Jun 17 15:09:36 pegasus spiceproxy[10456]: send HUP to 1658
Jun 17 15:09:36 pegasus spiceproxy[1658]: received signal HUP
Jun 17 15:09:36 pegasus spiceproxy[1658]: server closing
Jun 17 15:09:36 pegasus spiceproxy[1658]: server shutdown (restart)
Jun 17 15:09:36 pegasus spiceproxy[1659]: worker exit
Jun 17 15:09:36 pegasus pveproxy[1650]: restarting server
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 1652 finished
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 1651 finished
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 1653 finished
Jun 17 15:09:36 pegasus pveproxy[1650]: starting 3 worker(s)
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 10460 started
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 10461 started
Jun 17 15:09:36 pegasus pveproxy[1650]: worker 10463 started
Jun 17 15:09:38 pegasus pvestatd[10462]: send HUP to 1313
Jun 17 15:09:38 pegasus pvestatd[1313]: received signal HUP
Jun 17 15:09:38 pegasus pvestatd[1313]: server shutdown (restart)
Jun 17 15:09:38 pegasus spiceproxy[1658]: restarting server
Jun 17 15:09:38 pegasus spiceproxy[1658]: worker 1659 finished
Jun 17 15:09:38 pegasus spiceproxy[1658]: starting 1 worker(s)
Jun 17 15:09:38 pegasus spiceproxy[1658]: worker 10506 started
Jun 17 15:09:39 pegasus pvestatd[1313]: restarting server
Jun 17 15:09:40 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:09:40 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:09:40 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:09:40 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:09:46 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:09:46 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:09:46 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
...
Jun 17 15:10:22 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:10:28 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:10:28 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:10:28 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:10:28 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Jun 17 15:10:28 pegasus systemd-timesyncd[485]: interval/delta/delay/jitter/drift 512s/+0.001s/0.034s/0.002s/-15ppm (ignored)
...
Jun 17 15:12:10 pegasus pmxcfs[4717]: [quorum] crit: quorum_initialize failed: 2
Jun 17 15:12:10 pegasus pmxcfs[4717]: [confdb] crit: cmap_initialize failed: 2
Jun 17 15:12:10 pegasus pmxcfs[4717]: [dcdb] crit: cpg_initialize failed: 2
Jun 17 15:12:10 pegasus pmxcfs[4717]: [status] crit: cpg_initialize failed: 2
Click to expand...
 
hi, a info, i can not start vms on the nodes and on cluster if they have this error since restart - what can i do? please help :( i have no more ideas...
 
I have the same issue and it has left me in a serious pickle. Help needed!
 
You must log in or register to reply here.
Top Bottom
Back