[quorum] crit: quorum_initialize failed: 2

jtvdw

Member
Dec 6, 2018
44
0
6
Cape Town, South Africa
Hi,

One of the 5 nodes (recently added to the cluster) started with this error: [quorum] crit: quorum_initialize failed: 2

Here is the output of "journalctl -xe" :

Oct 20 18:42:55 server pmxcfs[1193]: [quorum] crit: quorum_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [confdb] crit: cmap_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [dcdb] crit: cpg_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [status] crit: cpg_initialize failed: 2
Oct 20 18:42:56server audit[20766]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589776390370358/pages/" pid=20766 comm="gitlab-pages" srcname="
Oct 20 18:42:56 server kernel: audit: type=1400 audit(1571589776.394:61962): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589776390370358/pages/" pid
Oct 20 18:42:57 server audit[20844]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589777409217320/pages/" pid=20844 comm="gitlab-pages" srcname="
Oct 20 18:42:57 server kernel: audit: type=1400 audit(1571589777.410:61963): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589777409217320/pages/" pid
Oct 20 18:42:58 server audit[20852]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589778427004132/pages/" pid=20852 comm="gitlab-pages" srcname="
Oct 20 18:42:58 server kernel: audit: type=1400 audit(1571589778.430:61964): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589778427004132/pages/" pid
Oct 20 18:42:59 server audit[20866]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589779445129452/pages/" pid=20866 comm="gitlab-pages" srcname="
Oct 20 18:42:59 server kernel: audit: type=1400 audit(1571589779.450:61965): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589779445129452/pages/" pid

Please let me know if you need more details of additional information.

Thanks.
 
Hi,

this looks like your corosync does not work.
please send the output of these commands.
Code:
journalctl -u corosync.service
cat /etc/pve/corosync.conf
 
Hi,

Here is the output of "journalctl -u corosync.service" :

root@ponder:~# journalctl -u corosync.service
-- Logs begin at Sun 2019-10-20 09:13:32 SAST, end at Mon 2019-10-21 08:33:53 SAST. --
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] link: host: 1 link: 0 is down
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] host: host: 1 has no active links
Oct 20 18:13:57 ponder corosync[1202]: [KNET ] rx: host: 1 link: 0 is up
Oct 20 18:13:57 ponder corosync[1202]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:22:19 ponder systemd[1]: corosync.service: Main process exited, code=killed, status=11/SEGV
Oct 20 18:22:19 ponder systemd[1]: corosync.service: Failed with result 'signal'.
Oct 20 18:56:59 ponder systemd[1]: Starting Corosync Cluster Engine...
Oct 20 18:56:59 ponder corosync[17159]: [MAIN ] Corosync Cluster Engine 3.0.2-dirty starting up
Oct 20 18:56:59 ponder corosync[17159]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf snmp pie relro indnow
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] Initializing transport (Kronosnet).
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] kronosnet crypto initialized: aes256/sha256
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] totemknet initialized
Oct 20 18:56:59 ponder corosync[17159]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync configuration map access [0]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cmap
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync configuration service [1]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cfg
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cpg
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync profile loading service [4]
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Oct 20 18:57:00 ponder corosync[17159]: [WD ] Watchdog not enabled by configuration
Oct 20 18:57:00 ponder corosync[17159]: [WD ] resource load_15min missing a recovery key.
Oct 20 18:57:00 ponder corosync[17159]: [WD ] resource memory_used missing a recovery key.
Oct 20 18:57:00 ponder corosync[17159]: [WD ] no resources configured.
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync watchdog service [7]
Oct 20 18:57:00 ponder corosync[17159]: [QUORUM] Using quorum provider corosync_votequorum
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: votequorum
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: quorum
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [TOTEM ] A new membership (6:22740) was formed. Members joined: 6
Oct 20 18:57:00 ponder systemd[1]: Started Corosync Cluster Engine.
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 6 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [CPG ] downlist left_list: 0 received
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [QUORUM] Members[1]: 6
Oct 20 18:57:00 ponder corosync[17159]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 has no active links

And here is the output of "cat /etc/pve/corosync.conf" :

root@ponder:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: eric
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.8.19
}
node {
name: jimmy
nodeid: 4
quorum_votes: 1
ring0_addr: 192.168.8.14
}
node {
name: ponder
nodeid: 6
quorum_votes: 1
ring0_addr: 192.168.8.17
}
node {
name: ronald
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.8.16
}
node {
name: tawneee
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.8.15
}
node {
name: vorbis
nodeid: 5
quorum_votes: 1
ring0_addr: 192.168.8.13
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: pve6
config_version: 8
interface {
linknumber: 0
}
ip_version: ipv4-6
secauth: on
version: 2
}

Thanks.
 
Can you also send the ceph.conf
Code:
cat /etc/pve/ceph.conf
 
Hi,

Here is the output, couldn't find the "/etc/pve/ceph.conf" file on any of the cluster nodes: (Below is what I did find)

root@ponder:~# cat /etc/pve/ceph.conf
cat: /etc/pve/ceph.conf: No such file or directory

root@ponder:~# locate ceph.conf
/usr/lib/tmpfiles.d/ceph.conf

root@ponder:~# cat /usr/lib/tmpfiles.d/ceph.conf
d /run/ceph 0770 ceph ceph -

Thanks.
 
Hi,

No worries, these thing happens.

Not really, nothing heavy running. We do have a Gitlab server running on that spesific node but that never cuased any issues. It was running on a pve5 node but was migraed to the new pve6 node.

Is there perhaps other logs I can have al ook at and send them to you.

Thanks.
 
You can also check the syslogs.
Can you please send the current versions of packages you use on this node?

Code:
pveversion -v
 
Hi,

I resolved the issue. The disk space on the container was filled up, increased the space and it fixed the issue.

Thanks.

Can you explain how a container disk space fillup has something to do with corosync on the pve-hypervisorhost?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!