[quorum] crit: quorum_initialize failed: 2

[jtvdw]

Hi,

One of the 5 nodes (recently added to the cluster) started with this error: [quorum] crit: quorum_initialize failed: 2

Here is the output of "journalctl -xe" :

Oct 20 18:42:55 server pmxcfs[1193]: [quorum] crit: quorum_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [confdb] crit: cmap_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [dcdb] crit: cpg_initialize failed: 2
Oct 20 18:42:55 server pmxcfs[1193]: [status] crit: cpg_initialize failed: 2
Oct 20 18:42:56server audit[20766]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589776390370358/pages/" pid=20766 comm="gitlab-pages" srcname="
Oct 20 18:42:56 server kernel: audit: type=1400 audit(1571589776.394:61962): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589776390370358/pages/" pid
Oct 20 18:42:57 server audit[20844]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589777409217320/pages/" pid=20844 comm="gitlab-pages" srcname="
Oct 20 18:42:57 server kernel: audit: type=1400 audit(1571589777.410:61963): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589777409217320/pages/" pid
Oct 20 18:42:58 server audit[20852]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589778427004132/pages/" pid=20852 comm="gitlab-pages" srcname="
Oct 20 18:42:58 server kernel: audit: type=1400 audit(1571589778.430:61964): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589778427004132/pages/" pid
Oct 20 18:42:59 server audit[20866]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589779445129452/pages/" pid=20866 comm="gitlab-pages" srcname="
Oct 20 18:42:59 server kernel: audit: type=1400 audit(1571589779.450:61965): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-308_</var/lib/lxc>" name="/tmp/gitlab-pages-1571589779445129452/pages/" pid

Please let me know if you need more details of additional information.

Thanks.

 

[wolfgang]

Hi,

this looks like your corosync does not work.
please send the output of these commands.

journalctl -u corosync.service
cat /etc/pve/corosync.conf

 

[jtvdw]

Hi,

Here is the output of "journalctl -u corosync.service" :

root@ponder:~# journalctl -u corosync.service
-- Logs begin at Sun 2019-10-20 09:13:32 SAST, end at Mon 2019-10-21 08:33:53 SAST. --
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] link: host: 1 link: 0 is down
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:13:55 ponder corosync[1202]: [KNET ] host: host: 1 has no active links
Oct 20 18:13:57 ponder corosync[1202]: [KNET ] rx: host: 1 link: 0 is up
Oct 20 18:13:57 ponder corosync[1202]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:22:19 ponder systemd[1]: corosync.service: Main process exited, code=killed, status=11/SEGV
Oct 20 18:22:19 ponder systemd[1]: corosync.service: Failed with result 'signal'.
Oct 20 18:56:59 ponder systemd[1]: Starting Corosync Cluster Engine...
Oct 20 18:56:59 ponder corosync[17159]: [MAIN ] Corosync Cluster Engine 3.0.2-dirty starting up
Oct 20 18:56:59 ponder corosync[17159]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf snmp pie relro indnow
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] Initializing transport (Kronosnet).
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] kronosnet crypto initialized: aes256/sha256
Oct 20 18:56:59 ponder corosync[17159]: [TOTEM ] totemknet initialized
Oct 20 18:56:59 ponder corosync[17159]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync configuration map access [0]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cmap
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync configuration service [1]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cfg
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: cpg
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync profile loading service [4]
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Oct 20 18:57:00 ponder corosync[17159]: [WD ] Watchdog not enabled by configuration
Oct 20 18:57:00 ponder corosync[17159]: [WD ] resource load_15min missing a recovery key.
Oct 20 18:57:00 ponder corosync[17159]: [WD ] resource memory_used missing a recovery key.
Oct 20 18:57:00 ponder corosync[17159]: [WD ] no resources configured.
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync watchdog service [7]
Oct 20 18:57:00 ponder corosync[17159]: [QUORUM] Using quorum provider corosync_votequorum
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: votequorum
Oct 20 18:57:00 ponder corosync[17159]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Oct 20 18:57:00 ponder corosync[17159]: [QB ] server name: quorum
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 3 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [TOTEM ] A new membership (6:22740) was formed. Members joined: 6
Oct 20 18:57:00 ponder systemd[1]: Started Corosync Cluster Engine.
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 4 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 6 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 0)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [CPG ] downlist left_list: 0 received
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 1 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [QUORUM] Members[1]: 6
Oct 20 18:57:00 ponder corosync[17159]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 has no active links
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 20 18:57:00 ponder corosync[17159]: [KNET ] host: host: 2 has no active links

And here is the output of "cat /etc/pve/corosync.conf" :

root@ponder:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: eric
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.8.19
}
node {
name: jimmy
nodeid: 4
quorum_votes: 1
ring0_addr: 192.168.8.14
}
node {
name: ponder
nodeid: 6
quorum_votes: 1
ring0_addr: 192.168.8.17
}
node {
name: ronald
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.8.16
}
node {
name: tawneee
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.8.15
}
node {
name: vorbis
nodeid: 5
quorum_votes: 1
ring0_addr: 192.168.8.13
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: pve6
config_version: 8
interface {
linknumber: 0
}
ip_version: ipv4-6
secauth: on
version: 2
}

Thanks.

 

[wolfgang]

Can you also send the ceph.conf

cat /etc/pve/ceph.conf

 

[jtvdw]

Hi,

Here is the output, couldn't find the "/etc/pve/ceph.conf" file on any of the cluster nodes: (Below is what I did find)

root@ponder:~# cat /etc/pve/ceph.conf
cat: /etc/pve/ceph.conf: No such file or directory

root@ponder:~# locate ceph.conf
/usr/lib/tmpfiles.d/ceph.conf

root@ponder:~# cat /usr/lib/tmpfiles.d/ceph.conf
d /run/ceph 0770 ceph ceph -

Thanks.

 

[wolfgang]

I confused the post with another.
So ignore the ceph part.
Do you have any heavy network jobs running on this network?
Like backup jobs?

I would recommend you to use a second link to make corosync network redundant.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[jtvdw]

Hi,

No worries, these thing happens.

Not really, nothing heavy running. We do have a Gitlab server running on that spesific node but that never cuased any issues. It was running on a pve5 node but was migraed to the new pve6 node.

Is there perhaps other logs I can have al ook at and send them to you.

Thanks.

 

[wolfgang]

You can also check the syslogs.
Can you please send the current versions of packages you use on this node?

pveversion -v

 

[jtvdw]

Hi,

I resolved the issue. The disk space on the container was filled up, increased the space and it fixed the issue.

Thanks.

 

[jsterr]

Hi,

I resolved the issue. The disk space on the container was filled up, increased the space and it fixed the issue.

Thanks.

Can you explain how a container disk space fillup has something to do with corosync on the pve-hypervisorhost?