[SOLVED] After upgrade to 5.2.11 Corosync does not come up

[Jospeh Huber]

Hi,

first of all we use the non-subscription repos.
Today we want to upgrade to the newest version 5.2.11.
After upgrading one node it cannot join the cluster again...
The old version of the remaining running nodes is: 5.2-6.
It cannot join the cluster again. (VM start reports "waiting for quorum")
Nov 22 14:17:17 pxhost1 pmxcfs[3443]: [status] crit: cpg_initialize failed: 2
Nov 22 14:17:23 pxhost1 pmxcfs[3443]: [quorum] crit: quorum_initialize failed: 2
Nov 22 14:17:23 pxhost1 pmxcfs[3443]: [confdb] crit: cmap_initialize failed: 2
Nov 22 14:17:23 pxhost1 pmxcfs[3443]: [dcdb] crit: cpg_initialize failed: 2
Nov 22 14:17:23 pxhost1 pmxcfs[3443]: [status] crit: cpg_initialize failed: 2

The corosync service does not come up:
Nov 22 14:02:03 pxhost1 corosync[3657]: warning [MAIN ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Nov 22 14:02:03 pxhost1 corosync[3657]: warning [MAIN ] Please migrate config file to nodelist.
Nov 22 14:02:03 pxhost1 corosync[3657]: error [MAIN ] parse error in config: Multicast address family does not match bind address family
Nov 22 14:02:03 pxhost1 corosync[3657]: error [MAIN ] Corosync Cluster Engine exiting with status 8 at main.c:1416.

It seems that the corosync .conf is not compatible with corosync which is included in 5.2.11 ... my config looks like that:
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: pxhost1
nodeid: 3
quorum_votes: 1
ring0_addr: pxhost1
ring1_addr: pxhost1pm
}

node {
name: pxhost2
nodeid: 1
quorum_votes: 1
ring0_addr: pxhost2
ring1_addr: pxhost2pm
}

node {
name: pxhost5
nodeid: 2
quorum_votes: 1
ring0_addr: pxhost5
ring1_addr: pxhost5pm
}

}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: yyyyy-proxmox4
config_version: 4
ip_version: ipv4
rrp_mode: passive
secauth: on
version: 2
interface {
bindnetaddr: 111.222.333.121
ringnumber: 0
}
interface {
bindnetaddr: 10.0.99.0
ringnumber: 1
}
}

What can I do now?
On the cluster offline member I cannot change the file as root... it is read only and locked.

 

[Jospeh Huber]

Here is described how to change corosync.conf a stale member;
https://pve.proxmox.com/wiki/Separate_Cluster_Network#Write_config_when_not_quorate

But how to change?

 

[Jospeh Huber]

There was change in corosync 2.4.3...
https://github.com/corosync/corosync/wiki/Corosync-2.4.3-Release-Notes
totemconfig: Prefer nodelist over bindnetaddr

I think this will break ALL ProxMox default corosync configurations prior to 2.4.3 ...

 

[t.lamprecht]

There was change in corosync 2.4.3...
https://github.com/corosync/corosync/wiki/Corosync-2.4.3-Release-Notes
totemconfig: Prefer nodelist over bindnetaddr

I think this will break ALL ProxMox default corosync configurations prior to 2.4.3 ...

We have this on mutliple clusters here (new and old ones, created by default) without issues, so no, it won't break in general.

I'll investigate though. How does your network setup looks like, i.e.:

ip addr

output?

Also did you once changed addresses after setup? We now also default to write out the resolved ip address not the nodename, so this hould be less of an issue with newer created clusters.

 

[t.lamprecht]

Nov 22 14:02:03 pxhost1 corosync[3657]: error [MAIN ] parse error in config: Multicast address family does not match bind address family

This suggests that one of your ringX hostnames resolves to IPv6 (only??)? Strange.

 

[Jospeh Huber]

Hi,

I found the problem:
ring1_addr: pxhost1pm
ring1_addr: pxhost2pm
...

The ring1_addrs must be a reslovable hostname or an IP ... prior to 2.4.3 this is not an issue and was working.
If found it, because the newer Proxmox installations use the IP there and not the hostname.
After adding the entries to /etc/hosts and "service corosync restart" it was working again!

Thx

 

[t.lamprecht]

OK, yes that's really needed, we even do some checks (since 5.1 or 5.2 not sure anymore) on create and node join if the passed ringX_addresses are resolvable and configured, so for newer setups this should get detected on setup.

prior to 2.4.3 this is not an issue and was working.

it was an issue, it works only if you have one IP address configured per given bindnet subnet only, else it worked by luck (which one was selected on startup for binding to).