Proxmox cluster split after FritzBox Update

[fredd24]

Hi everyone,

I'm troubleshooting a two-node Proxmox cluster with a qdevice and have run out of ideas.

Environment​

• Proxmox VE 9.2.5
• corosync 3.1.10-pve3
• 2 cluster nodes:
  • node-a (site A)
  • node-b (site B)
• qdevice/qnetd on a third device (raspberry pi in same network as node-a)
• Sites connected via a site-to-site WireGuard VPN on two Fritz-Box routers. The "home"-router was recently updated to a new version (8.25), the "remote"-box is at 8.20.
• Cluster was stable for a long time
• Problem appeared shortly after a router firmware update on one site

Current behavior​

Both nodes can:

• ping each other
• SSH into each other
• communicate with the qdevice

However, Corosync never establishes a node-to-node connection.

On node-a:

corosync-cfgtool -s

shows:

nodeid: 1: localhost
nodeid: 2: disconnected

On node-b:

corosync-cfgtool -s

shows:

nodeid: 1: disconnected
nodeid: 2: localhost

Corosync logs​

Repeatedly seeing:

[KNET ] host: host: 1 has no active links

Previously I also saw:

[KNET ] pmtud: possible MTU misconfiguration detected.kernel is reporting MTU: 1500 bytes for host 1 link 0 but the other node is not acknowledging packets of this size.

followed by:

[TOTEM] Token has not been received[TOTEM] token timed out

MTU testing​

From node-b:

ping -M do -s 1472

returns:

Frag needed and DF set (mtu = 1392)

while:

ping -M do -s 1300

works correctly.

So the effective VPN path MTU appears to be around 1392 bytes.

Corosync configuration changes​

I changed:

ip_version: ipv4-6

to:

ip_version: ipv4

and added:

interface {linknumber: 0knet_mtu: 1300}

The PMTU warning disappeared after that, but the nodes still do not connect.

Connectivity tests​

UDP connectivity works.

On node-b:

nc -u -l -p 5405

On node-a:

echo test5405 | nc -u 5405

So the UDP packet arrives on port 5405 which should be the right one for corosync as well, right?

Tcpdump observations​

On node-b:

tcpdump -ni any udp port 5405

I only see outgoing Corosync packets:

Out IP :5405 > :5405

I never see incoming Corosync packets.

On node-a I see both inbound and outbound Corosync packets.

This is what confuses me most, because manual UDP tests on the same port work.

Other checks​

• No iptables rules
• No nftables rules
• Identical /etc/corosync/authkey hashes on both nodes
• Same Proxmox versions
• Same Corosync versions
• qdevice reachable

Corosync runtime values:

corosync-cmapctl | grep knet_mtu

returns:

runtime.config.totem.knet_mtu (u32) = 0totem.interface.knet_mtu (str) = 1300

Qdevice behavior​

On node-a:

Quorate: Yes
Flags: Quorate Qdevice

On node-b:

Quorate: No
Flags: Qdevice

Node-b shows:

A,NV,NMW

which suggests the qdevice is connected but is not granting its vote to the isolated partition.

Question​

Has anyone had a similar problem?

Any suggestions would be greatly appreciated.

Thanks!

 

[Johannes S]

Corosync expect quite low latency ( <5ms ) see https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network

Achieving this over wan is quite difficult if not impossible. Which usecase do you need a cluster for? Maybe the Datacenter-Manager and/or pve-zsnc are a bitter fit for your needs.