[SOLVED] Cluster Nodes Grey(?) after Enabling Datacenter Firewall

[anson]

Hi all,

It appears that my nodes are showing a grey question mark after enabling Datacenter's Firewall with default Input Policy being DROP. Other nodes are now inaccessible with this error in node's Summary 'hostname lookup 'pve123' failed - failed to get address info for: pve123: No address associated with hostname (500)'.

I could see Quorate is still showing 'Yes' and I still have a Quorum.

I am trying to revert changes by forcing local mode (pmxcfs -l) to write changes to cluster.fw, but this will not work because changes are not being written to other cluster's members /etc/pve/firewall/cluster.fw

Please advise how can I fix this?

Thank you

root@pve121:~# pvecm status
Quorum information
------------------
Date: Sun Aug 11 12:01:24 2019
Quorum provider: corosync_votequorum
Nodes: 14
Node ID: 0x00000006
Ring ID: 10/70156
Quorate: Yes

Votequorum information
----------------------
Expected votes: 14
Highest expected: 14
Total votes: 14
Quorum: 8
Flags: Quorate

This issue seems to be similar to https://forum.proxmox.com/threads/cluster-loses-quorum-when-activating-firewall.49514/ but I'm not sure how the OP manage to make changes to cluster.fw and propagate the settings in cluster when multicast traffic was dropped

 

[anson]

Finally managed to solve the issue...solution below. Hope this helps someone out there in future.

SOLUTION:
1. Stop both corosync and pve-cluster on all nodes except one.
2. Run pvecm expected 1 and revert cluster firewall settings to 'No' (enable: 0) on the remaining node.
3. Start corosync and pve-cluster on the rest of the nodes (or reboot).