pve management IP change can't access other nodes communication error

[mbj047]

Hi,
I have a pve v6.2 cluster. There is two interfaces one is the management and the other is for the cluster network. The cluster was running great until I had to change the management IP address subnet. I made no changes to the cluster network. Now when I am logged into a node I can't access the gui of any of the other nodes, I get a communication failure (0) or 596.
There are a lot of posts similar to this but none of the solutions seem to work
Here is what I have done/tried -
Update new subnet / changed /etc/hosts to reflect those changes
rebooted
ssh into each machine from each machine to update ssh keys
ran pvecm updatecerts --force
another reboot

The cluster seem fine
Quorum provider: corosync_votequorum
Nodes: 5
Node ID: 0x00000001
Ring ID: 1.1ed1
Quorate: Yes

Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 5
Quorum: 3
Flags: Quorate

Any suggestions?

Thanks

 

[Moayad]

Hi,

Please post your network config.

 

[mbj047]

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.50.4.150/22
gateway 10.50.4.2
bridge-ports eno1
bridge-stp off
bridge-fd 0
#VM_USE_ONLY

auto vmbr1
iface vmbr1 inet static
address 192.168.40.5/24
bridge-ports eno2
bridge-stp off
bridge-fd 0
#DO_NOT_USE_CLUSTER_ONLY

 

[Moayad]

Please check the resolve file, also check if have you enabled some firewall rules

 

[mbj047]

I did forget to mention I have checked the firewall there isn't anything there since I've never set anything -
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
The resolve.conf -
search jsonar.corp
nameserver 8.8.8.8

The domain name is fictional but it worked before the ip change.

 

[mbj047]

Anymore ideas to check?
Thanks