Dismantled cluster, cannot access webUI from different subnet

[fubacca]

I had a 2 node cluster, with a quorum device running on a third standalone node. I dismantled the cluster, all vms are running fine and I can access the web UI from systems on the same subnet as the old cluster nodes. I have a jumphost in a different subnet that I use for remote management and I am unable to get to the web UI of the 2 hosts, but can reach the standalone node no problem. The Sophos firewall is not blocking the traffic and I can reach the 3rd node fine. I can SSH and ping all nodes from the jumphost fine as well.

I have verified that the prox firewall is disabled on all nodes

I have verified that the gateway is configured on all nodes.

I have verified the Sophos XG firewall routing between the subnets is allowing the traffic, the same firewall rul is used for access to node 3 as the previously clustered nodes and the same rule applies to ssh access and that is working as well.

The network config file is identical between the 2 nodes

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
        mtu 9000

auto eno2
iface eno2 inet manual
        mtu 9000

iface usb0 inet manual

auto enp129s0f0
iface enp129s0f0 inet manual
        mtu 9000

auto enp129s0f1
iface enp129s0f1 inet manual
        mtu 9000

iface enxbe3af2b6059f inet manual

auto bond0
iface bond0 inet static
        address 192.168.0.91/24
        gateway 192.168.0.1
        bond-slaves eno1 enp129s0f0
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp129s0f0
        mtu 9000

auto bond1
iface bond1 inet manual
        bond-slaves eno2 enp129s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp129s0f1
        mtu 9100

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9100

 

[longer]

Does it make a difference if you restarted pveproxy and pvedaemon?
systemctl restart pveproxy pvedaemon
If thats doesnt help as well, check for residual iptables rules that may have been added during cluster setup and are blocking port 8006 from external subnets.

Also ensure the clean up part after cluster dismantled is fully performed:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cleanup_steps_after_node_removal

 

[fubacca]

Yes I have restarted those services on all nodes in question and I used the official guide to break the cluster and cleanup the residual configs and settings. iptables is clean on all nodes, no configuration at all.

Thanks for the reply, I have been fighting this for a few days now and just cant seem to find anything that has been helpful. One node would be easy to reload. the other would require a considerable effort of backups and restores to move the vms over once I reload the 1st node that isnt running anything right now.