Dismantled cluster, cannot access webUI from different subnet

[fubacca]

I had a 2 node cluster, with a quorum device running on a third standalone node. I dismantled the cluster, all vms are running fine and I can access the web UI from systems on the same subnet as the old cluster nodes. I have a jumphost in a different subnet that I use for remote management and I am unable to get to the web UI of the 2 hosts, but can reach the standalone node no problem. The Sophos firewall is not blocking the traffic and I can reach the 3rd node fine. I can SSH and ping all nodes from the jumphost fine as well.

I have verified that the prox firewall is disabled on all nodes

I have verified that the gateway is configured on all nodes.

I have verified the Sophos XG firewall routing between the subnets is allowing the traffic, the same firewall rul is used for access to node 3 as the previously clustered nodes and the same rule applies to ssh access and that is working as well.

The network config file is identical between the 2 nodes

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
        mtu 9000

auto eno2
iface eno2 inet manual
        mtu 9000

iface usb0 inet manual

auto enp129s0f0
iface enp129s0f0 inet manual
        mtu 9000

auto enp129s0f1
iface enp129s0f1 inet manual
        mtu 9000

iface enxbe3af2b6059f inet manual

auto bond0
iface bond0 inet static
        address 192.168.0.91/24
        gateway 192.168.0.1
        bond-slaves eno1 enp129s0f0
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp129s0f0
        mtu 9000

auto bond1
iface bond1 inet manual
        bond-slaves eno2 enp129s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp129s0f1
        mtu 9100

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9100

 

[longer]

Does it make a difference if you restarted pveproxy and pvedaemon?
systemctl restart pveproxy pvedaemon
If thats doesnt help as well, check for residual iptables rules that may have been added during cluster setup and are blocking port 8006 from external subnets.

Also ensure the clean up part after cluster dismantled is fully performed:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cleanup_steps_after_node_removal

 

[fubacca]

Yes I have restarted those services on all nodes in question and I used the official guide to break the cluster and cleanup the residual configs and settings. iptables is clean on all nodes, no configuration at all.

Thanks for the reply, I have been fighting this for a few days now and just cant seem to find anything that has been helpful. One node would be easy to reload. the other would require a considerable effort of backups and restores to move the vms over once I reload the 1st node that isnt running anything right now.

 

[longer]

Np. You can check journalctl/access.log in the destination nodes, while you are attempting to connect from your management node.

 

[steve999]

Quick thought: since SSH and ping work across subnets but port 8006 does not, and the standalone node (which never carried the cluster config) is reachable, I would stop doubting the Sophos and look at what pveproxy itself is doing on the two ex-cluster nodes.

Things I would check, in this order:

1. /etc/default/pveproxy on both nodes. If LISTEN_IP, ALLOW_FROM or DENY_FROM was ever set there (a common way to restrict the GUI to a management subnet), the daemon silently drops connections from sources outside ALLOW_FROM while same-subnet access keeps working. Move the file away, run systemctl restart pveproxy, and test from the jumphost again.

2. pve-firewall status plus nft list ruleset on both nodes. Recent PVE versions use the nftables based proxmox-firewall, so "iptables is clean" can still miss rules living in nftables. Also check whether /etc/pve/firewall/cluster.fw survived the dismantle on each node, since a datacenter level firewall file left behind keeps applying after the cluster is gone.

3. From the jumphost run curl -kv https://node-ip:8006/ while running tcpdump -ni any port 8006 on the target node. If the SYN arrives but the connection stalls or resets, the problem is local to the node (pveproxy config or nft rules). If nothing arrives at all, it is routing after all, despite the shared firewall rule.

4. While attempting to connect, tail /var/log/pveproxy/access.log and journalctl -u pveproxy on the node. pveproxy logs rejected and dropped requests there, which immediately tells you whether the request even reaches the web server.

In my experience a leftover ALLOW_FROM in /etc/default/pveproxy is the sneaky one, because it survives cluster removal and produces exactly this pattern: local subnet fine, remote subnet refused or hanging, SSH unaffected.

 

[fubacca]

@steve999 thank you so much, I am still at a loss why this only crept up and was noticed just now, but apparently I had fat fingered the mtu on the bridge on the node that hosted my jumphost vm, it was set to 9100 and everything else worked fine just not web browser access to that prox node. Thanks for steering me in the right direction, sometimes you get so buried into troubleshooting , another pair of eyes or a nudge off course is all you need.