[SOLVED] One node in a cluster report "Host administratively prohibited" for all ports except SSH

[René Pfeiffer]

Hello!

I have a cluster with three nodes. The cluster was upgraded from Proxmox 5.4 to 6.1 yesterday. The nodes have a dedicated WAN, LAN, and Corosync link. The LAN is flat, there are no internal firewall configured. The interfaces configuration on the hosts look like this:

Host A

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address a.b.c.d …

auto eth1
iface eth1 inet manual

#Cluster Link
auto enx00116b684cd8
iface enx00116b684cd8 inet static
address 10.11.12.8
netmask 255.255.255.0

auto vmbr0
iface vmbr0 inet static
address 10.11.15.254
netmask 255.255.255.0
bridge-ports eth1
bridge-stp off
bridge-fd 0

Host B

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address a.b.c.e …

auto eno2
iface eno2 inet manual

# Cluster Link
auto enx00116b6849e2
iface enx00116b6849e2 inet static
address 10.11.12.9
netmask 255.255.255.0

auto vmbr0
iface vmbr0 inet static
address 10.11.15.253
netmask 255.255.255.0
bridge-ports eno2
bridge-stp off
bridge-fd 0

Host C

auto lo
iface lo inet loopback

allow-hotplug enp97s0f0

auto enp97s0f0
iface enp97s0f0 inet static
address a.b.d.f …

iface eno1 inet manual

iface eno2 inet manual

auto enp97s0f1
iface enp97s0f1 inet static
address 10.11.12.10
netmask 24

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
address 10.11.15.251
netmask 24
bridge-ports bond0
bridge-stp off
bridge-fd 0

Cluster link is established via the 10.11.12.0/24 network. Host A and B can communicate freely on 10.11.15.0/24. On Host C I can only reach SSH ports (except for the local machine). All other ports in 10.11.15.0/24 on VMs on Host C or vice versa except SSH trigger a ICMP Destination unreachable (Host administratively prohibited) message. The host firewall allows all traffic on the LAN bridges. I have not seen any rules with iptables, ebtables, ipset, or nftables. The ICMP error message is definitely created by Host C. Traceroutes from A and B look like this:

traceroute to 10.11.15.251 (10.11.15.251), 30 hops max, 60 byte packets
1 10.11.15.251 (10.11.15.251) 0.251 ms !X 0.196 ms !X 0.174 ms !X

Hosts A and B can talk to each other. Where does the ICMP Destination unreachable message come from?

Best regards,
René.

 

[Stoiko Ivanov]

ICMP Destination unreachable (Host administratively prohibited) message.

hm - in my experience the (Host administratively prohibited) messages are usually generated by some kind of filtering where packets are rejected rather than dropped
* do the nodes themselves have iptables configured? (check with `iptables -nvL` on each host)
* maybe some ACLs on the switch? (e.g. i think I remember that cisco switches replied with this kind of icmp packet if something was denied by an acl configured on them)
* check the mac-addresses and ip addresses in tcpdump - does everything match up?

I hope this helps!

 

[René Pfeiffer]

I doubled and tripled checked this. Only one host hat a firewall (for the WAN side). The others are filtered by a switch (also on the WAN side). The switch on the LAN ports has no ACLs. I verified the source of the to be Host C (I compared the MAC addresses from a tshark dump). The Proxmox cluster itself doesn't use firewalling (via the UI), some VMs have individual filters, but all filtering on Host C shows no rules (that's what I checked with the various filtering tools available). Strangely Host C can connect to its own addresses just fine.

Gateways are also not involved, because it's all LAN traffic.

I am now checking /etc for the latest changes. Connectivity worked with Proxmox 5.4, so I suspect something changed (obviously) during the upgrade.

Best regards,
René.

 

[Stoiko Ivanov]

* hmm - does it work if you remove the 'balance-rr' bond on host c and use only one interface?
* if this works - does it work with an 'active-backup' bond?

 

[René Pfeiffer]

I tried the bond mode, but the message stays the same. I inspected the filter code again. ICMP Host administratively prohibited is only created by the kernel network filter code. It turned out that the Host C had firewalld installed. This was an unintended consequence of a firewall deployment. This change was intended for a later stage, but somehow firewalld got installed and configured the LAN as public zone. This explains the ICMP messages. Reconfiguring did not work, because firewalld did not touch the rules configured in nftables (the configuration file had the iptables option turned on). I had to manually clear the nftables rules, change the LAN to the trusted zone, and restart firewalld. I also added the Corosync network to the trusted zone. This seems to have fixed the problem.

I suspect the firewalld package and our accidental installation of this package is the culprit here. The issue has nothing to do with the Proxmox oder Debian distribution upgrade. If you intend to use firewalld, make sure you edit the configuration file before deploying live rules (and possible clear all existing rule sets when firewalld is stopped). Or you use the Proxmox firewall configurations.

Best regards,
René.

 

[Stoiko Ivanov]

I suspect the firewalld package and our accidental installation of this package is the culprit here.

sounds like a very likely reason.

Thanks for reporting back with your solution - please mark the thread as 'SOLVED' - this helps others who are facing similar problems.