[SOLVED] Cluster-Wide Firewall Rules Not Working on One Node

[mihanson]

I have an odd issue with my cluster-wide firewall that I can't seem to figure out. I want
to limit Proxmox GUI access to one network (192.168.10.0/24) on all nodes. I've been able
to accomplish this on 5 of my 6 nodes, but don't understand why I can't make it work on the
final node.

I have 6 nodes in my cluster. 3 nodes are used as low-power ceph monitors (mon01, mon02,
mon03) and the other 3 nodes (pve01, pve02, pve03) are where all my storage, containers
and VMs live. pve01, pve02, pve03 each have 4 networks:
192.168.10.0/24 # Main management network
192.168.70.0/24 # Ceph Network
192.168.80.0/25 # Primary Corosync Network
192.168.80.128/25 # Backup Corosync Network

mon01, mon02, mon03 have 3 networks:
192.168.10.0/24 # Main management network
192.168.80.0/25 # Primary Corosync Network
192.168.80.128/25 # Backup Corosync Network

Here is an example of /etc/network/interfaces on pve01:

mihanson@pve01:~$ cat /etc/network/interfaces

auto lo
iface lo inet loopback

auto enp6s0
iface enp6s0 inet static
address 192.168.80.***/25
mtu 9000
#Corosync Backup

iface enp1s0f0 inet manual
mtu 9000
#Bridge Interface

auto enp1s0f1
iface enp1s0f1 inet static
address 192.168.70.***/24
mtu 9000
#Ceph Network

auto enp5s0
iface enp5s0 inet static
address 192.168.80.***/25
mtu 9000
#Corosync Primary

iface enp1s0f0.10 inet manual
mtu 9000
#VLAN 10

auto vmbr0
iface vmbr0 inet manual
bridge-ports enp1s0f0
bridge-stp off
bridge-fd 0
mtu 9000
#Untagged

auto vmbr10
iface vmbr10 inet static
address 192.168.10.***/24
gateway 192.168.10.***
bridge-ports enp1s0f0.10
bridge-stp off
bridge-fd 0
mtu 9000
#VLAN 10 Bridge

Here is an example of /etc/network/interfaces on a ceph-mon (mon01):

mihanson@mon01:~$ cat /etc/network/interfaces

auto lo
iface lo inet loopback

auto enp3s0
iface enp3s0 inet static
address 192.168.80.***/25
mtu 9000
#COROSYNC - PRIMARY

auto ens1
iface ens1 inet static
address 192.168.10.***/24
gateway 192.168.10.***
mtu 9000
#MANAGEMENT NET

auto enxc025e92941a9
iface enxc025e92941a9 inet static
address 192.168.80.***/25
mtu 9000
#COROSYNC - BACKUP - USB

The cluster-wide firewall is turned ON. The node firewalls are all turned ON. The node level
firewalls have NO rules defined. I have the following cluster-wide firewall rules:

mihanson@pve01:~$ sudo cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 1

[ALIASES]

LAN 192.168.1.0/24 # Untagged Network
CEPH_Network 192.168.70.0/24 # VLAN 70
Corosync_Primary 192.168.80.0/25 # VLAN 80
Corosync_Backup 192.168.80.128/25 # VLAN 81
VoIP 192.168.50.0/29 # VLAN50
Restricted 192.168.20.0/24 # VLAN20
Unrestricted 192.168.10.0/24 # VLAN10
Guest 192.168.100.0/27 # VLAN100
AV 192.168.30.0/28 # VLAN30
IoT 192.168.40.0/25 # VLAN40

[IPSET management] # IPs that can manage the cluster

192.168.10.*** # pc 1
192.168.10.*** # pc 2
192.168.10.*** # pc 3
192.168.10.*** # pc 4

[RULES]

IN ACCEPT -i vmbr10 -source unrestricted -dest unrestricted -p tcp -dport 3551 -log nolog # Allow APCUPSd
IN ACCEPT -source ceph_network -dest ceph_network -p tcp -dport 6800:7300 -log nolog # Allow Ceph OSD
IN ACCEPT -dest corosync_backup -p udp -dport 5405:5406 -log nolog # Allow Corosync Traffic
IN ACCEPT -dest corosync_primary -p udp -dport 5405:5406 -log nolog # Allow Corosync Traffic
IN Ceph(ACCEPT) -dest unrestricted -log nolog # Accept Ceph Monitor Traffic
IN REJECT -dest ceph_network -p tcp -dport 8006 -log nolog # Reject web traffic on Ceph network
IN REJECT -dest corosync_backup -p tcp -dport 8006 -log nolog # Reject web traffic on Corosync network
IN REJECT -dest corosync_primary -p tcp -dport 8006 -log nolog # Reject web traffic on Corosync network

With the cluster-wide firewall turned OFF I can access the Proxmox web GUI on any of the
6 nodes via https://192.168.{10,70,80}.xyz:8006. I would like to only be able to access
the GUI via https://192.168.10.xyz:8006. When I turn the cluster-wide firewall ON, I am able
to accomplish this on mon01, mon02, mon03, pve02, pve03. For some reason, the cluster-wide
REJECT rules are not working on pve01. All other ACCEPT rules are working on pve01. Can
anyone help me figure out why?

 

[fabian]

I'd look at the generated rules (pve-firewall compile / iptables-save). output of pve-firewall localnet might also be interesting

 

[mihanson]

I'd look at the generated rules (pve-firewall compile / iptables-save). output of pve-firewall localnet might also be interesting

Thank you! It looks like I hit the limit on characters, so I've attached the output as txt files.

 

[fabian]

yeah - could you repeat that for all the nodes WITH the firewall enabled?

 

[mihanson]

pve01: here (This is the node that doesn't block the 70, 80 subnet)
pve02: here
pve03: here
mon01: here
mon02: here
mon03: here

To my uneducated eye, it seems that iptables-save is empty on pve01, but it is not on the other nodes.

 

[mihanson]

I just figured this issue out. For some reason, on the node where the firewall rules were set, but not working, the pve-firewall was enabled, but not running. I don't know how it got into this enabled, but not runnng state, but once I started it on the CLI, all is well and the rules are working on this node as expected.

root@pve01:~# pve-firewall status
Status: enabled/stopped
root@pve01:~# pve-firewall start
root@pve01:~# pve-firewall status
Status: enabled/running