datacenter firewall oddity

[fti-fred]

I have a four node Proxmox 8.3.5 cluster. Just turning on the datacenter firewall for the first time.

Applicable bridges:
vmbr0 default, public network
vmbr2 private VM network

Default firewall options are set.

Rule 1, in, accept from specific public IPs to vmbr0 SSH and GUI at 8006.
works

Rule 2, in, accept vmbr2 from source all to destination all
only works from IPs on the same subnet

Remote subnet works to vmbr0 using public IP.

Remote subnet does not work to vmbr2 via VPN. Firewall log shows the traffic is accepted.

Remote subnet can reach VMs on vmbr2 from the same VPN without issue.

Thoughts?

Thank you

 

[fba]

Would you like to post the content of file /etc/network/interfaces for a better understanding of the setup?
What is the purpose of the connection to vmbr2, do you want to connect to the PVE GUI via IP of vmbr2?

 

[fti-fred]

We have HA firewall VMs with connections to the internet in vmbr0, and the private VM network vmbr2. All of the VMs beside the firewalls are on network vmbr2. We have site-to-site VPNs with direct access to the network on vmbr2.


/etc/network/interfaces

auto lo
iface lo inet loopback

auto enp65s0f0
iface enp65s0f0 inet manual

auto enp65s0f1
iface enp65s0f1 inet manual
mtu 9000

iface eno1 inet manual

iface eno2 inet manual

iface enxbe3af2b6059f inet manual

auto vmbr0
iface vmbr0 inet static
address x.x.x.x/30
gateway x.x.x.x
bridge-ports enp65s0f0
bridge-stp off
bridge-fd 0
#GVLAN 2190

auto vmbr1
iface vmbr1 inet static
address x.x.x.x/24
bridge-ports enp65s0f1.2188
bridge-stp off
bridge-fd 0
mtu 9000
#ceph

auto vmbr2
iface vmbr2 inet static
address x.x.x.x/24
bridge-ports enp65s0f1.2186
bridge-stp off
bridge-fd 0
#private vm

auto vmbr3
iface vmbr3 inet static
address x.x.x.x/24
bridge-ports enp65s0f1.2184
bridge-stp off
bridge-fd 0
#FG HA



/etc/pve/firewall/cluster.fw

[OPTIONS]

enable: 1

[IPSET fti-sc] # ATT and Comcast

x.x.x.x/30
x.x.x.x/26
x.x.x.x/30
x.x.x.x/29
x.x.x.x/26

[IPSET hv-host-ips] # Proxmox Node Public IPs

x.x.x.x/30
x.x.x.x/30
x.x.x.x/30
x.x.x.x/30

[RULES]

IN ACCEPT -i vmbr0 -source +dc/hv-host-ips -log nolog # HV host IPs
IN SSH(ACCEPT) -i vmbr0 -source +dc/fti-sc -log nolog # public internet Proxmox SSH - FTI
IN ACCEPT -i vmbr0 -source +dc/fti-sc -p tcp -dport 8006 -log nolog # public internet Proxmox GUI - FTI
IN ACCEPT -i vmbr1 -log nolog # ceph network
IN ACCEPT -i vmbr2 -log alert # VM private network
IN ACCEPT -i vmbr3 -log nolog # Fortigate HA heartbeat network​

 

[fba]

Sounds like a routing issue to me. Which routes are configured on the host (ip r s)?
Does the host have configured a route to the remote subnet via the vpn endpoint, which is in same subnet like vmbr2? Because if not, host will just use the default gateway and not the intended route via vpn.

 

[fti-fred]

Everything is reachable of if I turn off the firewall, or disable the rules.

 

[fti-fred]

Yep, this is the issue. I don't understand why it works with the firewall turned off since routing is still necessary...maybe ip forwarding...odd
Thank you