VMs lose outgoing traffic after enabling the firewall

[pawelke]

Hi,
I have a Proxmox server (8.4.13) where WAN interface is connected directly to the Internet, and on one of the virtual machines I have OPNSense running as a router for other virtual machines. In the Proxmox network configuration, all traffic except ports 22 and 8006 is routed to the VM with OPNSense. This is what the configuration looks like:

auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address [...]/31
        gateway [...]
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        post-up sysctl -w net.ipv4.ip_forward=1
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.0.16.1
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -j DNAT --to 10.0.16.1

auto vmbr1
iface vmbr1 inet static
        address 10.0.16.0/31
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A POSTROUTING -s '10.0.16.1/31' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.16.1/31' -o vmbr0 -j MASQUERADE

#OPNSense WAN - Proxmox LAN
auto vmbr2
iface vmbr2 inet manual
        ovs_type OVSBridge

#VM Net
auto vmbr3
iface vmbr3 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

I set up a firewall on OPNSense according to my needs (VPN, relevant services available from the Internet, GUI services available only from selected IPs). Everything has been working perfectly so far.

I decided to add an additional firewall for ports 22 and 8006 directly on Proxmox.
My host.fw file currently looks like this:

[OPTIONS]

enable: 0

[RULES]

OUT ACCEPT -log nolog
IN ACCEPT -source dc/office -p tcp -dport 8006 -log warning
IN ACCEPT -source dc/office -p tcp -dport 22 -log warning
IN DROP -p tcp -dport 8006 -log warning
IN DROP -p tcp -dport 22 -log warning
IN ACCEPT -log nolog
IN DROP -log warning

After enabling the firewall on Proxmox and in the Datacenter, the rules work as they should—the appropriate external traffic is visible on the VM, and ports 22 and 8006 are only accessible from the selected IP.
However, no outgoing traffic from the VM works. I cannot ping anything. In Datacenter, I set the outgoing firewall to ACCEPT, but it didn't help. Outgoing traffic is blocked even if I don't enable the firewall on the Proxmox node, but only at the Datacenter level. What's more, disabling the firewall on Datacenter doesn't help - traffic is not restored. Only rebooting the server helps.

So I have some questions:
- Why does outgoing traffic only return after rebooting the server? Can I do this faster by restarting/disabling any service?
- How do I configure the firewall correctly so that outgoing traffic works when firewall is enabled?

 

[Onslow]

Hi, pawelke, welcome to the Forum!

I don't know the answers to your questions even after rereading the proper chapter in the PVE Administration Guide.
To be honest, I find this chapter misleading and insufficient in many places. E.g. the syntax given is:

"Firewall rules syntax
[RULES]
DIRECTION ACTION [OPTIONS]
...
--source <string>"

and in the Examples one page below we read:

"IN SSH (ACCEPT) -i net0 -source 10.0.0.1-10.0.0.10"

That is, in the syntax definition there is ACTION without parenthesis, but in the Example there is (ACCEPT) in parenthesis.
And that SSH word before (ACCEPT) is not present in the syntax definition. I find the example not resulting from the definition.

Next: options in the definition are prefixed with -- (two dashes), but in the Example there is - (one dash).

If I was to use PVE firewall, I would be made to use "try and error" method :-(.

Anyway, let's get back to your issue.
The Guide is advising:

"If you want to see the generated iptables rules you can use:
# iptables-save"

and it is one of my advices, as well. I would add -c (for counters).

Personally, I usually use

iptables -L -v -n
iptables -t nat -L -v -n

then provoke some (failing) traffic and observe the counters - in which rule the counter is increasing.

I'm curious if you will have found a solution. Anyway, please post your findings because they can help future readers .