Enabling Firewall breaks connection to all VMs

[Dark18]

Hello,

I have a proxmox server running 7.4-13.
When I try to enable the firewall on datacenter level, all connections to my VMs break and I can't ping them. Even those VMs which have no firewall enabled in VM Firewall and Hardware settings. Also my NFS mount from my TrueNAS VM breaks.

Disabling the firewall and restarting the firewall with pve-firewall restart does not help.
Only restarting the entire server fixes the issue.

I have not set any rules and tried to set Input/Output policy to "ACCEPT". Also I tried to enable the firewall on node level.
I do have some custom iptables and ufw rules, but I also tried to disable them, which did not work either.

I enabled logging, but no logs are created.

There is a post that suggests not installing docker, since it messes with proxmoxs iptabes rules. But there is not docker installed on my host system.

Can anyone help me out?
Thanks in advance!

 

[mira]

Please provide the output of the following commands:
cat /etc/pve/firewall/cluster.fw
cat /etc/pve/firewall/host.fw

The IPs of the VMs you lost connection to, and the source IP from which you connected.

Both VM and NIC firewalls are disabled?


Please also provide your custom iptables and ufw rules.

 

[Dark18]

Please provide the output of the following commands:
cat /etc/pve/firewall/cluster.fw
cat /etc/pve/firewall/host.fw

The file host.fw does not exist. Even if I try to add host based rules, the file is not created while the rules appear in the UI.
In the directory, there are only cluster.fw and 10*.fw

The IPs of the VMs you lost connection to, and the source IP from which you connected.

Proxmox host: 192.168.86.1 (vmbr1), 192.168.99.4 (wg0)
VMs: 192.168.86.6, 192.168.86.8

Here is the content of cluster.fw:

[OPTIONS]

policy_in: ACCEPT
ebtables: 0
enable: 0

[RULES]

OUT ACCEPT -log info
IN ACCEPT -log info

Here is the content of 102.fw which belongs to 192.168.86.6:

[OPTIONS]

policy_in: ACCEPT

The IPs of the VMs you lost connection to, and the source IP from which you connected.

The VMs are connected to vmbr1, where the proxmox host has the IP 192.168.86.1.
The VMs start at 192.168.86.2.

I did some further testing with following results:

• Ping works from host (.86.1) to any VM (.86.6, .86.8) and also from any VM to any VM.
• Telnet/SSH connections work only from host to any VM. Telnet between VMs does not work.
• Telnet/SSH from my wireguard interface to any VM and host works

So there is something wrong with the communication between the VMs.
For the tests, I disabled all firewall options and enabled only the Datacenter one.

Here is the interface config (gateway is pfSense on .86.2)

auto vmbr1
iface vmbr1 inet static
    address 192.168.86.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0

The iptables rules forward the traffic from my wireguard interface to the VMs:

-A PREROUTING -i wg0 -p tcp -m tcp --dport 2223 -j DNAT --to-destination 192.168.86.6:22
-A POSTROUTING -o eno1 -j MASQUERADE
-A POSTROUTING -j MASQUERADE

The ufw rules only block outside access from LAN (eno1), since I do remote management via wireguard.

Both VM and NIC firewalls are disabled?

Yes, after enabling I tried to disable all firewalls: datacenter, proxmox host, VM settings, VM NIC

 

[mira]

Sorry, the host.fw is actually under /etc/pve/nodes/<NODE>/host.fw.
Could you try setting nf_conntrack_allow_invalid: 1 in the host.fw? This is sometimes required for communication between VMs since they share a conntrack when located on the same host.


So everything works with the current firewall settings except for VM <-> VM communication?

 

[Dark18]

Sorry, the host.fw is actually under /etc/pve/nodes/<NODE>/host.fw.
Could you try setting nf_conntrack_allow_invalid: 1 in the host.fw? This is sometimes required for communication between VMs since they share a conntrack when located on the same host.

I added the option, enabled the host firewall and then the datacenter firewall and also rebooted.
Unfortunately, it didn't change anything.

So everything works with the current firewall settings except for VM <-> VM communication?

Exactly. Ping does work, but any TCP connections don't.


When disabling ufw now, everything works.
So I deleted all of my ufw and iptables rules and set the default policy to accept using ufw default allow followed by ufw enable and ufw reload. This configuration should allow any connections.

But TCP communication between VMs is still not possible.
I also tried adding allow rules like ufw allow from 192.168.86.0/24 to 192.168.86.0/24 but this didn't change anything.


Could there be any interference between the firewall and the blank ufw configuration?
Also, thank you for helping!

 

[mira]

I'd suggest disabling it for now.

Does the connection between VMs work if you disable all firewalls?

Please provide the output of iptables-save.
And I'd suggest starting to use tcpdump on the host and both VMs to see where the packets are seen.
You can use tcpdump -envi <iface> -s0 -w $(hostname).pcap where <iface> is the interface the traffic goes through. In this case it would be `vmbrX` on the host and `ensXY` in your Linux guests.

 

[Dark18]

Does the connection between VMs work if you disable all firewalls?

Yes it works fine. Until I enable datacenter firewall once. After that, disabling the datacenter firewall has no effect and only a reboot restores the communication.

Please provide the output of iptables-save.

As the issue also occurs with no specific ufw and iptables rules set, here is the output with the default rules:
https://pastebin.com/raw/exUk2n7L

I will try tcpdump later.