[SOLVED] Strange behaviour

[LnxBil]

I encountered a strange behaviour in my cluster with respect to VM firewalling.

I created a pve node iptables rule to redirect traffic from 443 to 8006, which works great for

• all external clients
• VMs hosted on the cluster without firewall
• VMs with firewall enabled that are not running on the node I want to access
• BUT NOT on VMs with firewall enabled that are running on the node I want to access

If I check the firewall box on the interface configuration, the connection to the redirected service is not available. If I uncheck the box, it is working as expected.

I could replicate this by just using one PVE VM:

• install PVE 7 and update to current non-subscription repository and reboot (just in case it has been already fixed)
• add iptables rule iptables -t nat -A PREROUTING -d <pve-host-ip> -p tcp --dport 443 -j REDIRECT --to-port 8006
• create container of your choice (I used alpine) and check the firewall box
• enable datacenter firewall (accept all)
• enable container firewall (accept all)
• curl https://<pve-host-ip>/ from the outside, works
• curl https://<pve-host-ip>:8006/ from the container, works
• curl https://<pve-host-ip>/ from the container, does not work

So what's going on there and how can I fix it?

 

[shrdlicka]

Let me just ask if you VM machine is behind a NAT or is it just a normal linux brdige?


It sounds a bit like PREROUTING chain is not used if the packet originates locally [1], but I'm not sure if this is the case with packets from the VM.


[1] https://stuffphilwrites.com/wp-content/uploads/2014/09/FW-IDS-iptables-Flowchart-v2019-04-30-1.png

 

[LnxBil]

Let me just ask if you VM machine is behind a NAT or is it just a normal linux brdige?

Yes, normal bridge, nothing fancy.

It sounds a bit like PREROUTING chain is not used if the packet originates locally [1], but I'm not sure if this is the case with packets from the VM.

The problem is that if firewalling is enabled, you will have more interfaces besides the tap<VMID> and you will have more chains/tables the package travells through. If there is not firewall, it works, so it cannot be the default PREROUTING stuff. Any PVE-addon on top of the networking changes the behaviour, but I do not know what.

Were you able to reproduce the problem?

 

[shrdlicka]

I could reproduce it. Took some time to figure this out a colleague of mine found the solution. You need to replace the above redirect statement with this:

iptables -t nat -A PREROUTING -d [host] -p tcp --dport 443 -j DNAT --to [host]:8006

 

[LnxBil]

Thank you very much, it solves the problem indeed. Thank your colleague, too!

I was not aware of the difference between REDIRECT and DNAT.

 

[shrdlicka]

A bit of background:
see man iptables-extension

Under "REDIRECT" it states:

...
and packets arriving on interfaces that don't have an IP address configured are dropped.
...

Which will always be the case since we have some interfaces without IPs in the setup. In particular this will be the "tap" device.