Guest firewall letting blocked packets hit guest VM during fail over

[bobf]

I have an odd firewall glitch I hope to figure out a workaround for.

I'm deploying a 5 node cluster to be used in a live production environment. I've configured a test VM which is set for high availability. I added a firewall to the guest in proxmox which appears to work as expected. It has a default drop policy and allows only certain IPs through. When I ping the VM from an external host I'm blocked as expected.

The issue crops up when I migrate the VM to another node. If I have a ping running against the VM's IP it's blocked until the migration completes, then I start getting pings back from the guest. I will continue to get pings back until I stop the pings for a few seconds. Once I stop and restart a ping against the VM it will again be blocked as expected.

I haven't dug heavily into the guts of the firewall in Linux on the node yet, but I'm assuming this is a connection tracking issue where the pings are being granted through on the new node under the more permissive cluster firewall rules, and the connection tracking is keeping that hole open after the migration because the track already exists.

When I look at the output of conntrack on the target node I do see a valid connection for the icmp session in there and I'm assuming that is how this failure is creeping in.

Expected behavior would be the guest firewall to be active on the new node before the guest network interface is brought up. I'm concerned the current behavior would allow a malicious attacker to potentially exploit this during a fail over or migration to punch holes through the firewall.

I've been able to replicate this. To reproduce configure the cluster-wide firewall to permit icmp, set a guest firewall in proxmox to block all traffic except from a whitelist of IPs. Start a ping from a blocked machine to the IP on the VM, then migrate the VM to another node. In the middle of the migration you'll start getting pings and will continue to get pings as long as you keep the ping running. Once you stop the ping and restart it you'll be blocked as expected.

My cluster is running Proxmox 9.1.4.

Thanks!

 

[bobf]

Just an update. I have confirmed that I can exploit this to establish an SSH session with the guest by simply probing the ssh port every second. As soon as I migrate the VM to another node the session connects and I can use that session as long as I want. As soon as I drop that connection I'm once again locked out.

 

[shanreich]

It's a limitation with how the firewall works currently (periodically updating the ruleset). Possible improvements would be to force a re-generation of the firewall ruleset sometime during the migration or to generate the full firewall ruleset for all VMs on all nodes with every update cycle, so they already exist when VMs are migrated from node to node.

 

[bobf]

Are there any suggestions on how to mitigate this? In production migrations should be infrequent but I'm hesitant to tell customers they're secure behind a firewall when a relatively simple management operation can expose potentially exploitable services, however briefly.

 

[SteveITS]

Interesting, I’d have also expected firewall rules to be applied the whole time.

Can access be controlled by a firewall outside Proxmox? At least for external connections… That’s our primary method though we use Proxmox firewall in a few cases for intra cluster connections, as well as allowing IPs on the VM.