I have an odd firewall glitch I hope to figure out a workaround for.
I'm deploying a 5 node cluster to be used in a live production environment. I've configured a test VM which is set for high availability. I added a firewall to the guest in proxmox which appears to work as expected. It has a default drop policy and allows only certain IPs through. When I ping the VM from an external host I'm blocked as expected.
The issue crops up when I migrate the VM to another node. If I have a ping running against the VM's IP it's blocked until the migration completes, then I start getting pings back from the guest. I will continue to get pings back until I stop the pings for a few seconds. Once I stop and restart a ping against the VM it will again be blocked as expected.
I haven't dug heavily into the guts of the firewall in Linux on the node yet, but I'm assuming this is a connection tracking issue where the pings are being granted through on the new node under the more permissive cluster firewall rules, and the connection tracking is keeping that hole open after the migration because the track already exists.
When I look at the output of conntrack on the target node I do see a valid connection for the icmp session in there and I'm assuming that is how this failure is creeping in.
Expected behavior would be the guest firewall to be active on the new node before the guest network interface is brought up. I'm concerned the current behavior would allow a malicious attacker to potentially exploit this during a fail over or migration to punch holes through the firewall.
I've been able to replicate this. To reproduce configure the cluster-wide firewall to permit icmp, set a guest firewall in proxmox to block all traffic except from a whitelist of IPs. Start a ping from a blocked machine to the IP on the VM, then migrate the VM to another node. In the middle of the migration you'll start getting pings and will continue to get pings as long as you keep the ping running. Once you stop the ping and restart it you'll be blocked as expected.
My cluster is running Proxmox 9.1.4.
Thanks!
I'm deploying a 5 node cluster to be used in a live production environment. I've configured a test VM which is set for high availability. I added a firewall to the guest in proxmox which appears to work as expected. It has a default drop policy and allows only certain IPs through. When I ping the VM from an external host I'm blocked as expected.
The issue crops up when I migrate the VM to another node. If I have a ping running against the VM's IP it's blocked until the migration completes, then I start getting pings back from the guest. I will continue to get pings back until I stop the pings for a few seconds. Once I stop and restart a ping against the VM it will again be blocked as expected.
I haven't dug heavily into the guts of the firewall in Linux on the node yet, but I'm assuming this is a connection tracking issue where the pings are being granted through on the new node under the more permissive cluster firewall rules, and the connection tracking is keeping that hole open after the migration because the track already exists.
When I look at the output of conntrack on the target node I do see a valid connection for the icmp session in there and I'm assuming that is how this failure is creeping in.
Expected behavior would be the guest firewall to be active on the new node before the guest network interface is brought up. I'm concerned the current behavior would allow a malicious attacker to potentially exploit this during a fail over or migration to punch holes through the firewall.
I've been able to replicate this. To reproduce configure the cluster-wide firewall to permit icmp, set a guest firewall in proxmox to block all traffic except from a whitelist of IPs. Start a ping from a blocked machine to the IP on the VM, then migrate the VM to another node. In the middle of the migration you'll start getting pings and will continue to get pings as long as you keep the ping running. Once you stop the ping and restart it you'll be blocked as expected.
My cluster is running Proxmox 9.1.4.
Thanks!