Live migration seems to temporarily disable the PVE firewall.

[odsrdj]

Hi!

I want to use the PVE firewall to prevent guests from changing their assigned MAC and IP addresses, to prevent spoofing. It works perfectly while the VM is running on a single host, however I've noted that while live migrating the guest, there is a window of about 30 seconds (I guess while the migration is running?) where the network was accessible by the spoofed address (both IPv4 and MAC).

To reproduce:

• Run the VM like below (adjust addresses as necessary)
• Ping something (it should work)
• Inside the guest, change either IP or MAC address.
• Ping it again (it shouldn't work)
• Keep ping running and start a live migration to another node.
• A few pings come through.

Am I doing something wrong or is this a bug in PVE?

My configs are:

# cat /etc/pve/firewall/107.fw
[OPTIONS]
enable: 1
ipfilter: 1
policy_in: DROP
ndp: 0
dhcp: 0

[IPSET ipfilter-net0]
10.0.0.101


# cat /etc/pve/firewall/cluster.fw
[OPTIONS]
policy_in: REJECT
enable: 1

[RULES]
IN ACCEPT -i vmbr0 -log nolog
IN ACCEPT -i vmbr1 -log nolog

[group hosts]


# cat /etc/pve/qemu-server/107.conf
boot: c
bootdisk: scsi0
cipassword: ...
ciuser: root
ide2: ssdpool:vm-107-cloudinit,media=cdrom
ipconfig0: ip=10.0.0.101/24,gw=10.0.10.1
memory: 1536
meta: creation-qemu=6.1.0,ctime=1642660011
name: fw-test-client1
nameserver: 1.1.1.1
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
ostype: l26
scsi0: ssdpool:vm-107-disk-0,size=2252M
scsihw: virtio-scsi-pci
searchdomain: ...
serial0: socket
smbios1: ...
tablet: 0
vga: serial0
vmgenid: ...

My versions are

proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-10
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.7
ceph-fuse: 16.2.7
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-1
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Please let me know if you need my interfaces file or any other information.

Thanks!

Kind regards,
Srdjan

 

[fabian]

yes, there is a small window when moving the VM config from the source to the target node and resuming the VM on the target node both happens between two firewall update actions. the updates happen every 10s, so it should take at most ~10s, not 30. we are currently discussing ways to improve this edge case without negatively affecting migration times.

 

[odsrdj]

Ok, thanks for the info!

The 30 seconds might have been a mistake on my side, I haven't watched the time, but ping reported ~30 answers so I assumed 30s (at the default 1s/ping).

In my case, it would be ok if there were a per VM option to choose between "shortly disable firewall, <1s downtime" (current behavior) and "keep firewall, few seconds downtime", in case it enables an easier solution.

 

[spirit]

I think that if your are running a continuous ping,
the firewall accept the ping during the small windows, and keep it open until you stop it.
(same for tcp connections).
The firewall don't drop already established connections.