losing dirty-bitmaps during Live Migration

[aday]

I'm a recent convert to Proxmox and have been up and running for about a month. I've noticed over time, that moving a VM to a new node almost always causes the next backup to do a full disk read, and with multiple VMs having multiple TBs of storage, this stretches the nightly backup job into days instead of hours. My findings so far indicate this should not be happing and the dirty-bitmap should not be lost during Live Migration. I have had some VMs keep their dirty-bitmaps, but it is hit or miss. Last test case from earlier this week had 4/6 VMs lose dirty-bitmap.

I found this post that seems to backup my idea, dirty-bitmaps should persist during Live Migration: https://lore.proxmox.com/pve-devel/3352a1e6-caf7-0c78-7689-4ef541b99908@proxmox.com/

I've attached a txt file with a log of my troubleshooting and findings so far. Any help would be greatly appreciated!

 

[bbgeek17]

Hi @aday , welcome to the forum.

The journal line Cannot find device='z…' nor node-name='z…' on the target comes from the incoming dirty-bitmap migration: it couldn't find the block node the bitmap was attached to, so QEMU dropped the bitmap and let the migration finish normally. The z… names are PVE's names for a "zeroinit" filter node. PVE adds one to a running VM's disk when the disk is moved online (Move Disk while the VM is running) to storage like NFS, and it stays until the QEMU process is replaced (stop/start or migration). The PBS bitmap ends up attached to that node, and the freshly started QEMU on the target doesn't have it.

I would check whether 145, 156, 188 and 197 had a disk moved online since they were last started, and whether 114/138 did not. If so, the first backup after each move should also show created new. It should then be a one-time loss per VM: once a VM has been migrated or stopped and started, later migrations should keep the bitmap. It still looks like something qemu-server should handle, so once you have that confirmed it may be worth opening a bug at bugzilla.proxmox.com.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox