LVM snapshot-as-volume-chain: live VM snapshots lose PBS dirty bitmaps and force full disk scans

matraii

New Member
Mar 13, 2025
1
0
1

Summary​

Taking a manual snapshot of a running VM on shared LVM storage with
snapshot-as-volume-chain enabled causes the next PBS backup to report
"dirty-bitmap status: created new" and scan the entire virtual disk.

The VM is not stopped or started again during this workflow. This is not
the known loss of dirty bitmaps after a VM stop/start. The issue occurs
whether the manual snapshot is retained or subsequently deleted.

We previously used local ZFS storage, where taking a snapshot of a running
VM did not cause this behavior.

Environment​

Shared Fibre Channel storage: IBM Storwize V7000, presented to Proxmox VE
through multipath and regular shared LVM (not LVM-thin).
VM disks use qcow2 on LVM with snapshot-as-volume-chain enabled.
Backups are Proxmox VE VM backups to Proxmox Backup Server, using snapshot mode.
Subscription level: Community.

Relevant installed versions from the affected PVE node:
proxmox-ve: 9.2.0
pve-manager: 9.2.21
qemu-server: 9.2.11
pve-qemu-kvm: 11.0.3-4
libpve-storage-perl: 9.1.13
libproxmox-backup-qemu0: 2.0.3
proxmox-backup-client: 4.2.8-1
lvm2: 2.03.31-2+pmx1
Running kernel: 6.17.13-2-pve

The backup-client version above is from the PVE node; it is not a statement
of the PBS server's version. The installed QEMU package version is listed;
the running VM's QEMU version has not been queried separately.

Relevant storage configuration​

lvm: IBM_v7000_48tb
vgname vg_v7000_48tb
content images,rootdir
saferemove 0
shared 1
snapshot-as-volume-chain 1

lvm: IBM_v7000_32tb
vgname vg_v7000_32tb
content rootdir,images
saferemove 0
shared 1
snapshot-as-volume-chain 1

Relevant VM configuration (VMID 246)​

agent: 1
bios: ovmf
machine: q35
efidisk0: IBM_v7000_32tb:vm-246-disk-0.qcow2,efitype=4m,ms-cert=2023w,pre-enrolled-keys=1,size=528K
scsi0: IBM_v7000_48tb:vm-246-disk-0.qcow2,iothread=1,size=60G
scsi2: IBM_v7000_48tb:vm-246-disk-1.qcow2,iothread=1,size=2000G
scsihw: virtio-scsi-pci
parent: bkp

Steps to reproduce/check​

  1. Use a running VM with qcow2 disks on shared LVM storage configured with
    snapshot-as-volume-chain=1.
  2. Complete a PBS backup, then verify that a subsequent backup can use the
    existing dirty bitmap (fast incremental mode).
  3. Take a manual Proxmox VE snapshot while the VM remains running.
  4. Do not stop/start the VM or roll it back.
  5. Run another backup to the same PBS target.
  6. Observe whether the backup reports "dirty-bitmap status: created new"
    and scans the complete disk instead of only changed blocks.
  7. Also check the workflow where the manual snapshot is deleted before
    the next backup; we observe the same full-scan consequence.

Observed production example​

The manual snapshot "bkp" was created on 2026-11-17 at 18:51:45 and was
still present when the following backup ran. Times below are local
Europe/Budapest time (CET, UTC+1).

Selected backup log lines:

2026-11-17 22:33:36 INFO: Starting Backup of VM 246 (qemu)
2026-11-17 22:33:36 INFO: status = running
2026-11-17 22:33:37 INFO: backup mode: snapshot
2026-11-17 22:33:39 INFO: efidisk0: dirty-bitmap status: created new
2026-11-17 22:33:39 INFO: scsi0: dirty-bitmap status: created new
2026-11-17 22:33:39 INFO: scsi2: dirty-bitmap status: created new
2026-11-17 22:33:42 INFO: 0% (776.0 MiB of 2.0 TiB) in 3s, read: 258.7 MiB/s, write: 6.7 MiB/s
2026-11-18 01:05:36 INFO: 100% (2.0 TiB of 2.0 TiB) in 2h 31m 57s, read: 244.0 MiB/s, write: 11.7 MiB/s
2026-11-18 01:05:38 INFO: backup was done incrementally, reused 2.00 TiB (99%)
2026-11-18 01:05:38 INFO: Finished Backup of VM 246 (02:32:02)

This backup succeeded, but took 2 hours and 32 minutes to process
approximately 2 TiB, despite reusing 99% of the data. We understand that
PBS deduplication still works; the issue is losing fast incremental
change detection and having to scan the entire source disk.

Impact​

On large production VMs, a manual snapshot before maintenance adds hours
to the next backup. This disrupts the backup window, delays other backups,
and contributes to timeouts in other backup tasks.

Enterprise production requirements​

In its current state, this snapshot/backup workflow does not meet our
enterprise production requirements. Taking a routine snapshot of a running
VM before maintenance should not force hours of full-disk scanning or
disrupt the backup schedule for other production VMs.

Reliable snapshots and predictable incremental backups on shared SAN
storage are fundamental requirements for us to consider Proxmox VE a
viable alternative to VMware. As long as routine snapshot operations
break fast incremental backups in this configuration, Proxmox VE cannot
meet those requirements and is not a viable VMware replacement for our
shared-SAN production workloads.

We consider this an enterprise-readiness issue that needs to be addressed
in the snapshot/backup integration, rather than worked around by avoiding
snapshots or extending backup windows.

Expected behavior​

Creating a snapshot of a running VM, without rollback or stop/start,
should preserve PBS dirty-bitmap tracking so that subsequent backups
can continue using fast incremental mode. Deleting a snapshot should
also preserve tracking when the current guest-visible disk contents
are preserved.

Possible cause (not confirmed)​

In qemu-server 9.2.11, blockdev_external_snapshot() calls blockdev_replace().
That function attaches replacement block nodes and detaches the previous
node. There does not appear to be explicit transfer/merge of the PBS
dirty bitmap in this code path.

Relevant source:
https://github.com/proxmox/qemu-ser...86e68c431ca/src/PVE/QemuServer/VolumeChain.pm

This is a hypothesis based on source inspection, not a confirmed root cause.

Questions​

Is this a known limitation or bug of snapshot-as-volume-chain?
Is there an existing issue or fix for preserving PBS dirty bitmaps across
live snapshot creation/deletion?
Is there a supported workaround that retains fast incremental backups
while continuing to use shared FC LVM storage and manual VM snapshots?

Thank you.
 
Hello @matraii

thank you for posting on the forum!

I have tested and verified this behavior locally and opened a Bug report for this [1].
Currently this is a known limitation but we are looking into it if this can be improved.

Please feel free to CC yourself to the Bugreport to get updates on the progress of this feature.

Best regards
Jonas

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=8353