LVM snapshot-as-volume-chain: live VM snapshots lose PBS dirty bitmaps and force full disk scans

matraii

New Member
Mar 13, 2025
1
0
0

Summary​

Taking a manual snapshot of a running VM on shared LVM storage with
snapshot-as-volume-chain enabled causes the next PBS backup to report
"dirty-bitmap status: created new" and scan the entire virtual disk.

The VM is not stopped or started again during this workflow. This is not
the known loss of dirty bitmaps after a VM stop/start. The issue occurs
whether the manual snapshot is retained or subsequently deleted.

We previously used local ZFS storage, where taking a snapshot of a running
VM did not cause this behavior.

Environment​

Shared Fibre Channel storage: IBM Storwize V7000, presented to Proxmox VE
through multipath and regular shared LVM (not LVM-thin).
VM disks use qcow2 on LVM with snapshot-as-volume-chain enabled.
Backups are Proxmox VE VM backups to Proxmox Backup Server, using snapshot mode.
Subscription level: Community.

Relevant installed versions from the affected PVE node:
proxmox-ve: 9.2.0
pve-manager: 9.2.21
qemu-server: 9.2.11
pve-qemu-kvm: 11.0.3-4
libpve-storage-perl: 9.1.13
libproxmox-backup-qemu0: 2.0.3
proxmox-backup-client: 4.2.8-1
lvm2: 2.03.31-2+pmx1
Running kernel: 6.17.13-2-pve

The backup-client version above is from the PVE node; it is not a statement
of the PBS server's version. The installed QEMU package version is listed;
the running VM's QEMU version has not been queried separately.

Relevant storage configuration​

lvm: IBM_v7000_48tb
vgname vg_v7000_48tb
content images,rootdir
saferemove 0
shared 1
snapshot-as-volume-chain 1

lvm: IBM_v7000_32tb
vgname vg_v7000_32tb
content rootdir,images
saferemove 0
shared 1
snapshot-as-volume-chain 1

Relevant VM configuration (VMID 246)​

agent: 1
bios: ovmf
machine: q35
efidisk0: IBM_v7000_32tb:vm-246-disk-0.qcow2,efitype=4m,ms-cert=2023w,pre-enrolled-keys=1,size=528K
scsi0: IBM_v7000_48tb:vm-246-disk-0.qcow2,iothread=1,size=60G
scsi2: IBM_v7000_48tb:vm-246-disk-1.qcow2,iothread=1,size=2000G
scsihw: virtio-scsi-pci
parent: bkp

Steps to reproduce/check​

  1. Use a running VM with qcow2 disks on shared LVM storage configured with
    snapshot-as-volume-chain=1.
  2. Complete a PBS backup, then verify that a subsequent backup can use the
    existing dirty bitmap (fast incremental mode).
  3. Take a manual Proxmox VE snapshot while the VM remains running.
  4. Do not stop/start the VM or roll it back.
  5. Run another backup to the same PBS target.
  6. Observe whether the backup reports "dirty-bitmap status: created new"
    and scans the complete disk instead of only changed blocks.
  7. Also check the workflow where the manual snapshot is deleted before
    the next backup; we observe the same full-scan consequence.

Observed production example​

The manual snapshot "bkp" was created on 2026-11-17 at 18:51:45 and was
still present when the following backup ran. Times below are local
Europe/Budapest time (CET, UTC+1).

Selected backup log lines:

2026-11-17 22:33:36 INFO: Starting Backup of VM 246 (qemu)
2026-11-17 22:33:36 INFO: status = running
2026-11-17 22:33:37 INFO: backup mode: snapshot
2026-11-17 22:33:39 INFO: efidisk0: dirty-bitmap status: created new
2026-11-17 22:33:39 INFO: scsi0: dirty-bitmap status: created new
2026-11-17 22:33:39 INFO: scsi2: dirty-bitmap status: created new
2026-11-17 22:33:42 INFO: 0% (776.0 MiB of 2.0 TiB) in 3s, read: 258.7 MiB/s, write: 6.7 MiB/s
2026-11-18 01:05:36 INFO: 100% (2.0 TiB of 2.0 TiB) in 2h 31m 57s, read: 244.0 MiB/s, write: 11.7 MiB/s
2026-11-18 01:05:38 INFO: backup was done incrementally, reused 2.00 TiB (99%)
2026-11-18 01:05:38 INFO: Finished Backup of VM 246 (02:32:02)

This backup succeeded, but took 2 hours and 32 minutes to process
approximately 2 TiB, despite reusing 99% of the data. We understand that
PBS deduplication still works; the issue is losing fast incremental
change detection and having to scan the entire source disk.

Impact​

On large production VMs, a manual snapshot before maintenance adds hours
to the next backup. This disrupts the backup window, delays other backups,
and contributes to timeouts in other backup tasks.

Enterprise production requirements​

In its current state, this snapshot/backup workflow does not meet our
enterprise production requirements. Taking a routine snapshot of a running
VM before maintenance should not force hours of full-disk scanning or
disrupt the backup schedule for other production VMs.

Reliable snapshots and predictable incremental backups on shared SAN
storage are fundamental requirements for us to consider Proxmox VE a
viable alternative to VMware. As long as routine snapshot operations
break fast incremental backups in this configuration, Proxmox VE cannot
meet those requirements and is not a viable VMware replacement for our
shared-SAN production workloads.

We consider this an enterprise-readiness issue that needs to be addressed
in the snapshot/backup integration, rather than worked around by avoiding
snapshots or extending backup windows.

Expected behavior​

Creating a snapshot of a running VM, without rollback or stop/start,
should preserve PBS dirty-bitmap tracking so that subsequent backups
can continue using fast incremental mode. Deleting a snapshot should
also preserve tracking when the current guest-visible disk contents
are preserved.

Possible cause (not confirmed)​

In qemu-server 9.2.11, blockdev_external_snapshot() calls blockdev_replace().
That function attaches replacement block nodes and detaches the previous
node. There does not appear to be explicit transfer/merge of the PBS
dirty bitmap in this code path.

Relevant source:
https://github.com/proxmox/qemu-ser...86e68c431ca/src/PVE/QemuServer/VolumeChain.pm

This is a hypothesis based on source inspection, not a confirmed root cause.

Questions​

Is this a known limitation or bug of snapshot-as-volume-chain?
Is there an existing issue or fix for preserving PBS dirty bitmaps across
live snapshot creation/deletion?
Is there a supported workaround that retains fast incremental backups
while continuing to use shared FC LVM storage and manual VM snapshots?

Thank you.