PVE/PBS backup-start stalls cause Windows VSS/SQL errors - Bug #8245

[Winter0916]

Hello,

We have filed Proxmox Bugzilla #8245 regarding intermittent PVE-to-PBS backup-start latency that can keep Windows guests frozen long enough for VSS to time out.

We are posting our findings here because similar QMP/backup stalls have appeared in other forum discussions, and we would welcome feedback from other administrators and Proxmox developers.

Bug report: https://bugzilla.proxmox.com/show_bug.cgi?id=8245

1. What we are seeing​

During a snapshot-mode backup with QEMU Guest Agent filesystem freezing enabled, the relevant sequence is:

1. PVE issues fs-freeze; Windows VSS and the SQL Server VSS Writer quiesce I/O.
2. PVE issues QMP backup-start while the guest is frozen.
3. On some attempts, initialization takes too long before fs-thaw is issued.
4. Windows VSS reports a hold-writes timeout and SQL Server logs VDI-related errors.
5. The backup data transfer can nevertheless continue, sometimes ending in TASK OK even though application quiescing failed.

The Windows-side errors we have observed include:

• VSS 0x80042314 (VSS_E_HOLD_WRITES_TIMEOUT)
• VSS Event 12298
• SQL Server events 3041, 18210, and 24583
• Operating system error 995

Our primary concern is application consistency, not simply whether PBS received the backup chunks. We are not claiming these errors demonstrate database corruption.

2. Environments​

Production PVE cluster → Production PBS

• Four-node Ceph-backed PVE cluster
• pve-manager: 9.2.4
• qemu-server: 9.2.0
• pve-qemu-kvm: 11.0.2-1
• libproxmox-backup-qemu0: 2.0.2
• Running kernel: 6.17.9-1-pve
• PBS Server: 4.2.5-1
• PBS running kernel: 7.0.14-14-pve

Office standalone PVE → Office PBS

• pve-manager: 9.2.24
• qemu-server: 9.2.13
• pve-qemu-kvm: 11.0.3-4
• libproxmox-backup-qemu0: 2.0.3
• Running kernel: 6.17.13-3-pve
• PBS Server: 4.2.3-1
• PBS running kernel: 7.0.14-5-pve

Both PBS servers use HDD-backed ZFS datastores with two six-disk RAIDZ2 vdevs.

A separate PVE test cluster (pve-manager 9.2.21) also uses the office PBS. We are not counting it as a third independent PBS reproduction.

3. Measured historical findings​

Our July–September 2026 production investigation found:

• 1,381 failed guest-freeze attempts among 8,609 observed attempts in a historical PVE01 dataset.
• In two examined VMs, 498 of 523 freeze failures also coincided with QMP timeouts.
• Median archive-creation-to-PBS-configuration time was approximately 23 seconds in failing cases versus 4 seconds in successful cases. This is an initialization timing proxy, not an isolated .fidx download duration or a direct freeze-duration measurement.
• One correlated SQL/VSS failure occurred approximately 26 seconds after freezing began.
• Enabling fleecing did not eliminate the issue in two initial 20-VM backup waves: 5/20 failures without fleecing versus 6/20 with fleecing. These were separate waves, not a fully controlled causal trial.

These historical values are not presented as newly measured results from PVE 9.2.24. The office reproduction establishes that the symptom has also been observed on newer PVE backup components.

4. Leading hypothesis — PBS I/O latency​

We suspect that PBS response latency is extending the backup-start critical path.

In our installations, the leading potential amplifier is HDD/ZFS I/O contention during concurrent ingest, verification, garbage collection, and/or sync.

Similar forum reports involving remote or WAN-connected PBS suggest that network-related delays may expose a broader version of the same design weakness, although those reports do not establish an identical mechanism.

Public proxmox-backup-qemu source shows previous-manifest retrieval (index.json.blob) and prior fixed-index retrieval (drive-*.img.fidx) during backup preparation/registration.

We suspect that some of this remote metadata work occurs after fs-freeze. We have not isolated per-operation timings proving .fidx retrieval is the dominant delay.

A March 2026 qemu-server development patch explicitly documents QMP backup-start between fs-freeze and fs-thaw and a 125-second timeout for that QMP command. That command timeout is not itself a measured guest freeze duration.

https://lore.proxmox.com/all/20260326113251.68540-3-f.ebner@proxmox.com/

5. Suggested architectural change​

Would it be feasible to split PBS backup start into two stages?

Phase 1 — Prepare (guest running normally)

• Establish and authenticate the PBS connection.
• Retrieve and validate the previous backup manifest.
• Retrieve the applicable previous .fidx indexes.
• Prepare previous chunk-reference information and any other non-consistency-critical backup state.

Phase 2 — Snapshot (short guest freeze)

• Issue fs-freeze.
• Establish the required consistent backup snapshot point.
• Issue fs-thaw promptly, including on failure paths.
• Continue transferring backup data after thaw.

Any implementation must preserve dirty-bitmap correctness, previous-backup identity, encryption/incremental behavior, and reliable thaw/abort cleanup.

We are proposing an area for investigation, not claiming that moving any particular operation is already known to be safe.

6. Questions for the community and developers​

1. Has anyone correlated Windows VSS 0x80042314 or SQLVDI errors with a prolonged QMP backup-start interval, particularly when the backup still completes successfully?
2. Has anyone instrumented the time spent in PBS connection setup, previous-manifest retrieval, per-drive .fidx retrieval, and image registration?
3. Is there an existing prepare/activate interface, or would a two-stage backup-start API be required to move remote operations outside the freeze window?
4. Do others have comparable observations on SSD-backed PBS, HDD-backed PBS, or PBS reached over WAN/VPN connections?

7. Related forum discussions​

These reports describe related symptoms, but not necessarily the same underlying defect.

Prolonged freeze-to-thaw interval and QMP timeout:

https://forum.proxmox.com/threads/