PVE/PBS backup-start stalls cause Windows VSS/SQL errors - Bug #8245

Winter0916

New Member
Jul 7, 2025
3
1
3
Hello,

We have filed Proxmox Bugzilla #8245 regarding intermittent PVE-to-PBS backup-start latency that can keep Windows guests frozen long enough for VSS to time out.

We are posting our findings here because similar QMP/backup stalls have appeared in other forum discussions, and we would welcome feedback from other administrators and Proxmox developers.

Bug report: https://bugzilla.proxmox.com/show_bug.cgi?id=8245

1. What we are seeing​

During a snapshot-mode backup with QEMU Guest Agent filesystem freezing enabled, the relevant sequence is:

  1. PVE issues fs-freeze; Windows VSS and the SQL Server VSS Writer quiesce I/O.
  2. PVE issues QMP backup-start while the guest is frozen.
  3. On some attempts, initialization takes too long before fs-thaw is issued.
  4. Windows VSS reports a hold-writes timeout and SQL Server logs VDI-related errors.
  5. The backup data transfer can nevertheless continue, sometimes ending in TASK OK even though application quiescing failed.
The Windows-side errors we have observed include:

  • VSS 0x80042314 (VSS_E_HOLD_WRITES_TIMEOUT)
  • VSS Event 12298
  • SQL Server events 3041, 18210, and 24583
  • Operating system error 995
Our primary concern is application consistency, not simply whether PBS received the backup chunks. We are not claiming these errors demonstrate database corruption.

2. Environments​

Production PVE cluster → Production PBS

  • Four-node Ceph-backed PVE cluster
  • pve-manager: 9.2.4
  • qemu-server: 9.2.0
  • pve-qemu-kvm: 11.0.2-1
  • libproxmox-backup-qemu0: 2.0.2
  • Running kernel: 6.17.9-1-pve
  • PBS Server: 4.2.5-1
  • PBS running kernel: 7.0.14-14-pve
Office standalone PVE → Office PBS

  • pve-manager: 9.2.24
  • qemu-server: 9.2.13
  • pve-qemu-kvm: 11.0.3-4
  • libproxmox-backup-qemu0: 2.0.3
  • Running kernel: 6.17.13-3-pve
  • PBS Server: 4.2.3-1
  • PBS running kernel: 7.0.14-5-pve
Both PBS servers use HDD-backed ZFS datastores with two six-disk RAIDZ2 vdevs.

A separate PVE test cluster (pve-manager 9.2.21) also uses the office PBS. We are not counting it as a third independent PBS reproduction.

3. Measured historical findings​

Our July–September 2026 production investigation found:

  • 1,381 failed guest-freeze attempts among 8,609 observed attempts in a historical PVE01 dataset.
  • In two examined VMs, 498 of 523 freeze failures also coincided with QMP timeouts.
  • Median archive-creation-to-PBS-configuration time was approximately 23 seconds in failing cases versus 4 seconds in successful cases. This is an initialization timing proxy, not an isolated .fidx download duration or a direct freeze-duration measurement.
  • One correlated SQL/VSS failure occurred approximately 26 seconds after freezing began.
  • Enabling fleecing did not eliminate the issue in two initial 20-VM backup waves: 5/20 failures without fleecing versus 6/20 with fleecing. These were separate waves, not a fully controlled causal trial.
These historical values are not presented as newly measured results from PVE 9.2.24. The office reproduction establishes that the symptom has also been observed on newer PVE backup components.

4. Leading hypothesis — PBS I/O latency​

We suspect that PBS response latency is extending the backup-start critical path.

In our installations, the leading potential amplifier is HDD/ZFS I/O contention during concurrent ingest, verification, garbage collection, and/or sync.

Similar forum reports involving remote or WAN-connected PBS suggest that network-related delays may expose a broader version of the same design weakness, although those reports do not establish an identical mechanism.

Public proxmox-backup-qemu source shows previous-manifest retrieval (index.json.blob) and prior fixed-index retrieval (drive-*.img.fidx) during backup preparation/registration.

We suspect that some of this remote metadata work occurs after fs-freeze. We have not isolated per-operation timings proving .fidx retrieval is the dominant delay.

A March 2026 qemu-server development patch explicitly documents QMP backup-start between fs-freeze and fs-thaw and a 125-second timeout for that QMP command. That command timeout is not itself a measured guest freeze duration.

https://lore.proxmox.com/all/20260326113251.68540-3-f.ebner@proxmox.com/

5. Suggested architectural change​

Would it be feasible to split PBS backup start into two stages?

Phase 1 — Prepare (guest running normally)

  • Establish and authenticate the PBS connection.
  • Retrieve and validate the previous backup manifest.
  • Retrieve the applicable previous .fidx indexes.
  • Prepare previous chunk-reference information and any other non-consistency-critical backup state.
Phase 2 — Snapshot (short guest freeze)

  • Issue fs-freeze.
  • Establish the required consistent backup snapshot point.
  • Issue fs-thaw promptly, including on failure paths.
  • Continue transferring backup data after thaw.
Any implementation must preserve dirty-bitmap correctness, previous-backup identity, encryption/incremental behavior, and reliable thaw/abort cleanup.

We are proposing an area for investigation, not claiming that moving any particular operation is already known to be safe.

6. Questions for the community and developers​

  1. Has anyone correlated Windows VSS 0x80042314 or SQLVDI errors with a prolonged QMP backup-start interval, particularly when the backup still completes successfully?
  2. Has anyone instrumented the time spent in PBS connection setup, previous-manifest retrieval, per-drive .fidx retrieval, and image registration?
  3. Is there an existing prepare/activate interface, or would a two-stage backup-start API be required to move remote operations outside the freeze window?
  4. Do others have comparable observations on SSD-backed PBS, HDD-backed PBS, or PBS reached over WAN/VPN connections?

7. Related forum discussions​

These reports describe related symptoms, but not necessarily the same underlying defect.

Prolonged freeze-to-thaw interval and QMP timeout:

https://forum.proxmox.com/threads/
 
Thanks @JeanL and @MarkusKo for the suggestions.

I wanted to clarify what we have already tested.

VSS configuration

We previously configured VssOption=5 (VSS_BT_COPY) at our developers' request to maintain control over the SQL Server backup chains.

This configuration was implemented for reasons unrelated to our current VSS timeout investigation.

Changing the VSS backup type does not eliminate the filesystem freeze/thaw operations or address delays in PVE/PBS backup initialization.

VirtIO drivers

We have tested updated Windows guest drivers and earlier releases.

The driver updates have made no material difference. The backup-start stalls and associated VSS/SQL errors remain essentially unchanged.

For reference, the virtio-win installation media we use contains these Windows Server 2022 storage drivers:

  • vioscsi: 100.102.104.29500 (build 295)
  • viostor: 100.103.104.30000 (build 300)
  • NetKVM: 100.103.104.30000 (build 300)
These are versions confirmed in the installation media; we have not independently established which exact versions each affected VM currently has loaded.

While guest driver issues remain a consideration, the lack of any measurable change after updating them, combined with the PVE-side timing evidence, suggests they are unlikely to be the primary cause.

Bugzilla and upstream development

Our original report, #8245, has been marked as a duplicate of #5080:

https://bugzilla.proxmox.com/show_bug.cgi?id=5080

Reviewing #5080 revealed that Fabian Grünbichler and Fiona Ebner discussed separating PBS backup preparation from the QEMU backup-start operation in December 2023.

In particular, comment #10 proposes a proxmox-backup-setup operation that would establish the PBS session and prepare the previous backup indexes before the guest is frozen.

This closely matches the architectural change we requested.

There is also an October 26, 2026 PBS development RFC for #7756 addressing previous-manifest/index retrieval delays under heavy server load:

https://lore.proxmox.com/all/20261026134559.352926-2-c.ebner@proxmox.com/T/

Our current assessment

Our leading hypothesis remains PBS-side response latency during backup initialization, potentially amplified by datastore I/O contention or network delays.

Windows VSS can time out well before the 125-second QMP timeout. A backup may subsequently complete successfully despite having encountered an application-quiescing error.

Improving PBS retrieval performance should help, but separating non-consistency-critical preparation from the guest freeze interval would address the underlying latency dependency more comprehensively.

We have a test environment available and would be interested in validating the proposed client-side changes.

Thanks again for the suggestions.
 
Try to select a local ssd / nvme drive as the backup fleecing target, not ceph storage.

Code:
Both PBS servers use HDD-backed ZFS datastores with two six-disk RAIDZ2 vdevs.

You should add a triple mirror ssd special device to your RAIDZ2 pool (only helps with new metadata, old metadata won't be migrated to special device), otherwise you won't get good results with PBS.

https://forum.proxmox.com/threads/s...y-adding-zfs-special-device.85668/post-376111
 

Windows Server 2022: VSS writer timeouts during PVE/PBS backups – possible backup initialization delay​

Hello everyone,

we are experiencing recurring Windows VSS writer timeouts during PVE → PBS backups, although the backup tasks sometimes finish successfully.

We suspect a possible connection to the backup initialization delay discussed here:

https://forum.proxmox.com/threads/p...cause-windows-vss-sql-errors-bug-8245.187346/

We have performed several tests but have not yet identified the root cause.

1. Environment​

Proxmox VE

  • PVE Manager: 9.2.26
  • pve-qemu-kvm: 11.1.3-1
  • qemu-server: 9.2.15
  • libproxmox-backup-qemu0: 2.0.4
  • ZFS VM storage
  • Dedicated 10 GbE backup/storage network
Affected VM

  • Windows Server 2022
  • Active Directory, DNS, DHCP and file services
  • Additional SQL Anywhere database application
  • Two VirtIO block disks, approximately 1 TiB combined
  • Machine: pc-i440fx-5.2
  • CPU: qemu64
  • QEMU Guest Agent enabled
Backup destination

  • PBS 4.2.9-1, running inside an LXC container
  • Eight HDDs in ZFS RAIDZ2
  • PBS datastore on a dedicated ZFS dataset
  • Backup nodes also handle conventional file/server backups

2. Problem and timeline​

A representative incremental backup on November 13, 2026:

19:27:57 PVE backup started
issuing guest-agent 'fs-freeze' command
starting backup via QMP command
issuing guest-agent 'fs-thaw' command

19:35:26 Backup finished successfully
Backup statistics:

Reused: 981.50 GiB (96%)
Transferred: 36.50 GiB
Duration: 00:07:29
However, Windows logged:

19:28:01 qemu-ga: guest-fsfreeze called

19:28:20 MSDTC Event 4879
Error 0x80000171

19:29:22 VSS Event 8229
Registry Writer
Error 0x800423f2
Freeze/Thaw timeout exceeded

19:29:32 qemu-ga: guest-fsthaw called
There were approximately 91 seconds between the logged freeze and thaw calls.

We understand that this does not necessarily mean the guest was actually frozen for 91 seconds.

The backup finished successfully, but several VSS writers subsequently reported errors.

3. VSS writer failures​

After rebooting Windows, all 17 VSS writers were stable.

Following a test backup:

WriterError
System WriterTimeout (9)
FRS WriterTimeout (9)
Shadow Copy Optimization WriterTimeout (9)
ASR WriterTimeout (9)
Registry WriterFailed (10)
MSSearch Service WriterTimeout (9)
COM+ REGDB WriterTimeout (9)
NTDS, DHCP Jet, SqlServerWriter and several other writers remained stable.

The QEMU Guest Agent and QEMU Guest Agent VSS Provider services were running. VirtIO driver package 0.1.302 was installed.

An earlier backup also produced:

timeout when try to receive Frozen event
from VSS provider:
Windows error 0x80004005

4. Tests already performed​

Database isolation

We stopped the SQL Anywhere database service and repeated the backup.

Result: The backup completed, but multiple VSS writers still timed out.

The database service was restarted afterward.

PBS task log

For the successful backup, PBS recorded a task running approximately from 19:29:19 to 19:35:23.

The log included:

download 'index.json.blob' from previous backup
register chunks in 'drive-virtio0.img.fidx'
download 'drive-virtio0.img.fidx'
created new fixed index 1
register chunks in 'drive-virtio2.img.fidx'
download 'drive-virtio2.img.fidx'
created new fixed index 2
...
TASK OK
Interestingly, the PBS task start timestamp is approximately 78 seconds after the logged guest-fsfreeze call.

The PBS log does not contain individual timestamps for the index operations, so we cannot determine exactly where the delay occurred.

Garbage collection

A PBS garbage collection task had already finished before this backup started.

ZFS and storage

  • Pool ONLINE, no reported read/write/checksum errors
  • Scrub completed without errors
  • No obvious disk latency problems during idle measurements
  • PBS fixed index files total approximately 8 MiB per backup
  • A cached/direct-read test of one index completed in approximately 4.35 ms
These measurements do not rule out storage latency during actual backup initialization.

5. ZFS changes during investigation​

Both backup nodes have substantial available RAM and primarily handle backup workloads.

We increased the ZFS ARC limits:

Backup nodeRAMARC beforeARC after
Node A54 GiB5.5 GiB16 GiB
Node B62 GiB9.43 GiB20 GiB
The changes were applied at runtime, persisted in /etc/modprobe.d/zfs.conf, and the initramfs images were updated.

The actual PBS datastore already uses:

primarycache all
secondarycache all
compression off
atime off
recordsize 128K
Separate datasets for conventional backups use primarycache=metadata.

No ZFS dataset cache properties were changed.

We have not yet established whether increasing ARC improves the VSS situation.

6. Questions​

We would appreciate clarification from the Proxmox developers:

  1. Is the backup initialization issue discussed in bug #8245 / #5080 still present in our installed package versions? Has the proposed change to retrieve previous backup metadata before guest freeze been released?
  2. Could the approximately 78-second gap between the guest-fsfreeze log entry and PBS task start be related to backup initialization or metadata retrieval? How should these timestamps be interpreted?
  3. Why can a PVE/PBS backup complete successfully while multiple Windows VSS writers subsequently report timeouts? How can we reliably verify application consistency?
  4. Is there a recommended way to obtain detailed timestamps for QEMU Guest Agent freeze/thaw, PBS initialization, previous index retrieval and QMP backup startup?

7. Current conclusion​

The issue remains unresolved.

  • VSS writer timeouts occur even with the additional database service stopped.
  • QEMU Guest Agent and VSS provider are installed and running.
  • PBS backup tasks can finish successfully despite VSS writer failures.
  • No conclusive storage bottleneck has been identified.
  • ARC limits were increased, but their impact has not yet been measured.
  • The timing appears potentially consistent with the previously discussed PVE/PBS initialization issue.
Any guidance on the relevant bug-fix status or additional diagnostics would be greatly appreciated.

Thank you!


Update: Reproduced VSS timeouts after increasing ZFS ARC​

We performed another controlled backup test on November 14, 2026, after rebooting the Windows Server 2022 VM.

Results:

  • All 17 VSS writers were stable before the backup.
  • The backup completed successfully.
  • The same seven VSS writers reported timeout/errors afterward.
  • Increasing the ZFS ARC limits did not resolve the issue.
Timeline:

10:08:04 PVE backup started
10:08:06 QEMU Guest Agent: guest-fsfreeze called
10:08:51 PBS backup task started
10:09:26 Windows VSS Event 8229 (0x800423f2)
10:09:41 QEMU Guest Agent: guest-fsthaw called
The interval between the logged freeze and thaw calls was 95 seconds, compared with 91 seconds in the previous test.

We also monitored ZFS I/O latency at one-second intervals on the PBS host.

Approximate measurements during the critical 95-second window:

MetricResult
Maximum read total_wait25 ms
Maximum write total_wait13 ms
Read intervals above 20 ms2
Write intervals above 20 ms0
Approximate read operations10
The time correlation is approximate because the ZFS recording did not include individual timestamps. ARC cache hits are not represented as physical disk reads.

These measurements do not suggest sustained ZFS disk overload during the critical interval, although they cannot exclude short stalls or delays elsewhere.

The PBS task log again shows retrieval and registration of previous backup indexes, but without timestamps for individual operations.

Our main question remains: Could backup initialization or previous-index processing delay the guest thaw sequence in our installed PVE/PBS versions?

We would appreciate guidance on additional diagnostics or the status of the related fixes.
 
Last edited:
Eight HDDs in ZFS RAIDZ2
= Slow IOPS because of a single HDD IOPS + ZFS overhead = Slow PBS datastore
Try enabling Fleecing on PVE side, might help.
As reminder : During backup, any disk writes inside the VM is paused until the original block is sent to PBS first. When PBS is slow, without fleecing, the VM slows down, up to Windows services time out (myself impacted). In the past, it could even lead to VM crashes or corrupted databases.
 
  • Like
Reactions: semischizo

[Potential Fix] Proxmox VE 9.2 – Windows Server 2022 VSS / QEMU Guest Agent Backup Issues​

After migrating a Windows Server 2022 VM (Active Directory, DNS, DHCP) from Collax to Proxmox VE 9.2, I experienced recurring VSS and QEMU Guest Agent issues during backups.

Symptoms:

  • VSS Writer failures (System Writer, Registry Writer, DHCP Jet Writer)
  • VSS errors 0x80070005 and 0x800423f2
  • QEMU Guest Agent timeouts
  • QEMU VSS Provider CommitSnapshots errors (0x80004005)
  • Reboots temporarily restored VSS Writer stability but did not resolve the underlying issue
What I changed:

I downgraded only the QEMU Guest Agent and its VSS Provider:

  • Previous version: 110.2.3 (VirtIO ISO 0.1.302)
  • New version: 108.0.2 (VirtIO ISO 0.1.266)
  • VirtIO SCSI and network drivers remained at version 0.1.302.
  • No Windows reboot was required.
Results:

The first Proxmox Backup Server backup after the downgrade completed successfully:

  • Guest Agent fs-freeze and fs-thaw completed without reported errors
  • 60.64 GiB transferred incrementally
  • Backup completed in 14 minutes and 43 seconds
  • All 17 VSS Writers remained stable with no errors after the backup
Conclusion:

Downgrading QEMU Guest Agent from 110.2.3 to 108.0.2 appears promising and resolved the issue during the initial backup test.

I will continue monitoring subsequent backups to confirm long-term stability.

Hopefully this helps others experiencing similar VSS or QEMU Guest Agent issues with Windows Server 2022 on Proxmox.
 
  • Like
Reactions: _gabriel