QGA fsfreeze on excluded disks (backup=0) can render a VM unresponsive for up to 60 minutes

Jan 22, 2022
17
7
23
28

Hi,​


We encountered a problematic behavior in Proxmox VE snapshot backups that caused a significant availability issue for a running VM.

The problem is that Proxmox freezes all guest filesystems, including those on disks explicitly excluded from the backup (backup=0). If one of those filesystems is affected by slow or stalled I/O, the freeze operation can block for an extended period, potentially leaving other already-frozen filesystems inaccessible and the VM effectively unresponsive.

Scenario​

Our VM has three virtual disks:

Code:
scsi0: rbd_main:vm-1000-disk-0,discard=on,iothread=1,size=5G,ssd=1
scsi1: rbd_ec_4_2:vm-1000-disk-0,backup=0,discard=on,iothread=1,size=6T,ssd=1
scsi2: rbd_remote_archive:vm-1000-disk-0,backup=0,discard=on,iothread=1,replicate=0,size=6120G,ssd=1
Only scsi0 (5 GB) is included in the PBS backup.

The remaining disks are intentionally excluded. In particular, scsi2 is backed by a remote archival Ceph cluster, which was experiencing significant I/O performance degradation at the time of the incident.

During the backup, we observed:

Code:
00:15:06 INFO: issuing guest-agent 'fs-freeze' command
01:15:06 ERROR: querying status after freezing guest fs failed - timeout after 60 minutes
01:15:06 INFO: starting backup via QMP command
01:15:07 INFO: issuing guest-agent 'fs-thaw' command
01:18:07 ERROR: VM 1000 qga command 'guest-fsfreeze-thaw' failed - got timeout
01:18:07 INFO: started backup task ...
01:18:07 INFO: transferred 764.00 MiB in <1 seconds
01:18:07 INFO: Finished Backup of VM 1000 (01:03:02)
This was not merely a backup performance issue. The VM became unresponsive during the incident, affecting its normal operation.

The backup spent approximately 63 minutes waiting for guest-agent operations, while the actual data transfer took less than one second.

We suspect that the degraded I/O performance of the excluded archival storage contributed to the freeze timeout. However, the underlying concern is independent of the exact cause of that timeout.

Current behavior​

Proxmox uses the QGA command guest-fsfreeze-freeze, which freezes all freezable local filesystems inside the guest, regardless of whether their underlying virtual disks are included in the backup.

This creates an unnecessary dependency between backup operations and excluded storage.

For example, when backing up only a small system disk, an unrelated archival filesystem on a separate, excluded disk can potentially block the freeze operation and affect the availability of the entire VM.

We understand that freezing all filesystems is a conservative default, particularly because filesystems can span multiple virtual disks through LVM, RAID, or similar configurations.

However, this behavior seems unnecessarily restrictive when the administrator knows that the filesystems are independent.

Suggested improvement​

Could Proxmox support configurable filesystem freeze scope for VM backups?

For example:

  • All filesystems (default): Preserve the current behavior.
  • Backed-up disks only: Freeze only filesystems that can be safely associated with disks included in the backup.
  • Explicit mountpoint selection: Allow the administrator to specify which guest filesystems should be frozen.
QEMU already provides guest-fsfreeze-freeze-list, which accepts an explicit list of mountpoints.

We recognize that automatically mapping guest filesystems to virtual disks is not trivial, especially with multi-device filesystems. An explicit mountpoint allowlist would therefore already be a useful improvement.

The existing option to disable filesystem freezing entirely is not an adequate substitute, since it also removes the consistency benefits for disks that are actually being backed up.

Ideally, administrators should be able to preserve filesystem consistency for selected backed-up disks without freezing unrelated, excluded filesystems.

Additional concern: freeze/thaw failures​

In our case, the backup proceeded after the freeze command timed out, and the subsequent thaw command also timed out.

The backup ultimately reported completion, despite the failures of these consistency-related operations.

This raises an additional concern regarding both guest availability and backup consistency when QGA freeze/thaw operations fail or time out.

Related issue​

A similar problem was previously reported in Bugzilla #4887:

https://bugzilla.proxmox.com/show_bug.cgi?id=4887

That report focused on unwanted VSS/SQL Server interactions involving excluded disks.

Our incident demonstrates another operational consequence of the same behavior: a VM availability incident during the backup of a small system disk, potentially triggered by degraded I/O on unrelated storage explicitly excluded from the backup.

Would Proxmox consider supporting selective filesystem freezing, while retaining the current behavior as the default?

Thanks!
 
  • Like
Reactions: jirutka