Hi all,
I ran into the CRS/backup race described in Bugzilla #7622 and in several recent forum discussions.
The failure mode is roughly:
1. A scheduled "all guests" backup builds or uses a node-specific backup queue.
2. CRS/HA migrates a guest before its turn in that queue.
3. vzdump later tries to back up that VM/CT on the old node.
4. The backup fails because the guest configuration is no longer present there.
I wanted a workaround with the following constraints:
- HA failover must remain enabled.
- No maintenance window that globally disables HA/CRS.
- No manual guest-lock injection.
- No root SSH mesh for production backup execution.
- Native Proxmox/PBS backup semantics and retention settings should remain intact.
- New VMs/CTs should automatically be included when the existing backup policy says all=1.
- The design must fail closed if cluster state becomes ambiguous.
## Design
The solution is an HA-aware backup orchestrator built on existing Proxmox primitives.
For every individual guest:
cluster inventory
|
v
resolve current owner immediately before backup
|
v
/nodes/<current-owner>/vzdump via the Proxmox cluster API
|
v
native vzdump backup lock
|
v
PBS / backup storage
The important point is that guest ownership is not assumed from the beginning of the job. The current owner is resolved immediately before each individual backup.
## Native Proxmox locking
I audited the installed PVE 9 source for both LXC and QEMU.
Both vzdump implementations use the native backup lock, and both migration paths call the normal guest lock check. So while an individual guest backup is active:
vzdump -> lock=backup -> migration rejected
The orchestrator does not inject or remove custom guest locks.
This gives the behaviour I wanted: HA/CRS remains available normally, but the guest currently being backed up is protected by the same native Proxmox lock that vzdump already uses.
## Cluster-wide duplicate prevention
A second problem is making sure two nodes cannot start the same scheduled backup run simultaneously.
For that I use pmxcfs state combined with Proxmox' own cluster locking primitive:
PVE::Cluster::cfs_lock_file(...)
The run slot is claimed once and stored in pmxcfs. A concurrent canary from all three cluster nodes produced:
contenders: 3
successful claims: 1
skipped duplicate claims: 2
replicated state: identical on all nodes
The slot is idempotent, so a second coordinator cannot execute the same scheduled run again.
I deliberately chose a persistent run-slot claim instead of a lease that could expire while a long backup is still running.
## The owner-resolution race
There is still a very small race between resolving the current owner and the native backup lock being acquired.
The orchestrator therefore allows exactly one retry, and only if both of these conditions are true:
1. the guest owner actually changed; and
2. the first API request failed with the specific old-node configuration-missing error.
Storage failures, PBS failures, permission errors, lock errors and unrelated vzdump failures are never retried as an ownership race.
## Existing backup policy remains authoritative
The orchestrator reads the existing vzdump job policy from /etc/pve/jobs.cfg.
Configured options are passed through. Options that are not configured remain unset, leaving Proxmox to apply its native/default behaviour.
For example, in my validation one backup job explicitly sets compression while the second backup target leaves compression unset. The orchestrator preserves that difference instead of silently inventing a value.
Retention/pruning stays native as well.
## Transport
Production backup execution uses the Proxmox cluster API:
/nodes/<current-owner>/vzdump
There is no production root-SSH mesh between cluster nodes.
SSH was used only in a few read-only validation harnesses to compare local node state. It is not part of the production backup transport.
## Validation environment
The current validation environment is:
PVE 9.2
3-node cluster
LXC + QEMU
HA-managed guests
PBS plus a second backup target
24 current non-template guests
Validated so far:
- installed PVE source contract for LXC backup/migration locking: PASS
- installed PVE source contract for QEMU backup/migration locking: PASS
- cluster API remote vzdump execution: PASS
- native backup lock visibility and cleanup: PASS
- real LXC-to-PBS backup: PASS
- native PBS retention/pruning: PASS
- strict ownership-race retry predicate: PASS
- cluster-wide pmxcfs coordinator/run-slot claim: PASS
- dynamic inventory discovery: PASS
- full plan rehearsal for backup job 1: 24/24 guests PASS
- full plan rehearsal for backup job 2: 24/24 guests PASS
- unset backup options remain unset: PASS
- no HA configuration mutation during validation: PASS
- no mutation of the existing legacy backup jobs during validation: PASS
- no production root-SSH transport: PASS
Still pending before I call the implementation production-validated:
- full all-guest execution rehearsal against backup target 1
- full all-guest execution rehearsal against backup target 2
- guarded scheduler cutover with rollback
- first scheduled production verification
So at this stage I am intentionally calling this a workaround under validation, not an officially supported Proxmox feature.
## Why I am posting this
The CRS/backup race appears to affect more than one installation, and I have not found an equivalent native mechanism that dynamically re-resolves guest ownership immediately before every individual backup while preserving HA.
I would especially appreciate feedback from Proxmox staff and experienced users on:
- whether there is a simpler native primitive I have missed;
- whether the per-guest current-owner re-resolution is safe from Proxmox' perspective;
- whether cfs_lock_file() plus an idempotent pmxcfs run-slot is an appropriate way to prevent duplicate coordinators;
- whether a similar approach could eventually make sense in the native backup scheduler.
Once the final execution and scheduler gates are complete, I intend to publish a sanitized open-source version of the orchestrator and the validation methodology.
I am also happy to share the relevant source-audit details and evidence from the PVE 9 validation.
I ran into the CRS/backup race described in Bugzilla #7622 and in several recent forum discussions.
The failure mode is roughly:
1. A scheduled "all guests" backup builds or uses a node-specific backup queue.
2. CRS/HA migrates a guest before its turn in that queue.
3. vzdump later tries to back up that VM/CT on the old node.
4. The backup fails because the guest configuration is no longer present there.
I wanted a workaround with the following constraints:
- HA failover must remain enabled.
- No maintenance window that globally disables HA/CRS.
- No manual guest-lock injection.
- No root SSH mesh for production backup execution.
- Native Proxmox/PBS backup semantics and retention settings should remain intact.
- New VMs/CTs should automatically be included when the existing backup policy says all=1.
- The design must fail closed if cluster state becomes ambiguous.
## Design
The solution is an HA-aware backup orchestrator built on existing Proxmox primitives.
For every individual guest:
cluster inventory
|
v
resolve current owner immediately before backup
|
v
/nodes/<current-owner>/vzdump via the Proxmox cluster API
|
v
native vzdump backup lock
|
v
PBS / backup storage
The important point is that guest ownership is not assumed from the beginning of the job. The current owner is resolved immediately before each individual backup.
## Native Proxmox locking
I audited the installed PVE 9 source for both LXC and QEMU.
Both vzdump implementations use the native backup lock, and both migration paths call the normal guest lock check. So while an individual guest backup is active:
vzdump -> lock=backup -> migration rejected
The orchestrator does not inject or remove custom guest locks.
This gives the behaviour I wanted: HA/CRS remains available normally, but the guest currently being backed up is protected by the same native Proxmox lock that vzdump already uses.
## Cluster-wide duplicate prevention
A second problem is making sure two nodes cannot start the same scheduled backup run simultaneously.
For that I use pmxcfs state combined with Proxmox' own cluster locking primitive:
PVE::Cluster::cfs_lock_file(...)
The run slot is claimed once and stored in pmxcfs. A concurrent canary from all three cluster nodes produced:
contenders: 3
successful claims: 1
skipped duplicate claims: 2
replicated state: identical on all nodes
The slot is idempotent, so a second coordinator cannot execute the same scheduled run again.
I deliberately chose a persistent run-slot claim instead of a lease that could expire while a long backup is still running.
## The owner-resolution race
There is still a very small race between resolving the current owner and the native backup lock being acquired.
The orchestrator therefore allows exactly one retry, and only if both of these conditions are true:
1. the guest owner actually changed; and
2. the first API request failed with the specific old-node configuration-missing error.
Storage failures, PBS failures, permission errors, lock errors and unrelated vzdump failures are never retried as an ownership race.
## Existing backup policy remains authoritative
The orchestrator reads the existing vzdump job policy from /etc/pve/jobs.cfg.
Configured options are passed through. Options that are not configured remain unset, leaving Proxmox to apply its native/default behaviour.
For example, in my validation one backup job explicitly sets compression while the second backup target leaves compression unset. The orchestrator preserves that difference instead of silently inventing a value.
Retention/pruning stays native as well.
## Transport
Production backup execution uses the Proxmox cluster API:
/nodes/<current-owner>/vzdump
There is no production root-SSH mesh between cluster nodes.
SSH was used only in a few read-only validation harnesses to compare local node state. It is not part of the production backup transport.
## Validation environment
The current validation environment is:
PVE 9.2
3-node cluster
LXC + QEMU
HA-managed guests
PBS plus a second backup target
24 current non-template guests
Validated so far:
- installed PVE source contract for LXC backup/migration locking: PASS
- installed PVE source contract for QEMU backup/migration locking: PASS
- cluster API remote vzdump execution: PASS
- native backup lock visibility and cleanup: PASS
- real LXC-to-PBS backup: PASS
- native PBS retention/pruning: PASS
- strict ownership-race retry predicate: PASS
- cluster-wide pmxcfs coordinator/run-slot claim: PASS
- dynamic inventory discovery: PASS
- full plan rehearsal for backup job 1: 24/24 guests PASS
- full plan rehearsal for backup job 2: 24/24 guests PASS
- unset backup options remain unset: PASS
- no HA configuration mutation during validation: PASS
- no mutation of the existing legacy backup jobs during validation: PASS
- no production root-SSH transport: PASS
Still pending before I call the implementation production-validated:
- full all-guest execution rehearsal against backup target 1
- full all-guest execution rehearsal against backup target 2
- guarded scheduler cutover with rollback
- first scheduled production verification
So at this stage I am intentionally calling this a workaround under validation, not an officially supported Proxmox feature.
## Why I am posting this
The CRS/backup race appears to affect more than one installation, and I have not found an equivalent native mechanism that dynamically re-resolves guest ownership immediately before every individual backup while preserving HA.
I would especially appreciate feedback from Proxmox staff and experienced users on:
- whether there is a simpler native primitive I have missed;
- whether the per-guest current-owner re-resolution is safe from Proxmox' perspective;
- whether cfs_lock_file() plus an idempotent pmxcfs run-slot is an appropriate way to prevent duplicate coordinators;
- whether a similar approach could eventually make sense in the native backup scheduler.
Once the final execution and scheduler gates are complete, I intend to publish a sanitized open-source version of the orchestrator and the validation methodology.
I am also happy to share the relevant source-audit details and evidence from the PVE 9 validation.