Hello everyone,
We recently lost a VM's disk to thin pool metadata corruption during a routine live migration, and traced it to how a shared thin pool is handed from one node to another. The trigger is the third-party StarWind "StarLVM" storage plugin. The underlying gap, though, is that qemu-server gives a storage plugin no chance to act between "migration completed" and "resume on target". Below is our analysis, and a small proposed qemu-server hook that would allow a storage plugin to do this safely.
Environment
- Proxmox VE 9.2.21 (pve-manager), qemu-server 9.2.11, 4-node cluster
- Shared FC/iSCSI LUN (Fujitsu Eternus) via multipath, one shared LVM VG
- StarWind plugin `starwind-proxmox-plugin-pve9` 1.1.41, storage type `starlvm`, `shared 1`
How StarLVM works
StarLVM adds snapshots to shared block storage. It creates **one dm-thin pool per VM** (`lvmth-<vmid>`) inside the shared VG and puts the VM's disks in it as thin LVs. Because the storage is marked `shared`, Proxmox allows live migration. During a live migration, that thin pool ends up active on the source and the target node at the same time.
What happened (times CEST, node names anonymised)
A rolling update live-migrated the VM from node A to node B. The task log of the migration and the kernel logs of both nodes give this sequence:
From then on, every guest write that needed an allocation failed. The guest (Windows Server) ended in Automatic Repair. `lvchange -an` later reported `Integrity check of metadata for pool ... failed`, and we restored the VM from backup.
We ruled out:
- space: the VG had 2.95 TB free, and the pool was at 64% data and 29% metadata,
- snapshots: there were none,
- LVM autoactivation: every pool activation on the nodes matches a migration or `startall` to the second,
- the pool being active on a third node.
The guest was busy during the migration: QEMU had to send 6.4 GiB for 4 GiB of RAM, and the downtime limit was raised automatically.
Root cause
The root cause is that dm-thin is not cluster-aware. A thin pool reads its metadata (superblock, mapping btree, space maps) from disk once, when it is activated, and from then on works from that in-kernel copy, committing it back periodically. Two hosts with the same pool active each have their own copy, and neither ever re-reads what the other wrote. This is why lvmlockd only allows exclusive activation of thin pools, and why Proxmox's own `lvmthin` storage is local-only.
During a live migration, qemu-server does this:
1. Target: `vm_start_nolock` sets `$ENV{PVE_MIGRATED_FROM}` and calls `PVE::Storage::activate_volumes`before QEMU starts, so before the RAM transfer (QemuServer.pm in 9.2.11, around line 5632). For StarLVM, that activates the thin pool on the target, which loads its metadata at that moment.
2. RAM transfer: the guest keeps running on the source, here for about 60 seconds. Whenever it writes to a block that isn't allocated yet, the source's dm-thin allocates it and commits new metadata to disk. The target's in-kernel copy doesn't see any of it.
3. Resume: `phase3_cleanup` sends `resume` to the target (QemuMigrate.pm around line 1749). `vm_resume` (QemuServer/RunState.pm) makes no storage call.
4. Source clean-up: only after that does `vm_stop_cleanup` call `deactivate_volumes` on the source (QemuMigrate.pm around line 1872).
So the target resumes with metadata that is up to a minute old, while the on-disk metadata has moved on. The target's next commit writes its stale view over the source's. That loses the source's allocations from the migration window, and it can hand blocks the source had already used to new writes. Here it surfaced 87 seconds later as an out-of-bounds metadata block. We can't rule out that the same race silently corrupted data on other VMs where it didn't hit the metadata. About 30 other VMs made the same migrations that morning without a visible error, which fits a race that depends on the guest allocating new blocks during the transfer.
Why a storage plugin can't fix this alone
The only safe moment to move a thin pool is after the source has paused and flushed, and before the target resumes. qemu-server calls no storage plugin method in that window:
- activate_volume runs on the target before the RAM transfer,
- deactivate_volume runs on the source after the resume.
A plugin can refuse dual activation, but that rules out live migration altogether. It can't hand the pool over safely.
Proposal: an optional pre-resume hook for shared storage
Call an optional plugin method at the end of `phase2`. At that point QEMU has reported the migration as completed, the source is paused and has flushed its block devices, the target hasn't been resumed, and the config hasn't moved yet. That last point means a failure can still fall back to the existing `phase2_cleanup` path, which resumes the source and stops the target.
- Only shared storages whose plugin implements `migration_handover()` are affected. For everything else this is a no-op.
- If the plugin dies, it must have restored the source, so that `phase2_cleanup` can resume it.
- If the plugin can't guarantee that (for example, the target's state is unknown after an SSH timeout), it dies with `{ keep_paused => 1 }`. The source then stays paused instead of risking two writers.
The diff is against qemu-server 9.2.11. The surrounding code is unchanged in 9.2.13.
How a thin-pool plugin would use it (reference implementation, sent to StarWind)
1. Target start (`activate_volume` with `PVE_MIGRATED_FROM` set):
- read a fingerprint of the pool's on-disk superblock (block 0 of `_tmeta`, O_DIRECT, straight from the PV) **before** activating,
- activate the pool with monitoring off,
- reload its table with the dm-thin `read_only` feature. Reads work (QEMU's probes, the OVMF pflash read). Anything that would change the metadata fails instead of corrupting it.
2. Hook, source side: point the paused QEMU's thin LVs at the `error` target, so the file descriptors stay valid but no longer hold the pool. Then remove the pool stack. Suspending the pool commits its metadata a last time, and the source never touches the pool again.
3. Hook, target side (over SSH): compare the fingerprint.
- Unchanged: reload the pool read-write.
- Changed: drop the stale pool, activate it fresh (LVM runs `thin_check`), and `lvchange --refresh` the thin LVs onto it.
- Then verify that the pool is `rw` without `needs_check`.
4. Resume: the normal Proxmox code path.
That implementation is a draft / example. It passes a behavior test against a simulated dmsetup/LVM, but it hasn't been tested on real hardware.
Questions
1. Would you consider a hook like this, or a different mechanism with the same guarantee (a storage callback after convergence, before `cont` on the target)? If so, we'll send it properly to pve-devel?
2. Is there an existing mechanism we've missed that guarantees exclusive activation of a shared volume across a live migration?
3. Is thick LVM with `snapshot-as-volume-chain` the recommended way to get snapshots on shared block storage today? We understand that one avoids this problem, because a thick LV keeps no per-host metadata, but the main disadvantage is snapshots when you have huge disks (10TB+) disks.
We recently lost a VM's disk to thin pool metadata corruption during a routine live migration, and traced it to how a shared thin pool is handed from one node to another. The trigger is the third-party StarWind "StarLVM" storage plugin. The underlying gap, though, is that qemu-server gives a storage plugin no chance to act between "migration completed" and "resume on target". Below is our analysis, and a small proposed qemu-server hook that would allow a storage plugin to do this safely.
Environment
- Proxmox VE 9.2.21 (pve-manager), qemu-server 9.2.11, 4-node cluster
- Shared FC/iSCSI LUN (Fujitsu Eternus) via multipath, one shared LVM VG
- StarWind plugin `starwind-proxmox-plugin-pve9` 1.1.41, storage type `starlvm`, `shared 1`
How StarLVM works
StarLVM adds snapshots to shared block storage. It creates **one dm-thin pool per VM** (`lvmth-<vmid>`) inside the shared VG and puts the VM's disks in it as thin LVs. Because the storage is marked `shared`, Proxmox allows live migration. During a live migration, that thin pool ends up active on the source and the target node at the same time.
What happened (times CEST, node names anonymised)
A rolling update live-migrated the VM from node A to node B. The task log of the migration and the kernel logs of both nodes give this sequence:
Code:
06:53:06 qmigrate starts on node A (source)
06:53:08 "starting VM on remote node B"
06:53:11 node B: thin pool of the VM activated (dm-thin loads its metadata)
06:53:15 RAM transfer starts, the guest keeps running and writing on node A
06:54:10 "migration completed", downtime 140 ms, VM resumes on node B
06:54:12 node A: QEMU process exits
06:54:13 node A: dmeventd "No longer monitoring thin pool ...-tpool" (pool released)
06:55:41 node B kernel:
device-mapper: space map common: metadata block out of bounds
device-mapper: thin: 252:47: metadata operation 'dm_thin_insert_block' failed: error = -22
device-mapper: thin: 252:47: aborting current metadata transaction
device-mapper: thin: 252:47: switching pool to read-only modeFrom then on, every guest write that needed an allocation failed. The guest (Windows Server) ended in Automatic Repair. `lvchange -an` later reported `Integrity check of metadata for pool ... failed`, and we restored the VM from backup.
We ruled out:
- space: the VG had 2.95 TB free, and the pool was at 64% data and 29% metadata,
- snapshots: there were none,
- LVM autoactivation: every pool activation on the nodes matches a migration or `startall` to the second,
- the pool being active on a third node.
The guest was busy during the migration: QEMU had to send 6.4 GiB for 4 GiB of RAM, and the downtime limit was raised automatically.
Root cause
The root cause is that dm-thin is not cluster-aware. A thin pool reads its metadata (superblock, mapping btree, space maps) from disk once, when it is activated, and from then on works from that in-kernel copy, committing it back periodically. Two hosts with the same pool active each have their own copy, and neither ever re-reads what the other wrote. This is why lvmlockd only allows exclusive activation of thin pools, and why Proxmox's own `lvmthin` storage is local-only.
During a live migration, qemu-server does this:
1. Target: `vm_start_nolock` sets `$ENV{PVE_MIGRATED_FROM}` and calls `PVE::Storage::activate_volumes`before QEMU starts, so before the RAM transfer (QemuServer.pm in 9.2.11, around line 5632). For StarLVM, that activates the thin pool on the target, which loads its metadata at that moment.
2. RAM transfer: the guest keeps running on the source, here for about 60 seconds. Whenever it writes to a block that isn't allocated yet, the source's dm-thin allocates it and commits new metadata to disk. The target's in-kernel copy doesn't see any of it.
3. Resume: `phase3_cleanup` sends `resume` to the target (QemuMigrate.pm around line 1749). `vm_resume` (QemuServer/RunState.pm) makes no storage call.
4. Source clean-up: only after that does `vm_stop_cleanup` call `deactivate_volumes` on the source (QemuMigrate.pm around line 1872).
So the target resumes with metadata that is up to a minute old, while the on-disk metadata has moved on. The target's next commit writes its stale view over the source's. That loses the source's allocations from the migration window, and it can hand blocks the source had already used to new writes. Here it surfaced 87 seconds later as an out-of-bounds metadata block. We can't rule out that the same race silently corrupted data on other VMs where it didn't hit the metadata. About 30 other VMs made the same migrations that morning without a visible error, which fits a race that depends on the guest allocating new blocks during the transfer.
Why a storage plugin can't fix this alone
The only safe moment to move a thin pool is after the source has paused and flushed, and before the target resumes. qemu-server calls no storage plugin method in that window:
- activate_volume runs on the target before the RAM transfer,
- deactivate_volume runs on the source after the resume.
A plugin can refuse dual activation, but that rules out live migration altogether. It can't hand the pool over safely.
Proposal: an optional pre-resume hook for shared storage
Call an optional plugin method at the end of `phase2`. At that point QEMU has reported the migration as completed, the source is paused and has flushed its block devices, the target hasn't been resumed, and the config hasn't moved yet. That last point means a failure can still fall back to the existing `phase2_cleanup` path, which resumes the source and stops the target.
- Only shared storages whose plugin implements `migration_handover()` are affected. For everything else this is a no-op.
- If the plugin dies, it must have restored the source, so that `phase2_cleanup` can resume it.
- If the plugin can't guarantee that (for example, the target's state is unknown after an SSH timeout), it dies with `{ keep_paused => 1 }`. The source then stays paused instead of risking two writers.
The diff is against qemu-server 9.2.11. The surrounding code is unchanged in 9.2.13.
Perl:
--- a/src/PVE/QemuMigrate.pm
+++ b/src/PVE/QemuMigrate.pm
@@ -1588,6 +1588,54 @@
die "Failed to complete storage migration: $err\n";
}
}
+
+ $self->storage_migration_handover($vmid);
+}
+
+# The source VM is paused and has flushed its block devices, and the target VM has not been
+# resumed yet. Storage plugins that can't keep a shared volume active on two nodes at once
+# (e.g. a thin pool on shared LVM) hand their volumes over to the target node here.
+#
+# A plugin implements migration_handover($storeid, $scfg, $vmid, $volnames, $target). If it
+# dies, it must have restored the source so the source VM can be resumed (phase2_cleanup does
+# that). If it can't guarantee that, it dies with { msg => ..., keep_paused => 1 } and the
+# source VM stays paused for the admin.
+sub storage_migration_handover {
+ my ($self, $vmid) = @_;
+
+ return if $self->{opts}->{remote};
+
+ my $storecfg = $self->{storecfg};
+ my $volumes = PVE::QemuServer::get_current_vm_volumes($storecfg, $self->{vmconf});
+
+ my $by_storage = {};
+ for my $volid (@$volumes) {
+ my ($storeid, $volname) = PVE::Storage::parse_volume_id($volid, 1);
+ next if !$storeid;
+ my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
+ next if !$scfg->{shared};
+ push $by_storage->{$storeid}->@*, $volname;
+ }
+
+ for my $storeid (sort keys %$by_storage) {
+ my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
+ my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+ next if !$plugin->can('migration_handover');
+
+ $self->log('info', "handing over volumes on storage '$storeid' to $self->{node}");
+ eval {
+ $plugin->migration_handover(
+ $storeid, $scfg, $vmid, $by_storage->{$storeid}, $self->{node},
+ );
+ };
+ if (my $err = $@) {
+ if (ref($err) eq 'HASH') {
+ $self->{vm_was_paused} = 1 if $err->{keep_paused}; # phase2_cleanup won't resume
+ die "storage handover failed - $err->{msg}\n";
+ }
+ die "storage handover failed - $err";
+ }
+ }
}
sub phase2_cleanup {How a thin-pool plugin would use it (reference implementation, sent to StarWind)
1. Target start (`activate_volume` with `PVE_MIGRATED_FROM` set):
- read a fingerprint of the pool's on-disk superblock (block 0 of `_tmeta`, O_DIRECT, straight from the PV) **before** activating,
- activate the pool with monitoring off,
- reload its table with the dm-thin `read_only` feature. Reads work (QEMU's probes, the OVMF pflash read). Anything that would change the metadata fails instead of corrupting it.
2. Hook, source side: point the paused QEMU's thin LVs at the `error` target, so the file descriptors stay valid but no longer hold the pool. Then remove the pool stack. Suspending the pool commits its metadata a last time, and the source never touches the pool again.
3. Hook, target side (over SSH): compare the fingerprint.
- Unchanged: reload the pool read-write.
- Changed: drop the stale pool, activate it fresh (LVM runs `thin_check`), and `lvchange --refresh` the thin LVs onto it.
- Then verify that the pool is `rw` without `needs_check`.
4. Resume: the normal Proxmox code path.
That implementation is a draft / example. It passes a behavior test against a simulated dmsetup/LVM, but it hasn't been tested on real hardware.
Questions
1. Would you consider a hook like this, or a different mechanism with the same guarantee (a storage callback after convergence, before `cont` on the target)? If so, we'll send it properly to pve-devel?
2. Is there an existing mechanism we've missed that guarantees exclusive activation of a shared volume across a live migration?
3. Is thick LVM with `snapshot-as-volume-chain` the recommended way to get snapshots on shared block storage today? We understand that one avoids this problem, because a thick LV keeps no per-host metadata, but the main disadvantage is snapshots when you have huge disks (10TB+) disks.
