[TUTORIAL] Why LVM-Thin Cannot Be Shared Storage in Proxmox

[bbgeek17]

Hi Everyone,

Every few months someone asks some version of "why can't I use LVM-thin on my shared SAN LUN?" Thick LVM works fine as shared storage, but has no native snapshots. LVM-thin has snapshots but is local-only... it's natural to wonder whether you can have both by putting a thin pool on the LUN and being careful about where it's active.

We looked at this ourselves about half a decade ago and decided it wasn't the right architecture for PVE. For everyone's safety and technical curiosity, we traced the question through DM-thin and the PVE live migration code. It turns out there's a specific mechanical reason it can't easily be made to work.

Here's a write-up that covers where a thin pool keeps its metadata and what happens during a live migration.

Why LVM-Thin Cannot Be Shared Storage in Proxmox:
https://kb.blockbridge.com/technote/proxmox-lvm-thin-shared-storage/

Enjoy,
Blockbridge Team

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[delltech1]

This is a very good point, thanks for sharing this.

I went through the technical details and the overlapping thin-pool activation during PVE live migration is something I need to look at more carefully in my plugin as well. The per-VM thin-pool design limits the failure domain, but that alone doesn’t solve the problem of the same thin pool being active in two kernels during the migration handoff.

I’ll investigate if I can implement a safe exclusive ownership/handoff mechanism for this. Until I can positively prove that, I don’t want to make claims that Thin live migration is safe.

The Thick Generations mode is different because after materialization it uses independent fully allocated linear LVs, so this particular dm-thin issue does not apply there.

Really useful write-up, thanks. This gives me another important case to work on.

 

[bbgeek17]

Hi @delltech1, welcome to the forum. I'm glad you found the post useful. Your continued promotion of your AI plugin here and elsewhere motivated me to share it, because what you're promoting corrupts data.

This is a clear example of how rapid AI-assisted development fails when the architecture isn't understood first. Perfect syntax, thorough documentation, and a plausible answer to every question. The problem is that a model doesn't continually check its premises; it builds on them.

Using dm-thin across nodes requires a strict guarantee that the previous owner is fenced. Not a timer, not a lease, not an assumption... a confirmed kill. This could be a SCSI persistent reservation, an IPMI reset, or both, probably with some out-of-band path to verify it. The reason this doesn't fit PVE is simple: a storage driver shouldn't fence a compute host.

Test this against real hardware, with real workloads, and real failures before promoting it. AI simulations only get you so far when the base assumptions are incomplete. Be mindful that other people's data is not your lab!

PS. I encourage you to interact with the community to get it right. That's what others have done, and the result is the QCOW-on-LVM support that Proxmox now ships. It's not perfect, but many issues were avoided by discussing it with the PVE developers.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

a confirmed kill. This could be a SCSI persistent reservation, an IPMI reset, or both, probably with some out-of-band path to verify it. The reason this doesn't fit PVE is simple: a storage driver shouldn't fence a compute host.

I agree with this sentiment entirely- but it doesnt necessarily mean the conclusion. fencing in this context is just the vg activation on the "source", not full node fence.

Any while I agree that this SHOULDNT be a function of the storage plugin, in context of the solution if it WERE to be it could work. As long as the foundation can be laid out properly it could work reliably. If the plugin is written to a specific storage solution the orchestration can include iqn ACL on/off which would guarantee exclusivity, in addition to something like ssh node1 while true; do #code to remove node1 from store; pvesm if vgchange -an "$VG_NAME"; then exit 0; fi; done (dont hate on my incomplete bash, I'm not actually writing this code )

I dont work enough with LVM thin in any context (since there are better options on a single host and no use in a cluster) to say whether it is more advantageous to pursue lvm thin than LVM thick or a CAF; If (and this is a big if) there is actual benefit to lvm thin over those other options it may be worth pursuing.

 

[delltech1]

Thank you for raising the fencing concern. A few points should be made clear:

1. This is an experimental community project. It is explicitly presented as a release candidate for disposable lab infrastructure and disposable data. It is not advertised as production-ready, certified, commercially supported, or covered by any SLA or warranty.
2. The dm-thin single-kernel requirement is understood. The plugin does not claim that dm-thin is cluster-aware and does not permit intentional simultaneous writable activation of the same thin-pool metadata on multiple hosts.
3. Native online migration of a Thin-backed VM is refused. Safe online movement uses the documented Thin → independent Thick volumes → node migration → optional new Thin pool workflow. The cross-node phase therefore operates on ordinary linear volumes, not shared dm-thin metadata.
4. FIX2 and the current hardening add fail-closed checks. Activation requires durable ownership evidence, audits every configured peer, revalidates the owner and epoch immediately before lvchange -ay, and detects the expected device-mapper UUID even when the mapper appears under an alternative name. Missing, unreachable, malformed, stale, or conflicting evidence blocks activation.
5. We tested the concern rather than dismissing it. The current candidate passed targeted conflict and race tests, 1,004 automated assertions, rolling installation and reboot on a disposable three-node PVE cluster, native Thin migration refusal, Thick activation, and a Thin → Thick → Thin bridge test. The final inventory contained 58 active Thin pools with zero duplicate dm-thin UUIDs across the three kernels.
6. These results define a tested envelope, not a universal safety proof. They do not certify every SAN, kernel, LVM/device-mapper version, multipath policy, latency profile, workload, scale, update, or failure sequence. Storage performance and failure behavior must be qualified on representative disposable infrastructure.
7. The project does not replace compute fencing. A storage plugin cannot prove that a failed host is physically dead in every possible failure mode. Sites requiring automatic takeover must provide and validate suitable PVE HA fencing and out-of-band infrastructure. A timeout, lease, SSH failure, or owner tag alone is never treated as proof that the previous kernel can no longer write.
8. The operational boundary is explicit. Users must maintain tested independent backups, recovery procedures, fencing, quorum, stable storage identity, bounded all-path-loss behavior, and their own change control. Root-level manual actions can bypass any userspace protection.
9. No support or production guarantee is offered. This is a GPLv3 community project supplied “as is”. There is no paid or unpaid support commitment, SLA, certification, warranty, update schedule, or obligation to investigate or fix an issue. Community interaction is voluntary.

If you have a reproducible failure against the current FIX2 candidate—exact version, topology, commands, timing and observed device-mapper state—we are interested in testing it. A general warning about dm-thin is valid, but a claim that this implementation corrupts data should be supported by a reproducible case or an identified path that bypasses the documented guards.