[TUTORIAL] Why LVM-Thin Cannot Be Shared Storage in Proxmox

[bbgeek17]

Hi Everyone,

Every few months someone asks some version of "why can't I use LVM-thin on my shared SAN LUN?" Thick LVM works fine as shared storage, but has no native snapshots. LVM-thin has snapshots but is local-only... it's natural to wonder whether you can have both by putting a thin pool on the LUN and being careful about where it's active.

We looked at this ourselves about half a decade ago and decided it wasn't the right architecture for PVE. For everyone's safety and technical curiosity, we traced the question through DM-thin and the PVE live migration code. It turns out there's a specific mechanical reason it can't easily be made to work.

Here's a write-up that covers where a thin pool keeps its metadata and what happens during a live migration.

Why LVM-Thin Cannot Be Shared Storage in Proxmox:
https://kb.blockbridge.com/technote/proxmox-lvm-thin-shared-storage/

Enjoy,
Blockbridge Team

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[delltech1]

This is a very good point, thanks for sharing this.

I went through the technical details and the overlapping thin-pool activation during PVE live migration is something I need to look at more carefully in my plugin as well. The per-VM thin-pool design limits the failure domain, but that alone doesn’t solve the problem of the same thin pool being active in two kernels during the migration handoff.

I’ll investigate if I can implement a safe exclusive ownership/handoff mechanism for this. Until I can positively prove that, I don’t want to make claims that Thin live migration is safe.

The Thick Generations mode is different because after materialization it uses independent fully allocated linear LVs, so this particular dm-thin issue does not apply there.

Really useful write-up, thanks. This gives me another important case to work on.

 

[bbgeek17]

Hi @delltech1, welcome to the forum. I'm glad you found the post useful. Your continued promotion of your AI plugin here and elsewhere motivated me to share it, because what you're promoting corrupts data.

This is a clear example of how rapid AI-assisted development fails when the architecture isn't understood first. Perfect syntax, thorough documentation, and a plausible answer to every question. The problem is that a model doesn't continually check its premises; it builds on them.

Using dm-thin across nodes requires a strict guarantee that the previous owner is fenced. Not a timer, not a lease, not an assumption... a confirmed kill. This could be a SCSI persistent reservation, an IPMI reset, or both, probably with some out-of-band path to verify it. The reason this doesn't fit PVE is simple: a storage driver shouldn't fence a compute host.

Test this against real hardware, with real workloads, and real failures before promoting it. AI simulations only get you so far when the base assumptions are incomplete. Be mindful that other people's data is not your lab!

PS. I encourage you to interact with the community to get it right. That's what others have done, and the result is the QCOW-on-LVM support that Proxmox now ships. It's not perfect, but many issues were avoided by discussing it with the PVE developers.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

a confirmed kill. This could be a SCSI persistent reservation, an IPMI reset, or both, probably with some out-of-band path to verify it. The reason this doesn't fit PVE is simple: a storage driver shouldn't fence a compute host.

I agree with this sentiment entirely- but it doesnt necessarily mean the conclusion. fencing in this context is just the vg activation on the "source", not full node fence.

Any while I agree that this SHOULDNT be a function of the storage plugin, in context of the solution if it WERE to be it could work. As long as the foundation can be laid out properly it could work reliably. If the plugin is written to a specific storage solution the orchestration can include iqn ACL on/off which would guarantee exclusivity, in addition to something like ssh node1 while true; do #code to remove node1 from store; pvesm if vgchange -an "$VG_NAME"; then exit 0; fi; done (dont hate on my incomplete bash, I'm not actually writing this code )

I dont work enough with LVM thin in any context (since there are better options on a single host and no use in a cluster) to say whether it is more advantageous to pursue lvm thin than LVM thick or a CAF; If (and this is a big if) there is actual benefit to lvm thin over those other options it may be worth pursuing.