[PROJECT] SharedLvmThin for Proxmox VE 9 — shared FC/iSCSI SAN storage with LVM-thin snapshots & multipath safety

[delltech1]

Hi everyone,

I am publishing SharedLvmThin, a free and open-source community storage
plugin for Proxmox VE 9.

It provides one LVM-thin pool per VM on an existing shared block-storage
VG and supports snapshots, rollback, resize, migration, cluster locking,
guarded autogrow, storage identity checks, and read-only health diagnostics.

RC5.2 has been tested on the Proxmox VE 9.2.x release line with Storage
API 14 and 15, including three-node cluster qualification. It is currently
a release candidate and should first be validated on disposable storage
matching your production SAN and multipath configuration.

The plugin does not configure the SAN, initialize unknown devices, perform
automatic metadata repair, or replace multipath, fencing, and quorum.

GitHub:
https://github.com/delltech1/proxmox-sharedlvmthin

RC5.2 release and DEB:
https://github.com/delltech1/proxmox-sharedlvmthin/releases/tag/v0.9.0-rc5.2

Feedback, reproducible issue reports, GitHub Stars, and anonymized hardware
compatibility results are welcome.

Work is underway on “Thick Generations” — an experimental companion mode for SharedLvmThin.

The goal is to let administrators use both models side by side:

- the existing space-efficient per-VM LVM-thin mode;
- fully allocated thick VM disks with independent, fully materialized snapshots.

Thick Generations uses dm-clone only during snapshot or rollback transitions. After materialization, every active disk and snapshot becomes an independent ordinary linear LV—without permanent snapshot chains.
The design focuses on deterministic crash recovery, immutable snapshots, strict storage-identity validation and fail-closed handling of ambiguous states.
Initial dm-clone, linear-pivot, reboot-reconstruction and C0–C9 crash-recovery tests are already passing. PVE lifecycle, cross-node, migration, backup/restore, path-loss and thin↔thick qualification are still in progress.
In short: thin and thick storage models living safely side by side, with administrators choosing the appropriate trade-off for each workload.
This remains experimental and is not part of the current public release yet.

 

[Johannes S]

How do you ensure the consistency and that there are no conflicting writes from multiple nodes?

 

[delltech1]

How do you ensure the consistency and that there are no conflicting writes from multiple nodes?

Good question — this is one of the main safety concerns the plugin was designed around.


SharedLvmThin does not rely on multiple PVE nodes independently modifying LVM metadata at the same time.


All shared-LVM metadata mutations are serialized through the Proxmox cluster storage locking mechanism. Before a mutation is allowed, the plugin also revalidates cluster quorum, the expected VG/PV/WWID identity and object ownership while the relevant lock is held.


In practice the model is:

• one shared VG is visible on all participating PVE nodes;
• each VM gets its own thin pool and its own owned LVs;
• LVM metadata-changing operations such as allocation, resize, snapshot, rollback, delete and autogrow are serialized;
• mutation is refused when quorum is unavailable or storage identity/ownership is ambiguous;
• generic LVM autoactivation is disabled for plugin-created pools/LVs, so another node must not independently autoactivate a managed volume;
• the same managed VM disk is not intended to be concurrently writable by multiple VMs/nodes.

Normal guest I/O is different from LVM metadata mutation. Once QEMU has opened the LV, SharedLvmThin is not in the guest I/O path. Proxmox HA/migration/fencing is responsible for ensuring that a VM has a single active owner. A VM must not be started on a second node until the previous owner has been stopped/fenced.


So there are two separate consistency protections:

1. Shared LVM metadata consistency
   Proxmox cluster lock + quorum + identity/ownership checks serialize metadata mutations.
2. Guest data consistency
   A managed LV has a single VM ownership domain; concurrent multi-writer attachment is not supported. HA/fencing must establish that the old VM owner is gone before the VM can start elsewhere.

The plugin deliberately fails closed if it cannot prove these conditions. It does not use lvmlockd, a cluster filesystem, or automatic SAN/LVM repair to guess its way out of an ambiguous state.


We specifically fault-tested concurrent operations such as snapshot/rollback/resize/autogrow and conflicting allocation paths across nodes, including failure while locks are held and quorum/identity failures. The expected result is either one serialized valid mutation or a safe failure with zero conflicting LVM mutation.


This is also why SharedLvmThin requires a properly designed Proxmox quorum/fencing setup for production shared storage.

 

[alexskysilk]

hmm. could this approach be used to facilitate other CoW filesystems (zfs/btrfs?)

--edit on reflection, no. answered my own question but it does seem to be a great option for nvof.

 

[delltech1]

Work is underway on “Thick Generations” — an experimental companion mode for SharedLvmThin.
The goal is to let administrators use both models side by side:
- the existing space-efficient per-VM LVM-thin mode;
- fully allocated thick VM disks with independent, fully materialized snapshots.
Thick Generations uses dm-clone only during snapshot or rollback transitions. After materialization, every active disk and snapshot becomes an independent ordinary linear LV—without permanent snapshot chains.
The design focuses on deterministic crash recovery, immutable snapshots, strict storage-identity validation and fail-closed handling of ambiguous states.
Initial dm-clone, linear-pivot, reboot-reconstruction and C0–C9 crash-recovery tests are already passing. PVE lifecycle, cross-node, migration, backup/restore, path-loss and thin↔thick qualification are still in progress.
In short: thin and thick storage models living safely side by side, with administrators choosing the appropriate trade-off for each workload.
This remains experimental and is not part of the current public release yet.

 

[delltech1]

hmm. could this approach be used to facilitate other CoW filesystems (zfs/btrfs?)

--edit on reflection, no. answered my own question but it does seem to be a great option for nvof.

Yes, exactly. ZFS/Btrfs would be a different architectural model, since they already provide their own CoW and snapshot semantics, so trying to layer this approach on top would mostly defeat the purpose.


NVMe-oF is much more interesting, though. The design itself is intentionally centered around shared block storage rather than a specific SAN transport, so in principle NVMe-oF is a very natural area to explore.


The important distinction is that SharedLvmThin currently qualifies FC/FCoE/iSCSI paths and their associated multipath/failure behaviour. I don't want to claim NVMe-oF support until we have actually qualified the NVMe multipath, namespace identity, path-loss/recovery and PVE lifecycle behaviour properly.


But yes — architecturally I think NVMe-oF is probably one of the most interesting next transport targets for this project, especially with Thick Generations where the steady state is simply independent linear LVs.


Thanks for bringing it up — it's definitely on my radar.