[PROJECT] BASTRIX SharedLVM for Proxmox VE 9 — FC/iSCSI Storage with Thin Pools, Thick Generations and Multipath Safety

[delltech1]

Hi everyone,

I am publishing BASTRIX SharedLVM, a free and open-source community storage
plugin for Proxmox VE 9.

The project provides two explicitly selectable allocation modes over an
existing dedicated shared LVM volume_STA group:

• Thin
One LVM-thin pool per VM, providing space-efficient allocation, snapshots,
rollback and guarded autogrow.

• Thick Generations
Fully allocated VM disks with independent, fully materialized snapshot
generations and deterministic physical-space reservation.

Both modes are included in the same Debian package. They can be configured as
separate Proxmox storage IDs over the same pinned shared VG and used side by
side. An ordinary Proxmox Storage Move can convert a disk from Thin to Thick
Generations or in the opposite direction.

Thick Generations uses persistent dm-clone only as a temporary transition
mechanism during snapshot and rollback materialization. Once materialization
finishes, the active disk and every retained snapshot are independent ordinary
linear LVs. There is no permanent snapshot or backing-volume chain in the
steady-state guest data path.

RC5.4 TG12 has been qualified on the Proxmox VE 9.2.x release line with Storage
API 14 and 15. Qualification included:

• Thin and Thick allocation, snapshot, rollback, resize and deletion;
• cross-node reconstruction and live migration;
• Thin-to-Thick and Thick-to-Thin Storage Move with block-hash verification;
• native Proxmox backup and cross-mode restore;
• Veeam HotAdd backup of Thin, Thick and mixed guests;
• supported-console Veeam restores to Thin, Thick and mixed layouts;
• two-node and three-node quorum and fail-closed behavior;
• installation and reinstallation across a three-node API 14/15 cluster;
• crash and partial-transaction recovery testing;
• a clean four-hour dual-mode endurance run;
• 156 Python tests and 236 Perl tests;
• reproducible Debian package builds.

The design remains deliberately safety-focused:

• storage identity, ownership, quorum and cluster locking are positively
verified before mutation;
• missing storage is treated as UNAVAILABLE, never as empty;
• foreign, legacy, unknown or ambiguous objects are not adopted or deleted;
• partial failures preserve data rather than attempting speculative cleanup;
• unknown block devices are never initialized automatically;
• the plugin performs no automatic metadata repair.

The plugin does not configure SAN connectivity, multipath, fencing or quorum.
It does not replace the storage array's availability and data-protection
mechanisms. Existing guest I/O during complete path loss remains governed by
QEMU, device-mapper, multipath and the underlying SAN.

RC5.4 TG12 is still a pre-release. It should first be validated on disposable
storage matching the intended production SAN, HBA, firmware, multipath and
failure policies. Multipathed iSCSI and a virtual Linux FCoE laboratory target
were exercised; representative physical enterprise FC hardware was not
available and is therefore not claimed as qualified.

GitHub:
https://github.com/delltech1/proxmox-sharedlvmthin

RC5.4 TG12 release and DEB:
https://github.com/delltech1/proxmox-sharedlvmthin/releases/tag/v0.9.0-rc5.4-tg12

Allocation modes:
https://github.com/delltech1/proxmo...k-generations/docs/allocationocation-modes.md

Installation guide:
https://github.com/delltech1/proxmo...mental/thick-generations/docs/installation.md

Feedback, reproducible issue reports, GitHub Stars and anonymized compatibility
results for Proxmox VE, SAN arrays, HBAs, multipath configurations and firmware
versions are very welcome.

Please do not include credentials, private IP addresses, WWIDs or other
sensitive infrastructure identifiers in public reports.

 

[Johannes S]

How do you ensure the consistency and that there are no conflicting writes from multiple nodes?

 

[delltech1]

How do you ensure the consistency and that there are no conflicting writes from multiple nodes?

Good question — this is one of the main safety concerns the plugin was designed around.


SharedLvmThin does not rely on multiple PVE nodes independently modifying LVM metadata at the same time.


All shared-LVM metadata mutations are serialized through the Proxmox cluster storage locking mechanism. Before a mutation is allowed, the plugin also revalidates cluster quorum, the expected VG/PV/WWID identity and object ownership while the relevant lock is held.


In practice the model is:

• one shared VG is visible on all participating PVE nodes;
• each VM gets its own thin pool and its own owned LVs;
• LVM metadata-changing operations such as allocation, resize, snapshot, rollback, delete and autogrow are serialized;
• mutation is refused when quorum is unavailable or storage identity/ownership is ambiguous;
• generic LVM autoactivation is disabled for plugin-created pools/LVs, so another node must not independently autoactivate a managed volume;
• the same managed VM disk is not intended to be concurrently writable by multiple VMs/nodes.

Normal guest I/O is different from LVM metadata mutation. Once QEMU has opened the LV, SharedLvmThin is not in the guest I/O path. Proxmox HA/migration/fencing is responsible for ensuring that a VM has a single active owner. A VM must not be started on a second node until the previous owner has been stopped/fenced.


So there are two separate consistency protections:

1. Shared LVM metadata consistency
   Proxmox cluster lock + quorum + identity/ownership checks serialize metadata mutations.
2. Guest data consistency
   A managed LV has a single VM ownership domain; concurrent multi-writer attachment is not supported. HA/fencing must establish that the old VM owner is gone before the VM can start elsewhere.

The plugin deliberately fails closed if it cannot prove these conditions. It does not use lvmlockd, a cluster filesystem, or automatic SAN/LVM repair to guess its way out of an ambiguous state.


We specifically fault-tested concurrent operations such as snapshot/rollback/resize/autogrow and conflicting allocation paths across nodes, including failure while locks are held and quorum/identity failures. The expected result is either one serialized valid mutation or a safe failure with zero conflicting LVM mutation.


This is also why SharedLvmThin requires a properly designed Proxmox quorum/fencing setup for production shared storage.

 

[alexskysilk]

hmm. could this approach be used to facilitate other CoW filesystems (zfs/btrfs?)

--edit on reflection, no. answered my own question but it does seem to be a great option for nvof.

 

[delltech1]

Work is underway on “Thick Generations” — an experimental companion mode for SharedLvmThin.
The goal is to let administrators use both models side by side:
- the existing space-efficient per-VM LVM-thin mode;
- fully allocated thick VM disks with independent, fully materialized snapshots.
Thick Generations uses dm-clone only during snapshot or rollback transitions. After materialization, every active disk and snapshot becomes an independent ordinary linear LV—without permanent snapshot chains.
The design focuses on deterministic crash recovery, immutable snapshots, strict storage-identity validation and fail-closed handling of ambiguous states.
Initial dm-clone, linear-pivot, reboot-reconstruction and C0–C9 crash-recovery tests are already passing. PVE lifecycle, cross-node, migration, backup/restore, path-loss and thin↔thick qualification are still in progress.
In short: thin and thick storage models living safely side by side, with administrators choosing the appropriate trade-off for each workload.
This remains experimental and is not part of the current public release yet.

 

[delltech1]

hmm. could this approach be used to facilitate other CoW filesystems (zfs/btrfs?)

--edit on reflection, no. answered my own question but it does seem to be a great option for nvof.

Yes, exactly. ZFS/Btrfs would be a different architectural model, since they already provide their own CoW and snapshot semantics, so trying to layer this approach on top would mostly defeat the purpose.


NVMe-oF is much more interesting, though. The design itself is intentionally centered around shared block storage rather than a specific SAN transport, so in principle NVMe-oF is a very natural area to explore.


The important distinction is that SharedLvmThin currently qualifies FC/FCoE/iSCSI paths and their associated multipath/failure behaviour. I don't want to claim NVMe-oF support until we have actually qualified the NVMe multipath, namespace identity, path-loss/recovery and PVE lifecycle behaviour properly.


But yes — architecturally I think NVMe-oF is probably one of the most interesting next transport targets for this project, especially with Thick Generations where the steady state is simply independent linear LVs.


Thanks for bringing it up — it's definitely on my radar.

 

[alma21]

- fully allocated thick VM disks with independent, fully materialized snapshots.
Thick Generations uses dm-clone only during snapshot or rollback transitions. After materialization, every active disk and snapshot becomes an independent ordinary linear LV—without permanent snapshot chains.

how should this work in more detail ?
- on thick LVs are the snapshots then also initially standard/thick ones with all the performance impact ?
- will dm-clone convert them meanwhile in the background to full/independent "snapshot" clone LVs ? is the snapshot afterwards deleted to return to native thick performance ?
- the dm-clone target is not implemented in LVM .... this would mean native device mapper handling on top of LVM , or ?
thanks

 

[delltech1]

how should this work in more detail ?
- on thick LVs are the snapshots then also initially standard/thick ones with all the performance impact ?
- will dm-clone convert them meanwhile in the background to full/independent "snapshot" clone LVs ? is the snapshot afterwards deleted to return to native thick performance ?
- the dm-clone target is not implemented in LVM .... this would mean native device mapper handling on top of LVM , or ?
thanks

Yes, broadly that is the intended model.
Each disk generation is a fully allocated LVM LV. When a snapshot is created, the current generation becomes the immutable snapshot and a new fully allocated writable generation is created.

dm-clone is used only temporarily during this transition. It provides the writable view immediately while unchanged regions are copied in the background. During hydration there is some additional metadata and copy I/O overhead.

After hydration completes, the stable device-mapper frontend is atomically changed back to a normal dm-linear mapping pointing only to the new generation. The temporary clone mapping and its metadata LV are removed. The immutable snapshot LV remains as an independent fully allocated snapshot until the administrator deletes it.

So the intended steady state is:
QEMU → stable mapper → dm-linear → independent thick LV

dm-clone is not managed natively by LVM. The prototype therefore uses LVM for LV allocation, identity and persistent transaction metadata, while the temporary clone/linear mappings are managed directly through device-mapper.

The trade-off is temporary hydration overhead and additional capacity usage in exchange for no permanent snapshot chain and ordinary linear performance after materialization. This is still being qualified as an experimental implementation, so some lifecycle details may change before release.

 

[delltech1]

New: RC5.4 TG12 dual-mode preview is available. One package now offers the established Thin mode and the new Thick Generations mode with fully allocated, independent generation LVs and a linear steady-state data path. Thin and Thick storage definitions can coexist over the same pinned VG, and ordinary PVE Storage Move converts disks in either direction.

 

[delltech1]

Project update — RC5.4 TG24 milestone

A newer experimental release, RC5.4 TG24, is now available. It combines the established per-VM Thin mode with the optional Thick Generations mode and has reached an important qualification milestone.

Thick Generations explores a different trade-off from the Proxmox VE 9 snapshot-as-volume-chain preview:

PVE volume chains:
separate layered qcow2 volumes → active backing chain

BASTRIX Thick Generations:
temporary dm-clone transition → fully materialized independent linear LV
PVE volume chains prioritize fast and space-efficient snapshots. Thick Generations requires additional capacity and background materialization time, but completed snapshots and active generations do not retain a permanent backing chain. This is an architectural comparison, not a performance claim.

TG24 adds and qualifies:

• Thin and Thick modes within the same shared VG;
• snapshot, rollback, resize and multi-disk lifecycle handling;
• Thin-to-Thick and Thick-to-Thin Storage Move;
• migration, native HA relocation and backup/restore workflows;
• transaction-scoped persistent recovery metadata;
• bounded installation preflight with full read-only diagnostics available separately;
• concurrent qualification with 150 Thin and 150 Thick test VMs.

TG24 remains a pre-release and is not a universal SAN or enterprise-scale certification. Please test it first on disposable storage matching your intended SAN, multipath, fencing and workload configuration.

Release and DEB:

https://github.com/delltech1/proxmox-sharedlvmthin/releases/tag/v0.9.0-rc5.4-tg24

Updated documentation:

https://github.com/delltech1/proxmox-sharedlvmthin

Feedback and sanitized hardware compatibility results are very welcome. Please do not publish private addresses, credentials, WWIDs or organization-specific storage identities.

 

[delltech1]

Update: RC5.4.1 TG25 is now available

TG25 is a maintenance and compatibility hotfix for the TG24 dual-mode milestone. It preserves the existing Thin and Thick Generations storage formats while improving PVE package-update, initramfs and post-reboot recovery safety.

It also adds the mandatory read-only sharedlvmthin compat-check for validating Storage API compatibility, required commands, services, quorum, initramfs contents and storage recovery state.

The release was tested with Proxmox VE 9 Storage API 14 and 15, including package upgrades, cold reboots, Thin/Thick lifecycle operations and cross-node migrations. It remains a release candidate—please validate it on disposable storage matching your SAN and multipath configuration.

Release and DEB:
https://github.com/delltech1/proxmox-sharedlvmthin/releases/tag/v0.9.0-rc5.4.1-tg25

Installation guide:
https://github.com/delltech1/proxmox-sharedlvmthin/blob/main/docs/installation.md

Feedback and hardware compatibility reports are welcome.