[SOLVED] Shared LVM on Dell ME5024 — best practice for snapshot capability alongside HA?

[danielt]

Migrating from vSphere 8 (Essentials Plus) to Proxmox VE 9.2. Looking for input on how others have handled the snapshot question on block-only SAN storage.


Environment

• 5 nodes, 768 GB RAM each: 1× R650xs (Xeon 5318Y), 2× R7615 (EPYC 9174F), 2× DL360 Gen10 (Xeon 6226R)
• Dell ME5024, SAS-attached, dual controller, 2 initiators per host
• 4 LUNs of ~4.4 TiB each across two pools
• Local storage per node: hardware RAID 5 (HPE Smart Array / Dell PERC) + LVM-thin
• Cluster currently 3 nodes, quorate; ~44 VMs still to migrate off two remaining ESXi hosts
• PBS 4.2 with an S3 backend (AWS, via VPC PrivateLink)

Current shared storage config


One LUN built as thick LVM, --shared 1, mapped to a host group on the array. Multipath per the Dell guide (DellEMC/ME5, ALUA, failback immediate). Working correctly — two paths, prio 50/10.


The problem


LVM-thin can't be shared, so shared storage means thick LVM, which means no snapshots. I understand the reason — static extent allocation is what makes it safe for multiple hosts, and it's also what prevents copy-on-write.


Options I've evaluated and why each doesn't fit:

• snapshot-as-volume-chain — full-size LV per snapshot, so a 200 GB VM needs another 200 GB. Two of our VMs are 2.6 TB and 5.8 TB and couldn't be snapshotted at all. Also qcow2-on-LVM overhead applies permanently, and it's still a technology preview in 9.2.
• ZFS over iSCSI — needs an appliance that creates zvols on demand; the ME5024 presents fixed LUNs only.
• Ceph — needs raw disks rather than LUNs, and 3× replication of our ~20 TB local capacity leaves too little.
• NFS — the ME5024 has no NAS capability.

Where I've landed


Splitting the array into two tiers: some LUNs as thick LVM shared (HA, live migration, raw disks) and one LUN as LVM-thin mapped to a single node (snapshots, raw, full speed). VMs placed per requirement. PBS as the actual protection mechanism, plus the ME5024's own volume-level snapshots before batch maintenance.


Questions

1. Is the two-tier split what others do with block SANs, or is there an approach I've missed?
2. Anyone running snapshot-as-volume-chain in production yet? Real-world experience on the space and performance cost, and on rollback-to-most-recent-only?
3. Any reason not to mix thick-shared and thin-single-node LUNs from the same array in one cluster?
4. Is there a supported way to get shared storage plus efficient snapshots from a block-only array that I'm not seeing?

 

[LnxBil]

Is the two-tier split what others do with block SANs, or is there an approach I've missed?

In the past, we did the same, but with thick-LVM of all storage space and use the ZFS-over-iSCSI appliance for all stuff, that needs a snapshot. We migrated the VM online to the other storage, did the intended stuff and migrate it back.

Any reason not to mix thick-shared and thin-single-node LUNs from the same array in one cluster?

IIRC, problem with LVM is that you should not manually play around with lvm commands in order to have a properly working thick LVM. In older versions of PVE (e.g. 5, maybe 6), there was cluster lock daemon that ensured consistent metadata, nowadays it's just the PVE way, but only as long as you don't change stuff manually. I would NOT mix it and go with the ZFS-over-iSCSI-appliance.

Is there a supported way to get shared storage plus efficient snapshots from a block-only array that I'm not seeing?

Only with vendor support or on your own with a cluster filesystem and using qcow2 files, but that has it's own drawbacks and is not officially supported, but you can have a lot of problems with it.

Anyone running snapshot-as-volume-chain in production yet? Real-world experience on the space and performance cost, and on rollback-to-most-recent-only?

Works OK with small VMs but the performance impact is not negligible in most cases. You have to really want to have snapshots so that I would use it. PBS backup is a good mitigation for missing LVM-thick snapshots. I would only want to use this if you have proper thin-provisioned storage backend so that you can overprovision the LUN on the SAN side for best space usage (with all of the drawbacks there...)

 

[danielt]

In the past, we did the same, but with thick-LVM of all storage space and use the ZFS-over-iSCSI appliance for all stuff, that needs a snapshot. We migrated the VM online to the other storage, did the intended stuff and migrate it back.


IIRC, problem with LVM is that you should not manually play around with lvm commands in order to have a properly working thick LVM. In older versions of PVE (e.g. 5, maybe 6), there was cluster lock daemon that ensured consistent metadata, nowadays it's just the PVE way, but only as long as you don't change stuff manually. I would NOT mix it and go with the ZFS-over-iSCSI-appliance.


Only with vendor support or on your own with a cluster filesystem and using qcow2 files, but that has it's own drawbacks and is not officially supported, but you can have a lot of problems with it.


Works OK with small VMs but the performance impact is not negligible in most cases. You have to really want to have snapshots so that I would use it. PBS backup is a good mitigation for missing LVM-thick snapshots. I would only want to use this if you have proper thin-provisioned storage backend so that you can overprovision the LUN on the SAN side for best space usage (with all of the drawbacks there...)

Thank you very much it was very helpfull.