Spanning Shared LVM over multiple PVs/LUNs from central storage appliance

[Cookiefamily]

Hello,

we have some Dell Storage Appliances that we would like to use with Proxmox. They export iSCSI LUNs and we would like to use the LVM (thick) shared storage with them.
Dell wants you to create smaller LUNs (maybe 2TiB or so) but a lot of them to be able to balance the load between the storage controllers and for them to use their own IO queues. This results in better performance.

You could create a separate LVM shared storage for each LUN. But when you go down that route there comes a lot of manual overhead with it, balancing VMs between lots of different PVE storages... playing tetris with VM storage once you want to resize a disk. Doesn't sound too fun.

A colleague suggested just creating a PV per LUN and adding them all to one VG that hosts the LVM shared storage. This would get rid of the administrative overhead and once enough storage from PV00 is used it would overflow to PV01 and spread the load across multiple LUNs.

Something with that idea doesn't really sit right with me... Thats why I wanted to ask in here: Has someone done this before? Are there caveats and pitfalls in there that you might not immediately think off?

 

[LnxBil]

Something with that idea doesn't really sit right with me... Thats why I wanted to ask in here: Has someone done this before? Are there caveats and pitfalls in there that you might not immediately think off?

Actually, I've never done it the other way around. LVM was explicitly created for such setups where you bind together multiple LUNs into ONE volume group and then carve out volumes from it. This is the way to go and the way it has always been. I'm using LVM in this setup for over 20 years.

Keep in mind that the data is just plugged right behind each other, so that the LVM fills from the front to the back, not in any other way, so it'll fill the first LUN, then the seconds... If you want to rebalance it, you need to move extents around, which is doable, but manual intervention.

 

[bbgeek17]

Dell wants you to create smaller LUNs (maybe 2TiB or so) but a lot of them to be able to balance the load between the storage controllers and for them to use their own IO queues. This results in better performance.

The historical pitfall with this design is : splitting LUNs across SP A/SP B improves parallelism and queue depth, but if each SP is already running near its own capacity, a failure/trespass event on one SP means the survivor has to absorb both workloads.
Most arrays aren't sized to run at 2x a single controller's normal load without a performance hit. You need to provision each SP with enough headroom (not run both near 100%) so that a single-SP failure doesn't cause a bigger outage than the hardware failure itself.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox