SAN iSCSI/LVM Big VG - Best practices

[baalkor]

Hello,

Quick question : Are there some best practices regarding LUN sizing/splitting regarding a iSCSI/LVM volume group that would host around 50TiB of machine's disk.
I've head that it would be better to split the big volume group in multiple disk of 10TB or 4TB to maximize performances.

However I do believes this is true I'm unable to find any documentation about where does "the rule of splitting big volume in multiple LUNs to be aggregated in a LVM" come from.

Do you have any idea ?

Sincerely,

 

[bbgeek17]

Hello @baalkor, this really depends on the capabilities of your backend storage and your specific configuration. Personally, I would consider multiple LVM pools if:

• You are dealing with very high-performance storage, where LVM itself can be a bottleneck.
• You are dealing with very low-performance storage, where isolation of backend RAID groups could improve IOPS.
• You expect to have a large number of virtual machines, where contention on a shared SCSI task queue could be an issue.

Some reference material that may be useful for you:
https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[baalkor]

Hello @bbgeek17 ,

Yes in our case it is a all NVME flash SAN array (Dell PowerStore).
Looking at the provided documentation, I do not see any thing regarding this topic.

 

[alexskysilk]

"Best Practices" in this context are largely dependent on your use case.. While performance is an important part of your strategy, its not the sole objective.

As you should be aware, LVM on shared storage is supported on lvm thick only. Snapshot facility is possible at the QEMU level but it incurs a substantial performance cost. If you wish to use your storage native snapshots, this only way this works is if you can coordinate the guest quiescence with snapshot. This is only practical if you have individual LUNs set for individual VMs. In a PVE environment, doing this is very cumbersome and hard to manage- and you'll still need to write your own snapshot orchestration.

On the other hard, making a single/few LUNs with multiple VMs is a lot easier to manage.

Another possibility is to map iscsi luns directly inside the guest (eg, small boot disk on shared LUN, and payload on an iscsi mount) although I'm not 100% sure this will live migrate (@bbgeek17 ?)

 

[bbgeek17]

Looking at the provided documentation, I do not see any thing regarding this topic.

Hi Baalkor,

There’s no documentation on this topic because it’s highly specific to the environment. You’ll need to try it out and see what works best for you. Another option is to ask your storage vendor for recommendation. Keep in mind that PVE does not include compatibility/supportability matrix for 3rd party storage providers. Its up to them to test and make recommendations.

A PowerStore sits right in the middle of the performance spectrum, so you generally shouldn’t need to split the volume group. That said, if you don’t mind the extra management overhead, there’s no harm in doing so!

Another possibility is to map iscsi luns directly inside the guest (eg, small boot disk on shared LUN, and payload on an iscsi mount) although I'm not 100% sure this will live migrate

If the connection is done within the context of the VM - yes it will migrate along with all other VM state. However, that will circumvent the block-efficient VirtIO path in QEMU and move the storage access to Network virtio, which may present other inefficiencies. It would also increase management overhead, and necessitate placing VMs on storage network which may not always be desirable.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox