Proxmox, Fibre Channel, and 3PAR

[B. Weishoff]

Hello.

So I've gone through what I think are the docs regarding PROXMOX and fibre channel presented LUN's. I'm using an older 3PAR 7200 in my lab, attempting to get shared storage between a PROXMOX cluster. So far, I have a fully operational cluster with a shared NFS volume, which performs as expected. The system can migrate to all PVE hosts, etc.

My question is this, with vmware, I can use the 3PAR with fiber presented LUNs as a shared filesystem. I'm a bit confused with the docs as to weather this can be accomplished with PROX. Am I just barking up the wrong tree, or am I SOL?

I can see the paths to the storage, but my understanding is you need to use multipath-tools and LVM, which is allegedly supported for shared storage in a cluster ? I walked through setting up multipath.conf, then used LVM to create a group & volume, and then laid down an ext4 filesystem on said volume, however, PROX doesn't seem to want to let me actually use it for guest systems? Is there some kind of trick to getting this to work ?

Thanks in advance.

-b

 

[bbgeek17]

My question is this, with vmware, I can use the 3PAR with fiber presented LUNs as a shared filesystem. I'm a bit confused with the docs as to weather this can be accomplished with PROX. Am I just barking up the wrong tree, or am I SOL?

Yes, you can do this but the concepts, technologies and functionality is different.
In VMware you have a proprietary shared clustered filesystem VMFS, that allows you to create VMDK files and safely access them from multiple hosts at the same time.
In Proxmox there is no native/built-in equivalent. You have a few options:
- shared LVM. Its an adopted technology that will carve up your LUN into predictable regions (hence no thin provisioning) and where Proxmox will control and arbitrate which host in the cluster has access to it. There is no multi-host access to the same region.
- Clustered Filesystem. You can use one of the Open Source variants, however the installation, configuration and support is on your own.
- Various options of putting a more flexible head on top of your FC LUNs that will present iSCSI, iSCSI/ZFS, NVMe/TCP back to Proxmox.
- Use direct LUN approach where you dedicate a LUN to a specific VM virtual disk (image). This does not scale well in large/complex environments.

can see the paths to the storage, but my understanding is you need to use multipath-tools and LVM

Yes, if you want path redundancy you need to configure Multipath. PVE is based on Debian with Ubuntu Kernel. Configuring multipath is a standard procedure, non-PVE specific.

then used LVM to create a group & volume

yes, correct. Ensure you use correct variant of LVM

and then laid down an ext4 filesystem on said volume, however, PROX doesn't seem to want to let me actually use it for guest systems? Is there some kind of trick to getting this to work ?

No, no filesystem is involved. You feed PVE the Volume Group in the storage configuration and PVE will auto-magically carve up LV slices from it and pass-through to VM as raw devices. What you do with these raw devices inside your VM is up to you.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[B. Weishoff]

Yes, you can do this but the concepts, technologies and functionality is different.
In VMware you have a proprietary shared clustered filesystem VMFS, that allows you to create VMDK files and safely access them from multiple hosts at the same time.
In Proxmox there is no native/built-in equivalent. You have a few options:
- shared LVM. Its an adopted technology that will carve up your LUN into predictable regions (hence no thin provisioning) and where Proxmox will control and arbitrate which host in the cluster has access to it. There is no multi-host access to the same region.
- Clustered Filesystem. You can use one of the Open Source variants, however the installation, configuration and support is on your own.
- Various options of putting a more flexible head on top of your FC LUNs that will present iSCSI, iSCSI/ZFS, NVMe/TCP back to Proxmox.
- Use direct LUN approach where you dedicate a LUN to a specific VM virtual disk (image). This does not scale well in large/complex environments.


Yes, if you want path redundancy you need to configure Multipath. PVE is based on Debian with Ubuntu Kernel. Configuring multipath is a standard procedure, non-PVE specific.

yes, correct. Ensure you use correct variant of LVM

No, no filesystem is involved. You feed PVE the Volume Group in the storage configuration and PVE will auto-magically carve up LV slices from it and pass-through to VM as raw devices. What you do with these raw devices inside your VM is up to you.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

It's amazing -- 5 minutes after I posted this, I figured it out. Thank you, though, @bbgeek17 , for your prompt reply!

So for those that may have the same question -- present the luns, get them configured using the multipathd, create a new volume group using the standard LVM procedure, but don't put a filesystem down. Once you've done this, simple use the PVE storage manager app at the CLI to add the storage (make sure all your nodes are online and see the storage, etc.). Easy-peasy.

Thanks all.

-B.

 

[bbgeek17]

Once you've done this, simple use the PVE storage manager app at the CLI to add the storage (make sure all your nodes are online and see the storage, etc.)

Keep in mind that in addition to thin provisioning you will also be missing out on Proxmox integrated snapshots, or any snapshots in most cases.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

For a possible solution to the snapshot problem, please see this similar question answered a few days ago.