Use FC storage in PVE cluster

[alexc]

I used to use standalone PVE hosts, with local attached disks. Nice and tiny setups

Now I have a chance to set up PVE cluster with shared storage. It involves some like 2 groups each consists of "6 hosts and 1 SAN". All hardware to be rented so we can make some adjustments at this point.

The question is, SAN will be with FC connection to hosts, and it sounds reasonable. But as I read https://pve.proxmox.com/wiki/Storage I can see I have no way to have shared storage capable of snapshots. We're not to use it extensively but anyway this is good feature.

So actually I wish to ask you for an advice, what to choose: CIFS as file type storage and qcow2 (so snapshots are possible thanks to qcow2) on top of it, or LVM over FC-based storage (no snapshots at all), or ZFS over iSCSI (seems to be less fast but full of features) - or what? The SAN itself is 3PAR 84xx series, filled with SSDs.

Please advice!

 

[bbgeek17]

If I am not mistaken, your storage does not provide native CIFS, so the only way you can use CIFS is if you front-end the storage with another Windows or Samba server. That means that you have to think about the redundancy of that server - clustering.
You are dealing with PVE ( Linux-based system) - there are no "pros" to using CIFS over NFS. If you decide to front-end your storage with a server, you might as well provide NFS with it. Much less hassle with authentications and compatibility.

You are correct that using LVM over FC, with a requirement to have this storage shared and available to all nodes, will remove Snapshot functionality, as you will be limited to "thick" LVM.

"ZFS over iSCSI" is not possible for you without an extra server, similar to NFS/CIFS. The ZFS daemon/service must be running on the device, and that's not possible directly on 3PAR. You will need a Linux/BSD server, and you will have to figure out how to make that service Highly Available.

When dealing with an external SAN you have to improvise and sometimes sacrifice to get everything or most of what you want (shared+snapshot+ha). PVE, understandably, does not provide a cookie-cutter solution for this.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexc]

If I am not mistaken, your storage does not provide native CIFS, so the only way you can use CIFS is if you front-end the storage with another Windows or Samba server. That means that you have to think about the redundancy of that server - clustering.
You are dealing with PVE ( Linux-based system) - there are no "pros" to using CIFS over NFS. If you decide to front-end your storage with a server, you might as well provide NFS with it. Much less hassle with authentications and compatibility.

You are correct that using LVM over FC, with a requirement to have this storage shared and available to all nodes, will remove Snapshot functionality, as you will be limited to "thick" LVM.

"ZFS over iSCSI" is not possible for you without an extra server, similar to NFS/CIFS. The ZFS daemon/service must be running on the device, and that's not possible directly on 3PAR. You will need a Linux/BSD server, and you will have to figure out how to make that service Highly Available.

When dealing with an external SAN you have to improvise and sometimes sacrifice to get everything or most of what you want (shared+snapshot+ha). PVE, understandably, does not provide a cookie-cutter solution for this.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

You're quite right, storage I listed above is not PVE-specific at its birth so no special optimization was made (poor HPE ). So, the only way for us to play with that SAN is to go with thick LVM (thick itself won't bother), so at least we'll end up with shared storage and we'll be able to create HA.

By the way, may you please be aware what's the price for Blockbridge for Proxmox? You see, we'd like to rent not buy storage so we won't like to pay for software even if we change our mind and rent server full of disks instead of ready-to-go SAN.

So to say, sad to see PVE with all of its history can't handle storage industry standards (like FC) without extra layers like LVM.

 

[bbgeek17]

You're quite right, storage I listed above is not PVE-specific at its birth so no special optimization was made (poor HPE ). So, the only way for us to play with that SAN is to go with thick LVM (thick itself won't bother), so at least we'll end up with shared storage and we'll be able to create HA.

Legacy storage solutions are a poor fit in a modern Service Provider stack. Blockbridge was built as a fully API driven multi-tenant SDS (software-defined storage) that is specifically oriented to Service Provider market. It was quite easy for us to build a Proxmox plugin that simply utilizes our existing cross-platform tools. With PVE having a great underlying storage processing layout and straightforward requirements we just needed to make a wrapper and offload all the intelligence back to the array, where it belongs.

By the way, may you please be aware what's the price for Blockbridge for Proxmox? You see, we'd like to rent not buy storage so we won't like to pay for software even if we change our mind and rent server full of disks instead of ready-to-go SAN.

Blockbridge is a Software-Defined Storage that runs on commodity servers full of disks, creating a ready-to-go modern SAN (iSCSI). It is available as a subscription, so that fits with "rent" intention". Please reach out to us directly to discuss the fit for your project.

So to say, sad to see PVE with all of its history can't handle storage industry standards (like FC) without extra layers like LVM.

The FC portion, as well as multi-path are really handled by the OS. You could also layer one of the many Clustered File Systems on top of the LUNs instead of FC. Of course, the management/set up of this solution is outside of PVE scope.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexc]

Surely new is new, and that’s even greater when it comes to deal with new approaches .
But as I think of server with many disks I start to think that it is less hardware prepared for handling hardware issues and failures with its part. 3par has 2 controllers so when one will fail second one will proceed, and average server has one one motherboard. Surely we can go deeper and rent two server with 2x disks and rent two SDS licenses but… looks like it won’t be easy to handle that and won’t be economically effective.

But what it is the secret way PVE used to work with storage that won’t available for older SANs? Not 3par but many old but good SAN are all about the same, store data and read it back as requested, isn’t it?

 

[alexc]

Hey, LVM is not an option (

We need to replicate guests to another storage (as a HA approach) and LVM over FC won’t let us do that as it has no snapshots, right? Only “local” ZFS capable of, as docs says.

 

[bbgeek17]

Surely new is new, and that’s even greater when it comes to deal with new approaches .
But as I think of server with many disks I start to think that it is less hardware prepared for handling hardware issues and failures with its part. 3par has 2 controllers so when one will fail second one will proceed, and average server has one one motherboard. Surely we can go deeper and rent two server with 2x disks and rent two SDS licenses but… looks like it won’t be easy to handle that and won’t be economically effective.

3par's 2 controllers are just somewhat customized commodity servers. Very few (if any) SAN vendors truly build their own controllers, and even those that do - use commodity parts that are assembled by a custom request. When you look at Dell/EMC or Netapp heads - they are just Intel servers in the end. It's all about software that runs on them.
As for disks - in 3PAR you are using some sort of RAID, so you still need more disks than your actual usage. There is really not that much difference between the two solutions at a very high level. The "devil" is in the implementation. It's very hard for any "custom build" vendor to innovate quickly and take advantage of the latest server architectures, they are always behind. As an example, recent AMD releases have blown Intel away in economy/performance for NVMe storage - only true SDS vendors can take advantage of it. You won't find a 3PAR running on AMD Epyc.

Our recommended solutions contain 2 servers - https://www.blockbridge.com/platforms/. The servers are NVMe chassis that you can fill with non-branded disks as needed, which will cost you 10x less than buying HPE disks. For clarity - the HA system is just one license for us.
It's all about scale - for a home/small user it's not the right solution. If your goal is to provide fast highly available storage with your virtualization - there is a specific customer segment that will be interested.

But what it is the secret way PVE used to work with storage that won’t available for older SANs? Not 3par but many old but good SAN are all about the same, store data and read it back as requested, isn’t it?

There are no secrets, PVE prefers Ceph which is geared for local disks in each node. It comes with its own advantages and disadvantages. Its the primary supported storage solution that PVE stands behind. There are no issues with any SAN per-say. The problem arises with sharing this storage and making the application aware of the shared nature to avoid data corruption. Clustered file systems are one such solution, its just not in the scope of PVE packaging/support.

P.S - yes, throwing replication requirement into the mix makes things even more complicated. I think there is some development around LVM replication in the LVM world. It has not made it over to PVE as far as I know.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox