Proxmox in enterprice environments with shared fc storage

[TheBigOne]

I'm curious why there's no native support for traditional cluster filesystems like GFS2 or OCFS2.

I fully get why Ceph is essential in hyper-converged setups, but it's impractical for synchronously replicated FC storage. It would bring IO overhead, complexity and capacity overhead.
Shared LVM only works with complex placement logic and heavy storage overcommit at the block level.

On top of that, shared LVM snapshots are still in tech preview, and each snapshot reserves space equal to the VM's full disk size. If I have ten 1 TB VMs on a datastore, keeping 3 snapshots each means reserving 40 TB.

Is there a specific reason why cluster filesystems comparable to VMFS aren't supported?

Is there anyone in the community with an enterprise setup and shared, sync replicated storage? How did you implement proxmox in a production environment?
Did I muss anything? I could not find any useful solution for FC Blockstorage.
LVM Shared with a big single VG would have 1PB and would be a bit risky in striped mode.

 

[LnxBil]

Shared LVM only works with complex placement logic

You mean in a stripped setup? Yes, I agree. In a filesystem, you have this too, but don't see it (see next answer)

Is there a specific reason why cluster filesystems comparable to VMFS aren't supported?

As someone who has worked with both of them in the past, I can say that they have a lot of drawbacks, one is performance (which VMFS also suffers from). Blockstorage without additional layers that complicate stuff with access times, concurrent locks and inherit fragmentation is faster and therefore prefered. Both cluster filesystems can be very complicated to setup and are not as stable as LVM has been for decades, but for me was it mainly the performance.

LVM Shared with a big single VG would have 1PB and would be a bit risky in striped mode.

Why? What do you fear? LVM is battle tested for decades, not with PB but with sizes comparable of magnitude decades ago.

EDIT: spelling

 

[TheBigOne]

When I add 64 PVEs each 64TB to one single VG, I`am afraid that it may be a longer downtime in case of any issue. A single PVE down will cause the full VG to crash. This can happen not only when a SAN Issue occours, but only in high load scenarios.
Also the VG Cluster locking during snapshots or lvm changes with hundrets of VMs may cause lock timeouts.
When we start patching, we snapshot all VMs at a time. Backups are also creating snapshots with locks and lvm a lot of metadata updates.

Each cluster would have a 4PB Volumegroup. I agree that LVM works fine, but my personal experiences are only locally on VMs with much smaller VGs.

Another problem is the read-ahead on the storage array may break, if the data of a single lvm is split to several PVs.
I`am thinking about queue-depth, queue trashing and invalidation. Everything would inrease latency on all read and writes, caused by one vm on a lun with a full queue.

I cannot see any performance benefits, except lvm placement is easier.

EDITS: typos and details

 

[LnxBil]

When I add 64 PVEs each 64TB to one single VG, I`am afraid that it may be a longer downtime in case of any issue. A single PVE down will cause the full VG to crash.
Also the VG Cluster locking during snapshots or lvm changes with hundrets of VMs may cause lock timeouts.
When we start patching, we snapshot all VMs at a time. Backups are also creating snapshots with locks and lvm a lot of metadata updates.

Snapshotting shared thick LVM? Yes, you would need LVM locking, which was required in the past and you will have the problems you talked about. AFAIK, newer PVE versions do not require the locking daemon if and only if you do all metadata changing calls through the PVE api, so that it will take care of the locking issues.

Another problem is the read-ahead on the storage array may break, if the data of a single lvm is split to several PVs.
I`am thinking about queue-depth, queue trashing and invalidation. Everything would inrease latency on all read and writes, caused by one vm on a lun with a full queue.

I cannot see any performance benefits, except lvm placement is easier.

Thanks for the explanation, read ahead could be a problem in deed. Manual rebalancing to have a proper RAID0 like striping is also a manual task, but at least online doable. Yet I do see the same kind of problems also with a filesystem, so this may still be valid after all. At the top of my head, I don't think that any filesystem can do something better than a proper volume manager can besides handling files and their permissions.

 

[TheBigOne]

Thanks for your explanation. I guess that I will do a lot of tests before we think about production usage. Since we automate everything, an ansible module will calculate the placement.
Striping has some advantages and disadvantages. Its not difficult to configure but in production I prefer to reduce complexity, risk and take first the performance into account.

Do you know when the LVM Shared Snapshot feature moves from Preview State to final?
BTW: Thanks for the hint to the locking deamon.

 

[Ali-dev]

Hi @TheBigOne,

Coming from VMware/ESXi (VMFS), this is definitely one of the biggest architectural shifts when moving to Proxmox in enterprise environments with SAN/FC storage.

The main reason PVE relies on LVM-thin / Shared LVM rather than clustering filesystems like GFS2 or OCFS2 out of the box is maintenance overhead and locking complexity. GFS2/OCFS2 require strict DLM (Distributed Lock Manager) and fencing setups, which historically have been notorious for split-brain lockups and node fencing crashes under heavy I/O in multi-node clusters.

In enterprise setups with sync-replicated FC (NetApp / Dell PowerStore / Pure):
- Many deploy Shared LVM on top of multipathed FC LUNs for base storage, keeping disk allocation thick/pre-allocated per VM.
- Or for snapshot flexibility without Ceph, running NFS/SMB gateways or active-active file heads on the SAN array presenting shared storage to Proxmox.

Are your SAN arrays capable of presenting NFS over 10/25G alongside FC, or are you strictly tied to FC block LUNs?

 

[TheBigOne]

Yes, FC has some major benefits to NAS. We use FC with GAD Volumes on Hitachi G5600 and Hitachis NAS Heads can not manage the same IO Loads as the block storage and it would put massive loads on the LAN. So it would be also a big shift in the datacenter layout when you move from FC to LAN. Just the LAN cards, Cisco hardware and cables would be a big change to save several 100K on license costs for VMware per year because not all workloads support proxmox or other solutions.

Adding software NFS heads on top (like Nutanix), would also be an overhead. When you have several Datacenter with G5600, You don`t think about to change on Hypervisor and Storage and Network at the same time
It would be easier to migrate everything to AWS and Azure and Google Cloud

Thin provisioning on lvm would be crazy, as long as there is no feature like sdrs but its anyways not working on shared storage.

I agree with the locking and split-brain issues on any FS cluster solution, which is also an issue with ceph. If you do not work with a 3 DC SAN layout.
The major benefit of VMware was the VMFS with Cluster Heartbeat on Datastores to prevent split-brain and I could not find any solution yet, which is comparible.

Thats why I put some effort to the ansbile module for disk placement to use LVM shared thick provisioning.
I just want to be sure that I did not miss a possible good solution.

 

[alma21]

LVM Shared with a big single VG would have 1PB and would be a bit risky in striped mode.

I would use multiple VGs/Storage Pools to reduce the failure domain

 

[alma21]

When I add 64 PVEs each 64TB to one single VG,

add-on: afaik the shared LVM/VG boundary is the cluster ... so a shared LVM VG is always dedicated to only one PVE cluster

 

[TheBigOne]

add-on: afaik the shared LVM/VG boundary is the cluster ... so a shared LVM VG is always dedicated to only one PVE cluster

Yes, I know. But an 8 or 10 Node Cluster can manage >500 VMs with several TB Disks and Memory. When you need to keep 3 Snapshots plus the Memory, then 1 PB is not much

I see now that there is not really an alternative to the shared LVM. Maybe a placement logic can be implemented to the software later. For now Ansible will do the job. I do not expect big issues, since VM creation is anyway a Task which can take some minutes without any problems.

Thanks to all

 

[bbgeek17]

@TheBigOne short answer: you didn't miss anything. There is no VMFS for Linux and GFS2/OCFS2 aren't it.

VMFS did one job and VMware controlled everything around it. GFS2 and OCFS2 are mid-2000s designs for small database clusters on spinning disk. Not 64 nodes on a petabyte of FC. Both bring their own lock manager and fencing. Those have to live next to Corosync and PVE's own fencing. Two systems deciding who is alive is how you end up with a dead node still holding locks. OCFS2 is in maintenance mode. GFS2 assumes RHEL and Pacemaker. This forum already has plenty of qcow2-on-cluster-FS corruption threads. Proxmox is right not to ship either. (btw, this is also why third-party HCI on PVE isn't a smart idea. If you're going to do HCI, stick with Ceph. Proxmox integrated it.)

Since you asked if anyone has solved this: we scoped a VMFS replacement for PVE earlier this year and shelved the full version. It is not a technical problem. With the right expertise it costs a few million to build something you'd trust with production data and a few million more to do proper interop testing against every array vendor. The problem is it competes with shared LVM which is free. Hard to sell a filesystem when the alternative costs nothing and most PVE people will take the tradeoffs (heck, most people in this forum don't even buy PVE licenses). There aren't enough shops like yours to make the math work. We may still build a stripped-down piece of it for our own use cases, but I doubt we'll sell it as a product.

For your setup, thick shared LVM on multipathed FC is the right call. Use several VGs instead of one so a problem stays contained, and keep all metadata changes going through the PVE API.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alma21]

and GFS2/OCFS2 aren't it.

... although OCFS2 has some nice features - with reflink in combination with raw (sparse) files you get FS native/clustered thin provisioning and snapshots ... and maybe raw is also not so vulnerable as qcow2-on-cluster-FS .

 

[bbgeek17]

although OCFS2 has some nice features - with reflink in combination with raw (sparse) files you get FS native/clustered thin provisioning and snapshots ... and maybe raw is also not so vulnerable as qcow2-on-cluster-FS

Fair point. Reflink with sparse raw files gets you thin provisioning and snapshots without qcow2... so qcow2 metadata is out of the picture.

What's left is the stuff underneath. o2cb has its own heartbeat and fencing that doesn't know corosync exists. Getting it working is simple. Maintaining it, qualifying it on every kernel change, and supporting it through upgrades, network partitions, and failure scenarios is where it gets fun.

I have absolutely nothing against OCFS2. It's a solid CFS with the grey hairs to prove it. The problem is running a cluster in a cluster (without explicitly designing the system for it)... just because you can doesn't mean you should.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Johannes S]

I have absolutely nothing against OCFS2. It's a solid CFS with the grey hairs to prove it. The problem is running a cluster in a cluster (without explicitly designing the system for it)... just because you can doesn't mean you should.

Or to be more blunt: Just because want everything to work like vmware that doesn't mean that it makes sense. ProxmoxVE and the underlying technologies are after all a different system with it's own tradeoffs, pros and cons.

 

[alma21]

What's left is the stuff underneath. o2cb has its own heartbeat and fencing that doesn't know corosync exists. Getting it working is simple. Maintaining it, qualifying it on every kernel change, and supporting it through upgrades, network partitions, and failure scenarios is where it gets fun.

maybe this was the reason why SUSE/SLES supported only OCFS2 in combination with pacemaker/corosync & kernel DLM .... I'm not a cluster stack / FS developer but imho why not switching the pacemaker CRM to the PVE cluster manager and provide better PVE integration/config .... I know already discussed in many posts ...

 

[bbgeek17]

maybe this was the reason why SUSE/SLES supported only OCFS2 in combination with pacemaker/corosync & kernel DLM .

You've got it. That's exactly why SUSE put OCFS2 on corosync, kernel DLM and Pacemaker. One membership, one fencing authority.

As for doing the same on PVE... someone still has to pay to develop, debug, and support it. Not my circus not my monkeys

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox