VMFS equivalent in Proxmox? Shared iSCSI filesystem across multiple nodes

B

bazigan

New Member
Jun 4, 2024
8
1
3
West Jakarta, Indonesia
bazigantsamara.my.id
Hi everyone,


I’m trying to understand how Proxmox handles shared block storage compared to VMware VMFS, and I’d like to ask for some clarification or best practices.


In VMware, VMFS allows multiple ESXi hosts to mount the same LUN concurrently with read/write access, since it is a cluster-aware filesystem.


In Proxmox, when using iSCSI:


  • If I format the LUN with ext4 or XFS, it can only be safely mounted by a single host at a time.
  • Mounting the same ext4/XFS filesystem on multiple Proxmox nodes is not possible (or risks corruption), because those filesystems are not cluster-aware.

I understand this is expected behavior, but it raises a few questions for me:


  1. Is there any VMFS-like equivalent in Proxmox (i.e. a cluster-aware filesystem for shared block storage)?
  2. Are there any ongoing developments or plans in Proxmox for such a filesystem?
  3. Besides Ceph (RBD / CephFS), are there any recommended approaches to achieve shared iSCSI storage across multiple nodes?
  4. Would cluster filesystems like OCFS2 or GFS2 be considered viable or supported in production with Proxmox?
  5. NFS works for shared storage, but in my case performance becomes an issue — are there recommended optimizations or alternatives in this scenario?

I’m mainly looking for a shared storage solution that:


  • Can be accessed by multiple Proxmox nodes simultaneously
  • Supports live migration
  • Offers better performance than traditional NFS

Any insights, recommendations, or references would be greatly appreciated.


Thanks in advance!
 
We are running a cluster of 8 Proxmox VE nodes with OCFS2 on a SAN LUN since several years.
It works but it is not officially supported. There were some issues even with OCFS2 in recent kernels as the development of OCFS2 seems to have come to a halt.

The performance of OCFS2 is very good when doing IO in a VM. When using thin provisioned QCOW2 images the expansion of the image file can take some time as OCFS2 has to sync the newly allocated blocks across all PVE nodes.

We tried GFS2 but found that it performs bad in comparison to OCFS2.
 
Last edited:
  • Like
Reactions: Johannes S
bazigan said:
Is there any VMFS-like equivalent in Proxmox (i.e. a cluster-aware filesystem for shared block storage)?
Click to expand...
With PVE on a shared LUN you use LVM to split it up. That is a block storage.

See https://pve.proxmox.com/wiki/Multipath that also touches the final setup after multipathing (which you will usually have in such setps).

bazigan said:
Are there any ongoing developments or plans in Proxmox for such a filesystem?
Click to expand...
No, that is a complex problem to solve. It would take up quite some development resources and time until it matured enough.

bazigan said:
Besides Ceph (RBD / CephFS), are there any recommended approaches to achieve shared iSCSI storage across multiple nodes?
Click to expand...
Don't use Ceph on top of iSCSI/FC/SAS where you again have a central storage box providing the actual storage. That kinda defeats the redundancy idea of Ceph.

bazigan said:
Would cluster filesystems like OCFS2 or GFS2 be considered viable or supported in production with Proxmox?
Click to expand...
We looked into them and they all had things why we decided against supporting them officially.
@gurubert already mentioned it, OCFS2 tends to run into issues with newer kernels.

GFS2 behaves quite different than Proxmox VE if the Corosync connection fails even for a brief moment. As in, the host needs a full reboot. That is a bit too brittle since a PVE node itself can handle a few seconds of no Corosync connection quite fine.
 
Last edited:
  • Like
Reactions: Johannes S and gurubert
  • Like
Reactions: bbgeek17
bazigan said:
Besides Ceph (RBD / CephFS), are there any recommended approaches to achieve shared iSCSI storage across multiple nodes?
Click to expand...
There are commercial vendors, including ours, that provide stable enterprise-grade storage solutions fully integrated with Proxmox VE.

Whether these platforms are a good fit depends on your specific requirements, including workload characteristics, existing infrastructure, operational preferences, and budget.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
You must log in or register to reply here.
Top Bottom