PVE+ISCSI

Jianming

New Member
Oct 30, 2025
16
0
1
Hello Everyone

I have three nodes and Dell ME storage.
ME only use the iSCSI protocol.
if PVE cluster mount iscsi, can not use thin-provision.
Dell ME storage unable to configure ZFS over ISCSI.

So any suggestion ? my cluster mount the iscsi, can use thin-provision and snapshot ?


thanks everyone
 
Hi @Jianming , you can now use PVE with iSCSI storage with LVM overlay and snapshots.
However, thin provisioning will need to come from the array at this point in time.

These articles may be helpful to you:
https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage/
https://kb.blockbridge.com/technote/proxmox-qcow-snapshots-on-lvm/index.html


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
what's the blockbridge ? is it open source ?
 
Has anyone configured GFS2 or OCFS2 ? How are their performances? OCFS2 may be deprecated.

OCFS2 works great and is actively maintained. It serves as a direct replacement for VMFS6 for those who have traditional VMWare + SAN architecture.

https://docs.kernel.org/filesystems/ocfs2.html

Debian has all the packages built into the OS, just have the SAN export the LUNs to the hosts via iSCSI, and setup the hosts correcty with iSCSI and multipathd. Configure O2CB with correct timings (this step is where everyone messes up), then configure a OCFS2 cluster file. Ensure every node has identical o2cb and ocfs2 configuration files (another place people screw up).

Once that's done, it's a simple mkfs.ocfs2 with -N (hosts x 2) -T vmstore.

It's not hard but there isn't any GUI for it and, like all older unix technology, assumes the admin knows what they are doing.

Performance wise it's pretty fast, about ~10% slower then LVM, which is really just a version of raw disk mapping, though I think that is more about qcow2 performance penality.
 
OCFS2 works great and is actively maintained
Is it though? According to @gurubert in https://forum.proxmox.com/threads/p...nd-ocfs2-shared-storage-with-io_uring.140273/ it doesn't seem to be maintained. That thread is older though so things might have changed since them.

Imho I would stick with ProxmoxVE native features because other stuff is unlikely to get support in case of problems. This is especially true if your admins doesn't have that much Linux knowledge to be able to troubleshoot for themselves.
 
Is it though? According to @gurubert in https://forum.proxmox.com/threads/p...nd-ocfs2-shared-storage-with-io_uring.140273/ it doesn't seem to be maintained. That thread is older though so things might have changed since them.

Imho I would stick with ProxmoxVE native features because other stuff is unlikely to get support in case of problems. This is especially true if your admins doesn't have that much Linux knowledge to be able to troubleshoot for themselves.

Yes random guy on forum says it's not for some random reason...

According to the maintainers, the ones maintaining the code, OCFS2 is actively maintained.

Oh the code base for the non-kernel tools used in Debian and RHEL distros.

https://github.com/markfasheh/ocfs2-tools

Last tools release was Oct 2025.

https://github.com/markfasheh/ocfs2-tools/tags

Now Proxmox developers not directly supporting it is another issue, and something that not only bewilders me but actually killed Proxmox as an option for us as it has many other medium sized enterprises.

We are running thirty hosts across five clusters (damn Oracle and it's insane licensing rules) with a large Dell PowerMax / PowerStore FC SAN for storage. The hosts themselves are UCS blades, so they have no local storage and boot off that SAN. This is the prototypical converged / hyper-converged VMWare architecture that every vendor was selling for a long time now. We were already big VMWare customers but with the Broadcom purchase our yearly licensing costs have skyrocketed into truly astronomical territory. We aren't big enough to negotiate with VMWare but aren't small enough to just YOLO it either. While we were searching for a more economical solution, I did a POC using Proxmox on a few spare blades while slicing off some new LUNs and using OCFS2 as a drop in replacement for VMFS6. Did a migration on some of our non-prod environment VM's (our non-prod environment is something like 4x the size of our production environment) and it went smoothly.

When we got down to discussions, IT leadership wasn't going to risk a 35bn USD company on "unsupported" configuration and weren't going to replace the existing physical infrastructure with one that was compliant with the Proxmox developers philosophy of how IT departments should run infrastructure. The final decision was to migrate to Azure instead and rent out Equinix IaaS if we need to while letting the existing hardware depreciate off the books over the next couple of years. Talking to others in the same space I'm in, it's the same decision everywhere. People really want off VmWare and were interested in Proxmox as it's the most "VMWare like" replacement, but then they run into the poor support of SAN storage and zero support of clustered file systems and just decide to migrate to Azure or AWS.

Anyhow yeah off my soapbox. The hardest part of OCFS2 is getting the exact same O2CB and OCFS cluster config file on every host in the cluster, and Proxmox is already running a cluster configuration via corosync. All they would have to do is have the config values used in those files be stored in their internal database and synced as files across all the cluster nodes and it would be a drop-in replacement for VMFS.
 
Last edited:
The hardest part of OCFS2 is getting the exact same O2CB and OCFS cluster config file on every host in the cluster, and Proxmox is already running a cluster configuration via corosync. All they would have to do is have the config values used in those files be stored in their internal database and synced as files across all the cluster nodes and it would be a drop-in replacement for VMFS.
This could also be achieved by utilicing something like rsync to sync the configuration files between the nodes. I'm aware that his won't help in environments who liketo tick compilance checkboxes.

What I don't get however which feature of vmfs you exactly missing? My experience with vmware and vmfs is quite limited (due to my access to our vmware cluster at work is quite limited ) but for me it's just a list of virtual disk images in a webui. This is the same with ProxmoxVE and the virtual discs on a shared storage (whatever that might be). I can see the need for thin-provisioning or snapshots being a showstopper but at least snapshots are now possible with the snapshots-as-chain feature. But whether they live on a vmfs-like filesystem or some other kind of shared storage shouldn't matter if the actual needed usecase (thin-provisoning, snapshots) are possible.

Regarding costs: I can imagine that migrating to azure is cheaper than sticking with vmware. I don't believe that this will stay that way though. At some point Microsoft will raise their fees to squeeze more out of it. Especially if we consider that they invested large sums in AI and need to get these costs back at some point.
 
Last edited:
What I don't get however which feature of vmfs you exactly missing?

Being a cluster aware filesystem for starters. Everything else stems from that requirement.

All the supported volume manager / file systems for Proxmox are non-cluster aware, meaning they intrinsically assume the host has explicate and uncontested access to the hardware. This works great when it's local storage and is disastrously dangerous when it's non-local storage, aka Storage Area Networks. LVM is not cluster aware and neither is ZFS nor XFS or the other common file systems most linux people associate with. Proxmox tries to get around this by coordinating access to LVM volumes internally, cluster nodes talk to each other and say "resource X is using volume Y so don't anyone else attempt to use it".

This is how VMWare use's VMFS, and why they are the industry gold standard in this regard. Inside your SAN you slice off a LUN, which is a storage object that has it's blocks distributed across many disks and managed by the SAN's storage processors. That LUN is then mapped to the physical hosts that are going to use it over Fiber Channel or iSCSI. Every host then see's the LUN has a hard drive. One of the hosts will then format that hard drive with VMFS, which supports multiple hosts accessing it at once. Now every host in the cluster can read and write to the file system at once, sharing information without risk of them writing over each other or corrupting data. This is important because in VMWare your disks are just vmdk files on that shared LUN, vmdk which supports thin provisioning and fast snapshots. KVM's answer to this is QCOW2.

LVM/ZFS/Ceph, all of those storage solutions are for local storage, sometimes called Direct Attached Storage (DAS). Again this is because the Linux community grew up almost entirely in cost conscious environments with cheap hardware, and FC SAN's are expensive and generally existed with Unix like OS's like Solaris, AIX, HPUX, and later VMKernel/Photon OS. Linux would occasionally get a port but they didn't become serious until it started becoming popular as a replacement for those unix OS's.

https://en.wikipedia.org/wiki/GFS2
https://en.wikipedia.org/wiki/OCFS2

OCFS/OCFS2 came into existence because Oracle wanted to start shipping it's big, and incredibly expensive, RAC product on Linux instead of Solaris. RAC requires central shared storage that all the database servers use for their databases as any RAC cluster member can serve up any database. Solaris came with it's own cluster aware file system and so Oracle had to replicate that feature to support Linux, they even released it GNU GPL so as to be accepted into the Linux main kernel, which it was. It is a solution made specifically for compute clusters with shared SAN storage.

https://www.ibm.com/docs/pt-br/linux-on-systems?topic=ocfs2-history

Feature wise, it's supporting snapshots / thin provisioning on shared SAN storage. That is an extremely crucial component of VMWare central architecture and why VMWare become so popular. All the big players support some version of this.

Being a clustered filesystem for starters. All the supported volume manager / file systems for Proxmox are non-cluster aware, meaning they intrinsically assume the host has explicate and uncontested access to the hardware. This works great when it's local storage and is disastrously dangerous when it's non-local storage, aka Storage Area Networks. LVM is not cluster aware and neither is ZFS nor XFS or the other common file systems most linux people associate with. Proxmox tries to get around this by coordinating access to LVM volumes internally, cluster nodes talk to each other and say "resource X is using volume Y so don't anyone else attempt to use it".

This is how VMWare use's VMFS, and why they are the industry gold standard in this regard. Inside your SAN you slice off a LUN, which is a storage object that has it's blocks distributed across many disks and managed by the SAN's storage processors. That LUN is then mapped to the physical hosts that are going to use it over Fiber Channel or iSCSI. Every host then see's the LUN has a hard drive. One of the hosts will then format that hard drive with VMFS, which supports multiple hosts accessing it at once. Now every host in the cluster can read and write to the LUN at once, sharing information without risk of them writing over each other or corrupting data. This is important because in VMWare your disks are just vmdk files on that shared LUN, vmdk which supports thin provisioning and fast snapshots. KVM's answer to this is QCOW2.

LVM/ZFS/Ceph, all of those storage solutions are for local storage, sometimes called Direct Attached Storage (DAS). Again this is because the Linux community grew up almost entirely in cost conscious environments with cheap hardware, and FC SAN's are expensive and generally existed with Unix like OS's like Solaris, AIX, HPUX, and later VMKernel/Photon OS. Linux would occasionally get a port but they didn't become serious until it started becoming popular as a replacement for those Unix OS's.

https://en.wikipedia.org/wiki/GFS2
https://en.wikipedia.org/wiki/OCFS2

OCFS/OCFS2 came into existence because Oracle wanted to start shipping it's big, and incredibly expensive, RAC product on Enterprise Linux instead of Solaris. RAC requires central shared storage that all the database servers use for their databases as any RAC cluster member can serve up any database. Solaris came with it's own cluster aware file system and so Oracle had to replicate that feature to support Linux, they even released it GNU GPL so as to be accepted into the Linux main kernel, which it was. It is a solution made specifically for compute clusters with shared SAN storage.

https://www.ibm.com/docs/pt-br/linux-on-systems?topic=ocfs2-history

Feature wise, it's supporting snapshots / thin provisioning on shared SAN storage. That is an extremely crucial component of VMWare / Virtual Machine centric architecture and why VMWare become so popular. All the big players support some version of this. Your virtual machines and associated configuration are just files on shared disks and any host can pick them up and run them, meaning the individual host is unimportant and can be removed anytime for any reason without risk of data corruption. This is why we keep seeing people ask about it on various forums. That ability is very important for larger organizations that run traditional VM Centric infrastructure, you know the people Broadcom is screwing the hardest.

VMWare does VMFS6
Hyper-V does CSVS
Citrix/XenServer does GFS2 (though XCP-ng can't due to licensing restrictions)

The closest thing Proxmox has is NFS, which isn't something that can be done over FC though can be done over storage VLANs that would otherwise do iSCSI. It also requires the SP support it, which not all do and has some performance considerations as the SP has to assume control over all arbitration.
 
Last edited: