cetrafs — Open-source shared-disk filesystem for Proxmox VE

C

ctrix

New Member
Oct 22, 2026
5
3
3
Hi everyone,

I've been working on cetrafs, a GPL-licensed Linux clustered filesystem designed for shared VM datastores, somewhat similar to VMware's VMFS.

The project actually started because I had some storage hardware originally purchased for another hypervisor vendor, which then decided to raise its prices outrageously. I wanted to keep using my existing infrastructure without being locked into that ecosystem, and needed a viable alternative.

It lets multiple Proxmox nodes mount the same iSCSI or FC LUN concurrently, using SCSI atomic operations for coordination instead of a distributed lock manager.

I've tested it on Proxmox clusters using two different storage setups:

  • A Synology NAS over iSCSI.
  • An HP ClusterOne SAN over Fibre Channel with multipath.
I've also documented Proxmox live migration workflows.

It's still beta software, and I'd be particularly interested in feedback from people familiar with Proxmox storage internals or shared SAN environments.

Project: https://cetrafs.com/

Source: https://github.com/cetrafs/cetrafs
 
  • Like
Reactions: gyptazy and EllerholdAG
Nice to see that you put effort (or tokens) into this and I really appreciate it, but I have to say, I would NEVER use this in production at the current stage without proper support. Both are nice projects and I often wanted to write something like this myself, but I know enough about filesystem that I would never try it myself, not even with AI. Just look at filesystems like ZFS with a huge developer and user base and what strange bugs get fixed like finally the zfs encryption and send/receive corruption fix I just learned about yesterday in 2.3.3. ZFS is also "just" a local filesystem, not a clustered one. Trusting a filesystem with data needs also to have combined myriads of time of experience with it and also the trust in the codebase and acceptance by other "filesystem people" to start with. If a filesystem is vetted and included into the mainline kernel, I would consider using it. The other way would be the support of a company behind it that I can pay for solving problems, at least in the enterprise environment.

Also, just having something that works, does not mean you can use it in production. I started using OCFS2 over 15 years ago, it was complicated to setup and maintain and not very fast, but it had the backing of a smallish company named Oracle behind it and they later got it into the mainline kernel. I'm not using it today because it wasn't up to the job I needed it to do - maybe it is now, but I doubt it or it would be used more in PVE and you would not have started to write your own filesystem. Oracle switched their focus on their other cluster filesystem ACFS that I use daily and in many Oracle database clusters. If even a huge company and the kernel developers could not provide a proper cluster filesystem that is accepted widely, I doubt a single developer can build something even comparable.

I don't want to discourage you, I just want to ground you somehow that this is a nice starting point to build something worthwhile, but it's a VERY long way to go. Maybe I'm completely wrong with my assessment with the advent of AI filesystem development and the extremely complicated environment it has to work in. I would be very happy to be proven wrong, so that we would have a working cluster filesystem besides the more or less working solutions we have for at least a decade.
 
  • Like
Reactions: Johannes S
While I agree with LnxBil, this process is exactly how new software is written.
Enough users want something (a cluster filesystem) so that a company or a Dev writes it and is looking for testers, supporters or even backers.
Yes at the moment this should be considered experimental BUT it can grow to something beautiful. By reading this forum this is much needed and will find a lot of users. I'd argue that home labbers are the perfect user group for this. They are experimenting with new stuff, they are willing for something to break and they often times have older hardware that triggers all kinds of bugs that newer enterprise grade servers don't have.

If this runs well, maybe a huge company (e. G. Proxmox) could pick it up and develop it further! Or it can go into the Linux kernel...

So I'd really like you to succeed (both cetrafs and meshxfs)!
 
  • Like
Reactions: gyptazy
Thanks to both of you for the thoughtful feedback. I think both perspectives are valuable, although I personally lean more towards the second one: being open to experimentation, trying new things, and helping projects move forward.

First of all, I'm not claiming CetraFS is production-ready. There's a project status page that explicitly describes it as experimental. I want people to know exactly what they're getting into.

That said, this isn't just something that happens to work in a lab. CetraFS currently has almost 1,500 tests, a fuzzing suite, and a regression testing suite. Every bug discovered gets a dedicated test to make sure it doesn't come back. Testing doesn't guarantee correctness, especially for a clustered filesystem, but it's a fundamental part of how I'm developing it.

Is it immature? Absolutely. Does that mean it's not worth trying? I don't think so.

I believe that with the right mindset, a willingness to experiment, and constructive feedback from the community, this project can grow into something genuinely useful.

And that's exactly what I'm hoping for at this stage: people willing to try it, push it to its limits, break things, report bugs, and share their experiences. That's the only way to make real progress.
 
  • Like
Reactions: EllerholdAG
ctrix said:
Thanks to both of you for the thoughtful feedback. I think both perspectives are valuable, although I personally lean more towards the second one: being open to experimentation, trying new things, and helping projects move forward.
Click to expand...

That's a nice idea if you happen to develop video games, a office suite or CAD software (aka any software, where new features are the most important thing). It's not so nice if we are talking about a file system which you want to be able to trust with your data ( or anything else related to infrastructure).

This is true in homelabs too: Allthough peoples tax records or family pictures are not really important for the state of the world they still mean the world to them and don't want to loose it.

The attitude ("let's move things forward") is the reason bcachefs was excluded from the Linux kernel mainline after it's first inclusion and is in general a red flag to me.
So I sincerely wish to express my thanks to @LnxBil for taking the time to write out his concerns. I also had doubts but neither the time nor the knowledge and experience to write them out in a concise format. And I didn't want to just vent out to the AI usage since I'm aware that my AI-luddite attitute isn't an argument per se since (like prooven by several open source projects) a competent engineer can do great things with AI (the ecological and social costs of AI are another matter but doesn't have anything to do with it's merits for software development).
 
Last edited:
I think you're taking this a bit too seriously. It's a personal project, the author clearly states that it's experimental, and he's just looking for people willing to test it and give feedback.

Of course nobody should blindly trust a new filesystem with important data, especially in production. But there's a big difference between saying "I'd never use this in production yet" and being so negative about the whole idea.

Every project has to start somewhere. If nobody is willing to experiment and help out, nothing new will ever get better. You don't have to use it, and nobody is asking you to. Let the guy work on his project, see how it develops, and judge it by the results.

Honestly, I think it's great that someone is putting time and effort into building something the Proxmox community could benefit from.
 
  • Like
Reactions: waltar
xybo said:
I think you're taking this a bit too seriously. It's a personal project, the author clearly states that it's experimental, and he's just looking for people willing to test it and give feedback.
Click to expand...

The trouble is, that a lot of people who come from vmware and basically want to have the same workflows are looking into things like ocfs as possible alternatives. Many of them don't even encounter the forum threads where people like @LnxBil or @gurubert report on the caveats you need to know to make it work or consider another storage backend depending on the usecase.
And ocfs is at least a lot more mature than the vmfs-on-xfs projects. So: Yes, even if it's just a "personal project" people will try to use it for important data. So I think it's fair to have some concerns and share them. It's better to have too much warning than the opposite. Because like with other non-supported stuff at some point somebody will come and ask how he can restore his family pictures/tax records/production database he used the experimental stuff with.

xybo said:
Of course nobody should blindly trust a new filesystem with important data, especially in production. But there's a big difference between saying "I'd never use this in production yet" and being so negative about the whole idea.
Click to expand...

I'm in general negative to experimental stuff for my data, yes.
 
Johannes S said:
I'm in general negative to experimental stuff for my data, yes.
Click to expand...

I'm not asking you to do it.
And yes, you should be as cautious as possible with your data, i agree that it's the way to go.

Again: data is yours, you're free do do what you want with them.
I would just appreciate you not to shit on someone else work without having at least tried it or understanding what it is

(and BTW, this is not VMFW over XFS, that one is another thread)
 
Johannes S said:
The trouble is, that a lot of people who come from vmware and basically want to have the same workflows are looking into things like ocfs as possible alternatives. Many of them don't even encounter the forum threads where people like @LnxBil or @gurubert report on the caveats you need to know to make it work or consider another storage backend depending on the usecase.
And ocfs is at least a lot more mature than the vmfs-on-xfs projects. So: Yes, even if it's just a "personal project" people will try to use it for important data. So I think it's fair to have some concerns and share them. It's better to have too much warning than the opposite. Because like with other non-supported stuff at some point somebody will come and ask how he can restore his family pictures/tax records/production database he used the experimental stuff with.



I'm in general negative to experimental stuff for my data, yes.
Click to expand...
I agree with Johannes on one point: nobody should entrust their only copy of important data to an experimental filesystem (you should never only have one copy). That's not what ctrix and I are asking anyone to do.

But there's a difference between being cautious about deploying new software and dismissing it before it's been properly evaluated. We're asking people to test these projects in environments where failures are acceptable, put them through their paces, and report what they find.

A filesystem doesn't become reliable simply by accumulating years of existence or by having a big company behind it. It becomes reliable through engineering, extensive testing, independent review, finding and fixing bugs, and validating those fixes. Time and real-world experience matter, too, but they aren't substitutes for that work.

Both CetraFS and MeshXFS are tackling a real need in the Proxmox community. Let's evaluate them on their technical merits and work to make them better.
 
melkier said:
Thiis is a good idea - i've added it as a possible inclusion in MeshXFS
Click to expand...
it's more something in qemu blockdev side, so it need patch on qemu-server. I'll try to see if it's improve performance when allocating blocks with for meshxfs && cetrafs.

I don't known if somebody already done fio benchmark ?

I had pretty poor results with ocfs2 && gfs2 in the past.
 
aderumier said:
it's more something in qemu blockdev side, so it need patch on qemu-server. I'll try to see if it's improve performance when allocating blocks with for meshxfs && cetrafs.

I don't known if somebody already done fio benchmark ?

I had pretty poor results with ocfs2 && gfs2 in the past.
Click to expand...
https://github.com/sshoecraft/meshxfs/tree/main/bench

I'll be doing comparison testing next week. The results show aggregates as the node grows but the storage is coming from a local nvme drive presented to a qemu vm. here's the rough plan:

Setups

  1. QNAP, 1G, physical: 2 and 4 nodes. Lock and iSCSI traffic share the single link (disclose this).
  2. Local SCST: 2, 4, 8 and 16 VM nodes, with blockio or fileio using o_direct=1 write_through=1, a working set larger than host RAM, and lock traffic on its own virtio bridge.

Filesystems

  • Native XFS on one node, as the reference row.
  • MeshXFS, in both lock modes where the target supports them: SCSI CAW and TCP lock network. Check whether the QNAP supports CAW.
  • GFS2 with DLM, corosync/pacemaker and fencing configured.
  • OCFS2, with the stack stated (o2cb or pcmk).
  • Same kernel on every node, fresh mkfs on the same LUN for each filesystem, and default mount options.

Workloads

  1. Per-node file fio: your existing suite, the uncontended baseline.
  2. Shared-file fio: all nodes doing 4K random reads and writes to one file, both overlapping and striped offsets.
  3. mdtest in one shared directory: create, stat, unlink, from all nodes at once. This is the main comparison.
  4. mdtest with a private directory per node: shows locking cost without contention.
  5. Real-world mix: parallel kernel-tree untar and rm -rf, plus find/ls -l storms on one tree.
  6. Optional, recovery: fence one node mid-workload and measure how long the survivors stall and when throughput returns.

Procedure

  • Drop caches on the nodes and the target between runs.
  • Run each test 3 times and report the median.
  • Capture throughput, IOPS and ops/sec, plus p50, p99 and max latency.
  • Script it into the existing harness so all four filesystems run identically, and log the configuration with each result.

Reporting

  • One table per setup: filesystem × node count × workload.
  • Charts of ops/sec against node count for the shared-directory mdtest and the shared-file fio. Those are the headline results.
  • State the limits up front: single 1G link, QNAP model, and VM caveats for the local runs.

I'll update the bench readme when done. I'll do the same for certa - but its a diff test because its targeted at VMFS capability (no concurrent writes to the same files/dirs).
 
@melkier a small clarification: cetrafs does allow concurrent read and write access to the same directory from multiple nodes. By default, the only thing it prevents is concurrent writes to the same file.

Default mode. When the filesystem is mounted without options, cetrafs takes a cross-node exclusive lock on every file opened for writing. Any other node that tries to open that file gets ETXTBSY. This is the safe default, and it matches how VMFS behaves.

nofilelock mode. The downside of the default is that it blocks live migration. During a live migration in proxmox, QEMU needs the image open on both the source and the destination at the same time. To support that, cetrafs has a relaxed mode, enabled by mounting with nofilelock (the mode applies to the whole cluster). In this mode, several nodes can open the same file for writing at the same time. That can cause corruption or data loss if nothing above the filesystem keeps writers apart, which is why it is not the default.

In practice:
  • If you don’t need live migration, mount without options and keep the automatic lock. Offline migration works fine.
  • If you do need it, use nofilelock. Preventing double writes then becomes the job of QEMU and Proxmox HA, the same split of responsibilities as vSphere HA on top of VMFS.
Even under nofilelock, a process can still take the same exclusive cross-node lock on a single file with flock(2), for example from a migration hook.

While looking into this, I just found out that QEMU already has the right mechanism: its image locking uses OFD fcntl byte-range locks, so a running VM holds the image exclusively, and QEMU hands the lock over transparently during live migration (locking protocol). Today those locks are only visible on the local node. If cetrafs propagates them across the cluster, running VMs would be protected under nofilelock while still allowing live migration, with no changes on the QEMU or Proxmox side. I’m going to try to implement this.

On performance: I haven’t run fio yet, so I’m looking forward to melkier’s comparison. My dd tests, with detect_zeroes set to zero are very encouraging. Details are on the Performance model page.

More in the docs: Enable live migration · Locking model
 
Excited for the effort! I dont have any spare hardware to devote to either project but I may just scrounge to make a lab. Question to both- how hard a dependency on sbp for locking control? could this be adapted to nvof?

Johannes S said:
It's better to have too much warning than the opposite. Because like with other non-supported stuff at some point somebody will come and ask how he can restore his family pictures/tax records/production database he used the experimental stuff with.
Click to expand...
There is a philosophical conundrum here. how much effort is incumbent upon a publisher of software that is licensed FREE NO OBLIGATION OF SUPPORT? Unless the vendor is a financially sound and contractually obligated to maintain an SLA, what difference does their "production grade" claims have in either direction? should they require a "sign here to attest you'd not an idiot" before allowing download?

Storage is the most sensitive component in IT. If I am not 100% confident in SOMEONE's ability and commitment to support my edge cases I'm not touching it. I dont have much sympathy to people who jump first and then check the water depth, regardless of what the sign said (or didnt.) I dont think neither @ctrix nor @melkier would be offended if I said I wouldnt touch either of their solutions for real- but I'm quite excited at the effort for both.
 
  • Like
Reactions: Johannes S
Hi ctrix, hi melkier,
I've been following this thread with a lot of interest. I work in the infrastructure team of a French university hospital, and like many people coming from vSphere, I really miss a VMFS-like shared filesystem in Proxmox VE. Shared LVM does the job, but it's not the same thing, and backup is a good example. In a Veeam environment, VMs on shared LVM require an additional file-level storage (NFS or similar) to hold snapshots during backups. That means extra infrastructure to size, secure and maintain, outside the SAN we already have. A shared file-level datastore directly on the SAN could remove that constraint, provided Veeam can work with it.
I agree with the concerns raised above: nobody should put production data on an experimental clustered filesystem. That's actually why I'd like to help. We have enterprise storage and servers that we can dedicate to an isolated test environment, which is probably what both projects need most right now: real arrays, real hardware, and people willing to break things and report back.

What we could offer:

Storage: Huawei OceanStor Dorado arrays (including an active-active HyperMetro setup across two sites) and NetApp ASA, over FC and/or iSCSI with dm-multipath
Several physical servers running Proxmox VE
CAW support results for these arrays (caw_probe / sg_vpd output)
Performance benchmarks with fio, comparing cetrafs and MeshXFS against shared LVM-thick, with and without snapshots, using a common methodology so results are comparable (happy to reuse melkier's bench plan)
Resilience testing with vdbench: we already run a vdbench cluster that proved very useful to validate our storage during outages and failovers. Its data validation mode is well suited to catching silent corruption on a clustered filesystem while things are failing underneath
Failure scenarios: path loss and multipath failover (no_path_retry vs stale_secs), controller failover, HyperMetro site failover, node crash and fencing, live migration under load
Veeam backup and restore testing. As far as I know, Veeam's Proxmox plugin doesn't support custom storage types today. Would mounting cetrafs outside PVE and declaring it as a standard shared "dir" storage be a viable workaround, or would that break assumptions the plugin relies on (nofilelock, mount handling)?
Detailed bug reports with logs, and a public write-up of the results
Before we start, it would help to know what would be most useful to you: which versions to test, which scenarios you haven't been able to cover yet, and how you'd like the results reported (GitHub issues, this thread, or something else).
The goal is to give both projects solid data from enterprise hardware, not to rush anything into production. Let me know if you're interested.
Best regards,
Thibaut
 
  • Like
Reactions: Onslow
Hi ctrix, hi melkier,

I've been following this thread with a lot of interest. I work in the infrastructure team of a French university hospital, and like many people coming from vSphere, I really miss a VMFS-like shared filesystem in Proxmox VE. Shared LVM does the job, but it's not the same thing, and backup is a good example. In a Veeam environment, VMs on shared LVM require an additional file-level storage (NFS or similar) to hold snapshots during backups. That means extra infrastructure to size, secure and maintain, outside the SAN we already have. A shared file-level datastore directly on the SAN could remove that constraint, provided Veeam can work with it.

I agree with the concerns raised above: nobody should put production data on an experimental clustered filesystem. That's actually why I'd like to help. We have enterprise storage and servers that we can dedicate to an isolated test environment, which is probably what both projects need most right now: real arrays, real hardware, and people willing to break things and report back.

What we could offer:

- Storage: Huawei OceanStor Dorado arrays (including an active-active HyperMetro setup across two sites) and NetApp ASA, over FC and/or iSCSI with dm-multipath
- Several physical servers running Proxmox VE
- CAW support results for these arrays (caw_probe / sg_vpd output)
- Performance benchmarks with fio, comparing cetrafs and MeshXFS against shared LVM-thick, with and without snapshots, using a common methodology so results are comparable (happy to reuse melkier's bench plan)
- Resilience testing with vdbench: we already run a vdbench cluster that proved very useful to validate our storage during outages and failovers. Its data validation mode is well suited to catching silent corruption on a clustered filesystem while things are failing underneath
- Failure scenarios: path loss and multipath failover (no_path_retry vs stale_secs), controller failover, HyperMetro site failover, node crash and fencing, live migration under load
- Veeam backup and restore testing. As far as I know, Veeam's Proxmox plugin doesn't support custom storage types today. Would mounting cetrafs outside PVE and declaring it as a standard shared "dir" storage be a viable workaround, or would that break assumptions the plugin relies on (nofilelock, mount handling)?
- Detailed bug reports with logs, and a public write-up of the results

Before we start, it would help to know what would be most useful to you: which versions to test, which scenarios you haven't been able to cover yet, and how you'd like the results reported (GitHub issues, this thread, or something else).

The goal is to give both projects solid data from enterprise hardware, not to rush anything into production. Let me know if you're interested.

Best regards,
Thibaut
 
You must log in or register to reply here.
Top Bottom
Back