cetrafs — Open-source shared-disk filesystem for Proxmox VE

C

ctrix

New Member
Oct 22, 2026
4
2
3
Hi everyone,

I've been working on cetrafs, a GPL-licensed Linux clustered filesystem designed for shared VM datastores, somewhat similar to VMware's VMFS.

The project actually started because I had some storage hardware originally purchased for another hypervisor vendor, which then decided to raise its prices outrageously. I wanted to keep using my existing infrastructure without being locked into that ecosystem, and needed a viable alternative.

It lets multiple Proxmox nodes mount the same iSCSI or FC LUN concurrently, using SCSI atomic operations for coordination instead of a distributed lock manager.

I've tested it on Proxmox clusters using two different storage setups:

  • A Synology NAS over iSCSI.
  • An HP ClusterOne SAN over Fibre Channel with multipath.
I've also documented Proxmox live migration workflows.

It's still beta software, and I'd be particularly interested in feedback from people familiar with Proxmox storage internals or shared SAN environments.

Project: https://cetrafs.com/

Source: https://github.com/cetrafs/cetrafs
 
  • Like
Reactions: EllerholdAG
Nice to see that you put effort (or tokens) into this and I really appreciate it, but I have to say, I would NEVER use this in production at the current stage without proper support. Both are nice projects and I often wanted to write something like this myself, but I know enough about filesystem that I would never try it myself, not even with AI. Just look at filesystems like ZFS with a huge developer and user base and what strange bugs get fixed like finally the zfs encryption and send/receive corruption fix I just learned about yesterday in 2.3.3. ZFS is also "just" a local filesystem, not a clustered one. Trusting a filesystem with data needs also to have combined myriads of time of experience with it and also the trust in the codebase and acceptance by other "filesystem people" to start with. If a filesystem is vetted and included into the mainline kernel, I would consider using it. The other way would be the support of a company behind it that I can pay for solving problems, at least in the enterprise environment.

Also, just having something that works, does not mean you can use it in production. I started using OCFS2 over 15 years ago, it was complicated to setup and maintain and not very fast, but it had the backing of a smallish company named Oracle behind it and they later got it into the mainline kernel. I'm not using it today because it wasn't up to the job I needed it to do - maybe it is now, but I doubt it or it would be used more in PVE and you would not have started to write your own filesystem. Oracle switched their focus on their other cluster filesystem ACFS that I use daily and in many Oracle database clusters. If even a huge company and the kernel developers could not provide a proper cluster filesystem that is accepted widely, I doubt a single developer can build something even comparable.

I don't want to discourage you, I just want to ground you somehow that this is a nice starting point to build something worthwhile, but it's a VERY long way to go. Maybe I'm completely wrong with my assessment with the advent of AI filesystem development and the extremely complicated environment it has to work in. I would be very happy to be proven wrong, so that we would have a working cluster filesystem besides the more or less working solutions we have for at least a decade.
 
  • Like
Reactions: Johannes S
While I agree with LnxBil, this process is exactly how new software is written.
Enough users want something (a cluster filesystem) so that a company or a Dev writes it and is looking for testers, supporters or even backers.
Yes at the moment this should be considered experimental BUT it can grow to something beautiful. By reading this forum this is much needed and will find a lot of users. I'd argue that home labbers are the perfect user group for this. They are experimenting with new stuff, they are willing for something to break and they often times have older hardware that triggers all kinds of bugs that newer enterprise grade servers don't have.

If this runs well, maybe a huge company (e. G. Proxmox) could pick it up and develop it further! Or it can go into the Linux kernel...

So I'd really like you to succeed (both cetrafs and meshxfs)!
 
Thanks to both of you for the thoughtful feedback. I think both perspectives are valuable, although I personally lean more towards the second one: being open to experimentation, trying new things, and helping projects move forward.

First of all, I'm not claiming CetraFS is production-ready. There's a project status page that explicitly describes it as experimental. I want people to know exactly what they're getting into.

That said, this isn't just something that happens to work in a lab. CetraFS currently has almost 1,500 tests, a fuzzing suite, and a regression testing suite. Every bug discovered gets a dedicated test to make sure it doesn't come back. Testing doesn't guarantee correctness, especially for a clustered filesystem, but it's a fundamental part of how I'm developing it.

Is it immature? Absolutely. Does that mean it's not worth trying? I don't think so.

I believe that with the right mindset, a willingness to experiment, and constructive feedback from the community, this project can grow into something genuinely useful.

And that's exactly what I'm hoping for at this stage: people willing to try it, push it to its limits, break things, report bugs, and share their experiences. That's the only way to make real progress.
 
  • Like
Reactions: EllerholdAG
ctrix said:
Thanks to both of you for the thoughtful feedback. I think both perspectives are valuable, although I personally lean more towards the second one: being open to experimentation, trying new things, and helping projects move forward.
Click to expand...

That's a nice idea if you happen to develop video games, a office suite or CAD software (aka any software, where new features are the most important thing). It's not so nice if we are talking about a file system which you want to be able to trust with your data ( or anything else related to infrastructure).

This is true in homelabs too: Allthough peoples tax records or family pictures are not really important for the state of the world they still mean the world to them and don't want to loose it.

The attitude ("let's move things forward") is the reason bcachefs was excluded from the Linux kernel mainline after it's first inclusion and is in general a red flag to me.
So I sincerely wish to express my thanks to @LnxBil for taking the time to write out his concerns. I also had doubts but neither the time nor the knowledge and experience to write them out in a concise format. And I didn't want to just vent out to the AI usage since I'm aware that my AI-luddite attitute isn't an argument per se since (like prooven by several open source projects) a competent engineer can do great things with AI (the ecological and social costs of AI are another matter but doesn't have anything to do with it's merits for software development).
 
Last edited:
I think you're taking this a bit too seriously. It's a personal project, the author clearly states that it's experimental, and he's just looking for people willing to test it and give feedback.

Of course nobody should blindly trust a new filesystem with important data, especially in production. But there's a big difference between saying "I'd never use this in production yet" and being so negative about the whole idea.

Every project has to start somewhere. If nobody is willing to experiment and help out, nothing new will ever get better. You don't have to use it, and nobody is asking you to. Let the guy work on his project, see how it develops, and judge it by the results.

Honestly, I think it's great that someone is putting time and effort into building something the Proxmox community could benefit from.
 
  • Like
Reactions: waltar
xybo said:
I think you're taking this a bit too seriously. It's a personal project, the author clearly states that it's experimental, and he's just looking for people willing to test it and give feedback.
Click to expand...

The trouble is, that a lot of people who come from vmware and basically want to have the same workflows are looking into things like ocfs as possible alternatives. Many of them don't even encounter the forum threads where people like @LnxBil or @gurubert report on the caveats you need to know to make it work or consider another storage backend depending on the usecase.
And ocfs is at least a lot more mature than the vmfs-on-xfs projects. So: Yes, even if it's just a "personal project" people will try to use it for important data. So I think it's fair to have some concerns and share them. It's better to have too much warning than the opposite. Because like with other non-supported stuff at some point somebody will come and ask how he can restore his family pictures/tax records/production database he used the experimental stuff with.

xybo said:
Of course nobody should blindly trust a new filesystem with important data, especially in production. But there's a big difference between saying "I'd never use this in production yet" and being so negative about the whole idea.
Click to expand...

I'm in general negative to experimental stuff for my data, yes.
 
Johannes S said:
I'm in general negative to experimental stuff for my data, yes.
Click to expand...

I'm not asking you to do it.
And yes, you should be as cautious as possible with your data, i agree that it's the way to go.

Again: data is yours, you're free do do what you want with them.
I would just appreciate you not to shit on someone else work without having at least tried it or understanding what it is

(and BTW, this is not VMFW over XFS, that one is another thread)
 
Johannes S said:
The trouble is, that a lot of people who come from vmware and basically want to have the same workflows are looking into things like ocfs as possible alternatives. Many of them don't even encounter the forum threads where people like @LnxBil or @gurubert report on the caveats you need to know to make it work or consider another storage backend depending on the usecase.
And ocfs is at least a lot more mature than the vmfs-on-xfs projects. So: Yes, even if it's just a "personal project" people will try to use it for important data. So I think it's fair to have some concerns and share them. It's better to have too much warning than the opposite. Because like with other non-supported stuff at some point somebody will come and ask how he can restore his family pictures/tax records/production database he used the experimental stuff with.



I'm in general negative to experimental stuff for my data, yes.
Click to expand...
I agree with Johannes on one point: nobody should entrust their only copy of important data to an experimental filesystem (you should never only have one copy). That's not what ctrix and I are asking anyone to do.

But there's a difference between being cautious about deploying new software and dismissing it before it's been properly evaluated. We're asking people to test these projects in environments where failures are acceptable, put them through their paces, and report what they find.

A filesystem doesn't become reliable simply by accumulating years of existence or by having a big company behind it. It becomes reliable through engineering, extensive testing, independent review, finding and fixing bugs, and validating those fixes. Time and real-world experience matter, too, but they aren't substitutes for that work.

Both CetraFS and MeshXFS are tackling a real need in the Proxmox community. Let's evaluate them on their technical merits and work to make them better.
 
melkier said:
Thiis is a good idea - i've added it as a possible inclusion in MeshXFS
Click to expand...
it's more something in qemu blockdev side, so it need patch on qemu-server. I'll try to see if it's improve performance when allocating blocks with for meshxfs && cetrafs.

I don't known if somebody already done fio benchmark ?

I had pretty poor results with ocfs2 && gfs2 in the past.
 
aderumier said:
it's more something in qemu blockdev side, so it need patch on qemu-server. I'll try to see if it's improve performance when allocating blocks with for meshxfs && cetrafs.

I don't known if somebody already done fio benchmark ?

I had pretty poor results with ocfs2 && gfs2 in the past.
Click to expand...
https://github.com/sshoecraft/meshxfs/tree/main/bench

I'll be doing comparison testing next week. The results show aggregates as the node grows but the storage is coming from a local nvme drive presented to a qemu vm. here's the rough plan:

Setups

  1. QNAP, 1G, physical: 2 and 4 nodes. Lock and iSCSI traffic share the single link (disclose this).
  2. Local SCST: 2, 4, 8 and 16 VM nodes, with blockio or fileio using o_direct=1 write_through=1, a working set larger than host RAM, and lock traffic on its own virtio bridge.

Filesystems

  • Native XFS on one node, as the reference row.
  • MeshXFS, in both lock modes where the target supports them: SCSI CAW and TCP lock network. Check whether the QNAP supports CAW.
  • GFS2 with DLM, corosync/pacemaker and fencing configured.
  • OCFS2, with the stack stated (o2cb or pcmk).
  • Same kernel on every node, fresh mkfs on the same LUN for each filesystem, and default mount options.

Workloads

  1. Per-node file fio: your existing suite, the uncontended baseline.
  2. Shared-file fio: all nodes doing 4K random reads and writes to one file, both overlapping and striped offsets.
  3. mdtest in one shared directory: create, stat, unlink, from all nodes at once. This is the main comparison.
  4. mdtest with a private directory per node: shows locking cost without contention.
  5. Real-world mix: parallel kernel-tree untar and rm -rf, plus find/ls -l storms on one tree.
  6. Optional, recovery: fence one node mid-workload and measure how long the survivors stall and when throughput returns.

Procedure

  • Drop caches on the nodes and the target between runs.
  • Run each test 3 times and report the median.
  • Capture throughput, IOPS and ops/sec, plus p50, p99 and max latency.
  • Script it into the existing harness so all four filesystems run identically, and log the configuration with each result.

Reporting

  • One table per setup: filesystem × node count × workload.
  • Charts of ops/sec against node count for the shared-directory mdtest and the shared-file fio. Those are the headline results.
  • State the limits up front: single 1G link, QNAP model, and VM caveats for the local runs.

I'll update the bench readme when done. I'll do the same for certa - but its a diff test because its targeted at VMFS capability (no concurrent writes to the same files/dirs).
 
You must log in or register to reply here.
Top Bottom
Back