cetrafs — Open-source shared-disk filesystem for Proxmox VE

Thanks for integrating my fix, this makes it now possible to use it with my CAW supported devices and is also required for further benchmarking.

Btw, I'll cover CetraFS at credativ's Virtualization Gathering. This time, it's all about storage and I will also provide 2 talks. One about your CetraFS, the other one is about my shared LVM Thin plugin. More information can be found here.
 
Thanks for integrating my fix, this makes it now possible to use it with my CAW supported devices and is also required for further benchmarking.

Btw, I'll cover CetraFS at credativ's Virtualization Gathering. This time, it's all about storage and I will also provide 2 talks. One about your CetraFS, the other one is about my shared LVM Thin plugin. More information can be found here.

No, thanks to you @gyptazy for the efforts and the support! And thanks for presenting cetrafs.

A couple of notes:
* next thing to implement is live growfs (0.9 introduced the offline version)
* then i'm planning for FSTRIM and FITRIM, so basically discard for the thin provisioned SANs

I think this would cover most of the gaps with VMFS but i'd appreciate some discussion about how to evolve this filesystem and what the community would need.
 
@ctrix Thank you for your hard work to make a filesystem like vmfs for Proxmox possible. I hope that Proxmox integrates this development into its native ecosystem and provides direct support for it. After all, this is precisely what many of us need in order to continue using SANs directly with Proxmox, just as we have previously done with VMware. Will it also be possible, at a later stage, to take a snapshot of a cetraFS file system from the SAN and then mount this snapshot alongside the live file system in order to extract data? This is precisely the functionality used nowadays with many NetApp systems in conjunction with VMware to create backups of the machines residing on them.
 
@ctrix Thank you for your hard work to make a filesystem like vmfs for Proxmox possible. I hope that Proxmox integrates this development into its native ecosystem and provides direct support for it. After all, this is precisely what many of us need in order to continue using SANs directly with Proxmox, just as we have previously done with VMware. Will it also be possible, at a later stage, to take a snapshot of a cetraFS file system from the SAN and then mount this snapshot alongside the live file system in order to extract data? This is precisely the functionality used nowadays with many NetApp systems in conjunction with VMware to create backups of the machines residing on them.

Thanks! Today the answer is no. It's a known gap: "snapshot/clone resignaturing" is on the not-built list in docs/ARCHITECTURE.md.

A SAN snapshot is a block-for-block copy of the volume, and that copy includes the cluster state:

- Same identity. It has the same filesystem UUID and the same Node Registry as the live volume.
- It thinks it's live. Every node that had the volume mounted still shows as registered, and their Zone Group ownership and file locks are all there.
- The mounting host is already in it. A host that mounts the snapshot finds its own row, so it treats the snapshot as its own volume after an unclean shutdown. It would then recover a volume that is really a frozen copy of one it has mounted right now.

None of this is tested, so I can't recommend it, even read-only. A read-only mount still registers and heartbeats on the device. It also doesn't replay the journals of nodes that were mid-operation when the snapshot was taken.

Doing this properly means a resignature step, the same idea as VMFS's:

- give the copy a new filesystem UUID;
- mark every registry row departed, and release every Zone Group and lock;
- replay the journals of the nodes that were writing, so the copy is crash-consistent;
- then mount it as an independent volume.

That fits as an offline tool next to fsck.cetra. It's a reasonable feature and I'll put it on the list. It isn't part of the next releases, though: that one is about UNMAP/trim and the format change.

One caveat: I'm not happy shipping a feature I can't test on my own hardware, and I have no array that takes snapshots I could use for this. A filesystem feature nobody has run end to end is a promise, not a feature. So when it comes, it will either come with a way to test it here, or with help from someone who has the hardware and is willing to donate or give me access for tests.
 
Does it support SMR shingled hard drives?

It depends on what kind of SMR:

- Host-managed SMR (zoned devices): no. cetrafs writes in place, including bitmaps, inode records and the atomic compare-and-write on 512-byte records that the clustering is built on. A zoned device only accepts sequential writes within a zone, and there's no zoned-mode support.

- Drive-managed (or host-aware) SMR: it should work, because the drive hides the shingling. VM images plus cluster metadata are a random-write workload, though, which is the worst case for those drives. Expect sustained write speed to collapse once the drive's cache fills. We haven't tested it, and I wouldn't recommend it.

- SMR disks inside a SAN array: the array hides the drives and cetrafs only sees the LUN. Whether that works well is up to the array.

Also, cetrafs is meant for a shared LUN. A single directly attached SATA disk has no SCSI COMPARE AND WRITE, so only one node can mount it.
 
Hi everyone,

cetrafs 0.10.1 is out: https://github.com/cetrafs/cetrafs/releases/tag/0.10.1

When I posted 0.9 here, the test gate was 1,757 assertions over 164 suites. 0.10.1 runs about 6,600 assertions over 837 suites, and a full make check takes about 8 hours on one machine. On top of that there is:

- a two-node suite of 68 scenarios on real FC and iSCSI storage;
- a crash rig that replays every write up to arbitrary cut points;
- fuzzing, and a KASAN build.

631 of the suites have a negative control: a switch that puts the bug back, to prove the test really catches it. The list of known open defects is empty. What is tested, how, and what the tests can't show: https://cetrafs.com/docs/concepts/testing/

I think it is very close to being ready for prime time. What it still needs is hardware testing on a wider range of SANs: so far it has run on HPE 3PAR over FC and on iSCSI and a few others and that isn't enough. The testing page shows what that judgement rests on.

What changed since 0.9.5:

- Bigger files: up to 1 EiB, about 65 million extents per file, and directories up to about 3.9 million entries.
- Trim, as promised: an emptied Block Group is unmapped when it goes back to the pool, fstrim works, and -o discard gives inline discard. https://cetrafs.com/docs/how-tos/reclaim-thin-space/
- Growing a volume now adds inodes as well as space.
- Integrity: a checksum on every bitmap sector, a backup superblock, a range check on every address read from disk, and a "needs fsck" mark that persists until a check runs.
- No fsck after a crash: the next mount repairs it. fsck.cetra --live checks a volume while other nodes are using it.
- Faster failover: an HA agent that has powered a host off can declare it dead instead of waiting out the stale window.
- A thin pool that fills up pauses writes instead of stopping the mount.
- Hundreds of fixes, most of them found by the new tests.

Tested on kernels 7.0.14, 7.1.8+deb13 and 7.2.11, and on Proxmox VE 7.0.14-pve. Storage was HPE 3PAR over 32 Gb FC multipath and iSCSI, on amd64 and arm64.

Before upgrading: 0.10 changes the on-disk format, and 0.9.x volumes aren't mounted. You need to evacuate the data, reformat and restore: https://cetrafs.com/docs/how-tos/upgrade-reformat/

I've also written up why the Proxmox plugin has to mount with nofilelock, and how live migration could work with the lock kept on every file:
https://cetrafs.com/docs/how-tos/live-migration/
https://cetrafs.com/docs/how-tos/live-migration-handover/
I'd be glad to discuss the second one with anyone who works on Proxmox's migration code.

So, same request as before, only more so: if you have a SAN and a test cluster, especially an array I haven't listed, please try to break it and tell me how.
 
  • Like
Reactions: waltar