Hi ctrix, hi melkier,
The goal is to give both projects solid data from enterprise hardware, not to rush anything into production. Let me know if you're interested.
Hi Thibaut,
Thank you — really. Real arrays, real fabrics and someone willing to break things is what this project is short of.
cetrafs 0.9 is out today:
https://github.com/cetrafs/cetrafs, tag 0.9. It is clearly more robust than anything before it. A new randomized churn suite checks every byte of every file against an oracle while the defragmenter and scrub run underneath, and it found several data-integrity defects my scripted tests had structurally missed — worst case, a heavily rewritten file reading back stale data. All fixed; the gate is now 1757 assertions over 164 suites, plus fuzzing, a crash rig and a multi-node suite. Still pre-1.0, but it is the first release I would ask someone to point enterprise hardware at. Test 0.9, nothing older.
My gaps, since they decide what helps: CI runs on loop devices, which have no atomic CAW, so it only ever exercises the serialised fallback. Two or three nodes, one vendor, one fabric. Never tested a controller failover, a metro cluster, a power loss or a backup product.
Three stages, in order — each gates the next.
1. CAW present, and actually atomic.
modprobe cetrafs caw_probe=/dev/sdX, then dmesg | tail. It is read-only and touches no data; it prints the class, the logical block size and max_caw_blocks. Zero means cetrafs refuses multi-node mounts on that device by design. Send me that line for Dorado and for ASA.
Then the bit I can't test. Every cross-node claim — Zone Group ownership, the file lock, the namespace lock, the inode bitmap — is one fused COMPARE AND WRITE on one 512-byte record, and it all rests on CAWs to a given LBA being serialised in a single atomicity domain. Across HyperMetro that is an open question, and the probe cannot answer it: it will report "available" at both sites while saying nothing about whether the two serialise against each other. The tell is a counter, cas_miscompares, in /proc/fs/cetrafs/<dev>/info. A miscompare is a node losing a race and retrying, so under real contention it must rise. If two nodes contend across sites and it stays at zero while a same-site pair makes it climb, that is the single most important thing you could tell me.
2. Path failure on an active path.
Liveness is a timeout: hb_interval_secs=15 and stale_secs=120 by default, tunable per mount through /proc/fs/cetrafs/<dev>/controls (ranges 5–30 and 15–240, with stale_secs at least 3x hb_interval_secs). With no_path_retry queue, I/O blocks indefinitely, the node cannot renew its heartbeat, peers declare it stale and revoke its lease — and when the paths come back it self-fences and every write returns EIO. So no_path_retry x polling_interval has to stay well under stale_secs, and a routine no_path_retry 30 with polling_interval 5 already exceeds the default.
What I need is a measured number: how long I/O actually stalls across a path loss, a controller failover and a site failover on your arrays, with your multipath.conf. That becomes an operational table I cannot write myself. One caveat: the fencing is an advisory lease, not target-enforced, so writes already in flight still land — STONITH stays the cluster manager's job.
3. Proxmox, the application layer.
This is where your vdbench cluster earns its keep: data validation during the failure, not after it. I would set the bar at three things together — zero validation errors, fsck.cetra clean, and no checksum complaints from cetrafs in dmesg on any node.
That last one needs a word of explanation. Every major on-disk structure carries a CRC32C that the kernel verifies on read. A lockless reader can legitimately catch a writer mid-update, so a mismatch is retried a few times first; if it still fails to verify, cetrafs logs it and returns EUCLEAN to the caller. One of those during a failover means a node read metadata that did not validate even after retrying, which is exactly the class of silent damage this kind of testing exists to find — so it matters even when the workload above it looks fine. I ask for all three because a filesystem can be perfectly consistent and still hand back the wrong bytes: fsck clean is necessary and never sufficient.
Guest performance is worth a matrix: aio=native vs threads vs io_uring, cache=none vs the others, raw vs qcow2. One warning or the numbers are unreadable — the first pass over a sparse image allocates blocks, the second only rewrites them. On my rig that is 20 MB/s against 290 MB/s on the same file. Always say which one you measured.
Then the scenarios I can barely reach: kill a VM hard and check it comes back; live migration under load, with and without an active snapshot; drop a whole node (echo b > /proc/sysrq-trigger) with VMs running; and whether Proxmox HA restarts them elsewhere with the images intact.
On Veeam: yes, and that is how I run it. Mount cetrafs outside PVE and declare it as a dir storage with shared set to 1; PVE only uses the path, the mount is yours to manage (fstab or a systemd unit). Mount it with nofilelock, which is the Proxmox shape anyway since live migration needs it — with that option there is no per-file lock in the way at all, so Veeam can open whatever it likes from wherever it likes. (The default mode takes a whole-file exclusive lock, VMFS-style, and would refuse a proxy on another node opening a running VM's disk. Just don't use it here. The mode is cluster-wide: the first node to mount sets it and a conflicting mount is refused, so set it once and leave it.)
Beyond that I have zero Veeam experience and no expectations to defend — every failure or data loss you see there is something I want to know about and will try to reproduce.
Reporting: GitHub issues are ideal, they stay attached to the code. For anything that smells like corruption, what saves a round trip is /proc/fs/cetrafs/<dev>/info and dmesg | grep -i cetrafs from every node, the mount options, the caw probe line, multipath.conf, and fsck.cetra output. I don't trust cross-node timestamps — clock skew has faked races for me before — so on-disk evidence beats a correlated log.
Two warnings before you start. Pre-1.0 means no on-disk compatibility between releases apart from one transitional window, so a version bump can mean reformatting. And growfs.cetra is offline-only in 0.9 and stamps the new format — a one-way door, so upgrade every node before growing or the old ones stop being able to mount.
Reach me however suits you: GitHub issues or discussions on the repo above, this thread, or email me directly. A call is fine too if that is faster at any point. And if anything is ambiguous mid-test, just ask — I would rather answer than have you spend array time on a scenario I could have clarified in two lines.
Thanks again. This is the help the project needs most.
