Hi everyone,
cetrafs 0.10.1 is out:
https://github.com/cetrafs/cetrafs/releases/tag/0.10.1
When I posted 0.9 here, the test gate was 1,757 assertions over 164 suites. 0.10.1 runs about 6,600 assertions over 837 suites, and a full make check takes about 8 hours on one machine. On top of that there is:
- a two-node suite of 68 scenarios on real FC and iSCSI storage;
- a crash rig that replays every write up to arbitrary cut points;
- fuzzing, and a KASAN build.
631 of the suites have a negative control: a switch that puts the bug back, to prove the test really catches it. The list of known open defects is empty. What is tested, how, and what the tests can't show:
https://cetrafs.com/docs/concepts/testing/
I think it is very close to being ready for prime time. What it still needs is hardware testing on a wider range of SANs: so far it has run on HPE 3PAR over FC and on iSCSI and a few others and that isn't enough. The testing page shows what that judgement rests on.
What changed since 0.9.5:
- Bigger files: up to 1 EiB, about 65 million extents per file, and directories up to about 3.9 million entries.
- Trim, as promised: an emptied Block Group is unmapped when it goes back to the pool, fstrim works, and -o discard gives inline discard.
https://cetrafs.com/docs/how-tos/reclaim-thin-space/
- Growing a volume now adds inodes as well as space.
- Integrity: a checksum on every bitmap sector, a backup superblock, a range check on every address read from disk, and a "needs fsck" mark that persists until a check runs.
- No fsck after a crash: the next mount repairs it. fsck.cetra --live checks a volume while other nodes are using it.
- Faster failover: an HA agent that has powered a host off can declare it dead instead of waiting out the stale window.
- A thin pool that fills up pauses writes instead of stopping the mount.
- Hundreds of fixes, most of them found by the new tests.
Tested on kernels 7.0.14, 7.1.8+deb13 and 7.2.11, and on Proxmox VE 7.0.14-pve. Storage was HPE 3PAR over 32 Gb FC multipath and iSCSI, on amd64 and arm64.
Before upgrading: 0.10 changes the on-disk format, and 0.9.x volumes aren't mounted. You need to evacuate the data, reformat and restore:
https://cetrafs.com/docs/how-tos/upgrade-reformat/
I've also written up why the Proxmox plugin has to mount with nofilelock, and how live migration could work with the lock kept on every file:
https://cetrafs.com/docs/how-tos/live-migration/
https://cetrafs.com/docs/how-tos/live-migration-handover/
I'd be glad to discuss the second one with anyone who works on Proxmox's migration code.
So, same request as before, only more so: if you have a SAN and a test cluster, especially an array I haven't listed, please try to break it and tell me how.