Testers wanted: MeshXFS clustered filesystem and Proxmox VE storage plugin

M

melkier

Member
May 22, 2024
14
9
8
Hello Proxmox VE community,

I'm looking for testers for **MeshXFS**, an open-source (GPL-2.0) POSIX clustered filesystem built on XFS.

- GitHub: https://github.com/sshoecraft/meshxfs
- Releases: https://github.com/sshoecraft/meshxfs/releases

MeshXFS lets several nodes mount the same block device read-write at the same time, with distributed locking, coherent caching, fencing and journal recovery. On the system it goes by `mxfs`: `mount -t mxfs`, `mkfs.mxfs`, `chk_mxfs`.

### Installing on Proxmox VE 9

Download the latest release's packages from the [Releases page](https://github.com/sshoecraft/meshxfs/releases) and install them on every node:

```bash
apt install proxmox-headers-$(uname -r) ./mxfs_<version>_amd64.deb
apt install ./pve-storage-mxfs_<version>_all.deb # optional: MeshXFS as a storage type in the web UI
```

- `mxfs` is the kernel module (built by DKMS against your running kernel) and the userspace tools. It is all you need to mount; you can then add the mount as shared directory storage with `pvesm`.
- `pve-storage-mxfs` is the Proxmox VE storage plugin. It requires `mxfs`.

The release is verified on the Proxmox VE 9 kernels **6.17.2-1-pve** (the 9.1 installer's) and **7.0.14** (what `proxmox-default-kernel` installs). Check `uname -r` on your nodes: other kernels usually build, but they are untested.

Every node mounting a MeshXFS filesystem must run the **same MeshXFS version**. Give VM disks on MeshXFS `cache=none`, the Proxmox default.

### Configurations to test

**1. Shared SAN.** One iSCSI LUN shared by every node, on a single path or on two paths with `dm-multipath`. Validated at **2, 4, 8 and 16 nodes**, with either lock manager:

- the network lock manager (the default), or
- on-disk locks using SCSI `COMPARE AND WRITE`, with no lock network at all.

The LUN must support SCSI persistent reservations, and for the on-disk locks an atomic `COMPARE AND WRITE`.

When a node dies, hangs, or is cut off from the others, the survivors fence it at the storage with SCSI persistent reservations, replay its journal and keep writing. Tested under load: a node killed, a node hung mid-write, a network partition, one path lost and returned, every path of a node lost.

**2. Two hosts, no shared storage.** Each host's own disk, mirrored with **DRBD 8.4 dual-primary**. The complete, tested setup guide:
https://github.com/sshoecraft/meshxfs/blob/main/docs/drbd-setup.md

A two-host pair has no third vote, so its fencing is deliberately conservative:

- If the replication link breaks, the host with the lower DRBD address carries on. The other freezes, restarts itself and rejoins once it is back.
- If the **lower-address host itself dies**, the survivor cannot tell that from a cut cable. It does not take over: it waits, unmounted, until that host is back.
- To have the survivor take over in that case, give the pair a third vote (a corosync QDevice, `authority=quorum`) or a node fence (IPMI, a PDU). The guide covers both.

### What the current release passed on Proxmox VE

Three separate sets of tests:

- **Three Proxmox VE 9 hosts on one multipath SAN LUN**, on kernels 6.17 and 7.0: storage paths lost and returned under load, live migration while the guest kept writing, HA restart after a host lost power, snapshots and rollback, backup and restore, disk moves, templates and linked clones, and containers.
- **Two physical Proxmox VE 9 hosts on DRBD**, set up from the guide: a network cable pulled under load and plugged back in.
- **The installed packages** on two Proxmox VE 9 nodes on each kernel: mount, cross-node reads by checksum, reboot with data intact, and a node hung in the middle of a write and fenced by the other.

### What I'd like tested

- Installation and the storage plugin on your own Proxmox VE 9 cluster
- Real VM workloads, repeated live migrations and HA failover
- A host powered off without a clean shutdown, a pulled network or storage cable, one multipath path cut
- Sustained I/O and fsync-heavy workloads
- Anything unexpectedly slow, even when nothing fails

If you're interested, reply with your Proxmox VE version, kernel and storage setup. Please report problems as GitHub issues with the kernel log from every node:
https://github.com/sshoecraft/meshxfs/issues

Feedback on installation, documentation, recovery behaviour and performance is all welcome. Thanks!
 
  • Like
Reactions: _gabriel, Blueloop, xybo and 3 others
Maybe contact a Youtuber that's into proxmox? It would be nice to have a video walkthru of setting up and testing something like this

Virtualizationhowto and ElectronicsWizardry are likely candidates.
 
  • Like
Reactions: waltar
Hi, This is huge !

I'm going to test it at scale, definitively ! Looking for a working clusterfs since years (sorry ocfs2/gfs2, you sucks...)

I have found the announce on phoronix, is this fs had been build with proxmox usage in mind ? (we really need a vmware vmfs alternative for proxmox, for people with basic san storage)

Btw, do you plan to support nvme over tcp too ? (with nvme reservations)
 
Last edited:
  • Like
Reactions: Gilberto Ferreira
Kingneutron said:
Maybe contact a Youtuber that's into proxmox? It would be nice to have a video walkthru of setting up and testing something like this

Virtualizationhowto and ElectronicsWizardry are likely candidates.
Click to expand...
This is a great idea! I will, thanks!
 
aderumier said:
Hi, This is huge !

I'm going to test it at scale, definitively ! Looking for a working clusterfs since years (sorry ocfs2/gfs2, you sucks...)

I have found the announce on phoronix, is this fs had been build with proxmox usage in mind ? (we really need a vmware vmfs alternative for proxmox, for people with basic san storage)

Btw, do you plan to support nvme over tcp too ? (with nvme reservations)
Click to expand...
Yes, the project was started with proxmox in mind - I have a 2 node proxmox cluster at home and felt that proxmox really needed its own VMFS.

im unable to test nvme over tcp in my env ... i might be able to emulate but you'll need to test

It has been added to the list - will update when its been worked on.
 
  • Like
Reactions: Gilberto Ferreira
Hi there...
I am testing... Everything was installed without no warnings
However I couldn't see any options in Datacenter -> Storage -> Add.
And when I tried to add via cli, I got this message:
proxmox01:~# pvesm add mxfs
Plugin "PVE::Storage::Custom::MXFSPlugin" is implementing an older storage API, an upgrade is recommended
400 not enough arguments
pvesm add <type> <storage> [OPTIONS]

proxmox01:~# pveversion
pve-manager/9.2.21/4f6e0ac86f9e8c7f (running kernel: 7.0.14-25-pve)
 
Hi again
I suppose that with drbd 9 doesn't work.
I tried and get this:
proxmox01:~# mount -t mxfs /dev/drbd0 /data
mount: /data: fsconfig() failed: mxfs: P-DRBD-ARM-REFUSED minor=0 -- replication protocol is '' (configured 'C'), not C; a DRBD device is admitted only with its fence evidence (set the resource up as docs/drbd-setup.md shows).
dmesg(1) may have more information after failed mount system call.
proxmox01:~# dmesg | tail
[Tue Oct 20 13:22:48 2026] drbd r0/0 drbd0: gen-obligation: MINTING(promoted) -> DISCHARGED(promoted)
[Tue Oct 20 13:22:48 2026] mxfs (drbd0): MXFS envelope v1: XFS data at offset 4672712704 (9126392 sectors) proto_gen=22 dirshard=off cluster=(none)
[Tue Oct 20 13:22:48 2026] mxfs: force_transport=1 -> TCP transport
[Tue Oct 20 13:22:48 2026] mxfs: P-DRBD-ARM-REFUSED minor=0 -- replication protocol is '' (configured 'C'), not C; a DRBD device is admitted only with its fence evidence (set the resource up as docs/drbd-setup.md shows)
[Tue Oct 20 13:22:49 2026] mxfs: P-HRX-STATS queued=0 inline=0 max_depth=0 max_wait_ms=0 asks_first=0 asks_merged=0 ask_max_wait_ms=0 -- page hand-offs from peers, processed on their own worker
[Tue Oct 20 13:22:49 2026] mxfs (drbd0): MXFS DLM init failed -- aborting mount of cluster (envelope) volume
[Tue Oct 20 13:22:49 2026] drbd r0: Preparing cluster-wide state change 1828632018: 0->all role( Secondary )
[Tue Oct 20 13:22:49 2026] drbd r0: State change 1828632018: primary_nodes=0, weak_nodes=0
[Tue Oct 20 13:22:49 2026] drbd r0: Committing cluster-wide state change 1828632018 (0ms)
[Tue Oct 20 13:22:49 2026] drbd r0: role( Primary -> Secondary ) [mount:4795 auto-demote]

cat /etc/drbd.d/r0.res
resource r0 {
startup {
wfc-timeout 15; # Tempo aguardando o par se ele estava online antes (em segundos)
degr-wfc-timeout 15; # Tempo aguardando o par se ele já estava desligado antes
}

net {
protocol C;
fencing resource-only;
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
max-buffers 32k;
sndbuf-size 2M;
rcvbuf-size 2M;
}
handlers {
### fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
fence-peer "/usr/sbin/mxfs-drbd-fence-peer";
unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
}
connection-mesh {
hosts proxmox01 proxmox02;
}
on proxmox01 {
address 172.19.150.10:7788;
node-id 0;
device /dev/drbd0;
disk /dev/vdb;
meta-disk internal;
}
on proxmox02 {
address 172.19.150.20:7788;
node-id 1;
device /dev/drbd0;
disk /dev/vdb;
meta-disk internal;
}
disk {
discard-zeroes-if-aligned yes;
rs-discard-granularity 4k;
no-disk-flushes;
no-disk-barrier;
c-plan-ahead 0;
resync-rate 180M;
c-fill-target 2M;
c-min-rate 30M;
c-max-rate 210M;
}
}
 
Gilberto Ferreira said:
Hi there...
I am testing... Everything was installed without no warnings
However I couldn't see any options in Datacenter -> Storage -> Add.
And when I tried to add via cli, I got this message:
proxmox01:~# pvesm add mxfs
Plugin "PVE::Storage::Custom::MXFSPlugin" is implementing an older storage API, an upgrade is recommended
400 not enough arguments
pvesm add <type> <storage> [OPTIONS]

proxmox01:~# pveversion
pve-manager/9.2.21/4f6e0ac86f9e8c7f (running kernel: 7.0.14-25-pve)
Click to expand...

PVE 9.2 support has been added in the .132 release: https://github.com/sshoecraft/meshxfs/releases#release-v0.90.132 ... please test
 
Gilberto Ferreira said:
And yes... It's work as a charm...
View attachment 100411
Click to expand...
0.90.132 is published: https://github.com/sshoecraft/meshxfs/releases/tag/v0.90.132

For this issue it changes what mxfs-drbd@ does when something else promotes the resource:

  • If Pacemaker manages the resource (an ocf:linbit:drbd resource for it, or a Filesystem resource for its device) or the resource has become-primary-on, mxfs-drbd@ now refuses to start, and journalctl -u mxfs-drbd@<resource> names which one. Before, it retried the mount without end and said nothing.
  • If the peer was made Primary by hand (drbdadm primary) with nothing mounted there, the node says so after its first refused mount, with the command that lets it go ahead.
Running the DRBD resource under Pacemaker is not supported in 0.90.132. The supported setup is the one in docs/drbd-setup.md: mxfs-drbd@ alone brings DRBD up, promotes it and mounts, drbd.service is disabled, and neither the resource nor the mount is in Pacemaker. With that setup, after both nodes of a pair go down, both mount again once they are back.
 
Gilberto Ferreira said:
Hi again
I suppose that with drbd 9 doesn't work.
I tried and get this:
proxmox01:~# mount -t mxfs /dev/drbd0 /data
mount: /data: fsconfig() failed: mxfs: P-DRBD-ARM-REFUSED minor=0 -- replication protocol is '' (configured 'C'), not C; a DRBD device is admitted only with its fence evidence (set the resource up as docs/drbd-setup.md shows).
dmesg(1) may have more information after failed mount system call.
proxmox01:~# dmesg | tail
[Tue Oct 20 13:22:48 2026] drbd r0/0 drbd0: gen-obligation: MINTING(promoted) -> DISCHARGED(promoted)
[Tue Oct 20 13:22:48 2026] mxfs (drbd0): MXFS envelope v1: XFS data at offset 4672712704 (9126392 sectors) proto_gen=22 dirshard=off cluster=(none)
[Tue Oct 20 13:22:48 2026] mxfs: force_transport=1 -> TCP transport
[Tue Oct 20 13:22:48 2026] mxfs: P-DRBD-ARM-REFUSED minor=0 -- replication protocol is '' (configured 'C'), not C; a DRBD device is admitted only with its fence evidence (set the resource up as docs/drbd-setup.md shows)
[Tue Oct 20 13:22:49 2026] mxfs: P-HRX-STATS queued=0 inline=0 max_depth=0 max_wait_ms=0 asks_first=0 asks_merged=0 ask_max_wait_ms=0 -- page hand-offs from peers, processed on their own worker
[Tue Oct 20 13:22:49 2026] mxfs (drbd0): MXFS DLM init failed -- aborting mount of cluster (envelope) volume
[Tue Oct 20 13:22:49 2026] drbd r0: Preparing cluster-wide state change 1828632018: 0->all role( Secondary )
[Tue Oct 20 13:22:49 2026] drbd r0: State change 1828632018: primary_nodes=0, weak_nodes=0
[Tue Oct 20 13:22:49 2026] drbd r0: Committing cluster-wide state change 1828632018 (0ms)
[Tue Oct 20 13:22:49 2026] drbd r0: role( Primary -> Secondary ) [mount:4795 auto-demote]

cat /etc/drbd.d/r0.res
resource r0 {
startup {
wfc-timeout 15; # Tempo aguardando o par se ele estava online antes (em segundos)
degr-wfc-timeout 15; # Tempo aguardando o par se ele já estava desligado antes
}

net {
protocol C;
fencing resource-only;
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
max-buffers 32k;
sndbuf-size 2M;
rcvbuf-size 2M;
}
handlers {
### fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
fence-peer "/usr/sbin/mxfs-drbd-fence-peer";
unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
}
connection-mesh {
hosts proxmox01 proxmox02;
}
on proxmox01 {
address 172.19.150.10:7788;
node-id 0;
device /dev/drbd0;
disk /dev/vdb;
meta-disk internal;
}
on proxmox02 {
address 172.19.150.20:7788;
node-id 1;
device /dev/drbd0;
disk /dev/vdb;
meta-disk internal;
}
disk {
discard-zeroes-if-aligned yes;
rs-discard-granularity 4k;
no-disk-flushes;
no-disk-barrier;
c-plan-ahead 0;
resync-rate 180M;
c-fill-target 2M;
c-min-rate 30M;
c-max-rate 210M;
}
}
Click to expand...
MeshXFS supports DRBD 8.4 at this time, the driver built into the Proxmox kernel (drbd-utils, without drbd-dkms). Your r0.res is DRBD 9 (connection-mesh, node-id), and DRBD 9's /proc/drbd carries no per-device state, which is what the replication protocol is '' refusal means.

For 8.4, use the resource from docs/drbd-setup.md (https://github.com/sshoecraft/meshxfs/blob/main/docs/drbd-setup.md) as written. MeshXFS also requires fencing resource-and-stonith (in disk {} on 8.4) and disconnect for all three after-sb-* policies. Then systemctl disable drbd and systemctl enable --now mxfs-drbd@r0, and keep the resource and the mount out of Pacemaker.
 
I am doing some tests here, but everything in VM on top of Virt-Manager, using vmbc as ipmi.
It's works as intendent.
If the node 02 dies, the node 01 remains, but not the other way around.
But I guess that was planned...
I look forward to be able to used it with 2 nodes on top of Pacemaker with IPMI fance, via iLO or iDrac, as I already do, but using OCFS2, which is not suitable for vm unless you activate io_sync = threads, and uefi or tpm image doesn't support it.
 
Gilberto Ferreira said:
I am doing some tests here, but everything in VM on top of Virt-Manager, using vmbc as ipmi.
It's works as intendent.
If the node 02 dies, the node 01 remains, but not the other way around.
But I guess that was planned...
I look forward to be able to used it with 2 nodes on top of Pacemaker with IPMI fance, via iLO or iDrac, as I already do, but using OCFS2, which is not suitable for vm unless you activate io_sync = threads, and uefi or tpm image doesn't support it.
Click to expand...

hrmm if you have a 2 node and 02 dies 01 should remain AND if 01 dies 02 should remain - if it doesn't thats a problem. it's currently working on 4k sector support for .134 release and then i'll ask it to do Pacemaker/IPMI fence ... i no longer have ilo/idrac systems to test on so I'm not able to test locally.
 
  • Like
Reactions: Gilberto Ferreira
melkier said:
i no longer have ilo/idrac systems to test on so I'm not able to test locally.
Click to expand...
I usually do my tests with virt-manager and vbmc ans a fence ipmi device. Works fine.
melkier said:
if you have a 2 node and 02 dies 01 should remain AND if 01 dies 02 should remain
Click to expand...
That was not I experience here.
If I turn 01 down, 02 restart. But not the other way around.
Perhaps is just me doing something wrong.
 
melkier said:
hrmm if you have a 2 node and 02 dies 01 should remain AND if 01 dies 02 should remain - if it doesn't thats a problem. it's currently working on 4k sector support for .134 release and then i'll ask it to do Pacemaker/IPMI fence ... i no longer have ilo/idrac systems to test on so I'm not able to test locally.
Click to expand...

I was incorrect about this:
The tester's result is how a DRBD pair behaves by default. With no QDevice and no node fence configured, participant 0 (the lower DRBD address) wins every lost link. So when participant 0 dies, participant 1 can't tell that from a cut
cable: it freezes, restarts and waits. The guide says so in docs/drbd-setup.md §7 ("pve1 dies"), and the ruling lists it under "What it does not cover".
- With a QDevice (authority=quorum), either node takes over. That was fixed and verified in 0.90.128.
- With IPMI, the fix is missing. agent=exec would let either node fence the other, but it needs a program that speaks MeshXFS's fence protocol, and no IPMI adapter ships. A site with vbmc, iLO or iDRAC therefore still gets the default.

I've added this as a "defect" and will be in the next release (.135+) the next release (.134) is 4K sector support.
 
You must log in or register to reply here.
Top Bottom
Back