PVE with glusterfs as shored storage and "Filesystem gets read only"/On Boot


Well-Known Member
Sep 24, 2018
Hi All,

I use PVE 5.x (upgrade to 6.x pending till the corosync probles are solved) with a 7 node HA cluster. On the nodes beside PVE also glusterfs (3-way-replicated) is installed as shared fs.
In general it works well. I get problematic situations if the glusterfs gets readonly for whatever reason. Then the VMs just hang and also a shutdown or such is not really possible. I tried all PVE options like Reset and such that I know. Basically I end up with the "hard way" to click stop and then manually kill the kvm process on the machine.

Is there any chance to also get such a "kill" function also in PVE? Or is there any setting for KVM/Qemu I can do to handle such "underlaying FS got read only" in a better way? Especially also it stayed read-only after GlusterFS was back working. A solution that automatically fixes such status would also help a lot in an HA featureset :)

That means you clicked stop on your host but it didn't work (like shutdown) and that is why you had to kill the process?
That means you clicked stop on your host but it didn't work (like shutdown) and that is why you had to kill the process?
Yes correct. I tried "Stop" ... also "Reset" after some time ... but the process was not ended till I manually killed it (or 20-30 mins was not enough time)

And BTW: I use HA feature, so HA manager is involved too on stopping the vm
Is there a chance that you could narrow this
gets readonly for whatever reason

I installed GlusterFS on a virtual 3-node PVE 6 cluster and started a VM that is stored on a Gluster volume. Then I set
gluster volume set gv0 features.read-only on

While shutting down the VM via GUI did not work
TASK ERROR: VM quit/powerdown failed - got timeout

clicking stop did really stop my VM and the KVM process was gone, too.
Is there a chance that you could narrow this


Sure: Make sure the glusterfs looses client-quorum :)

So when you have 3 node cluster (setup as replica 2 or such) and then turn off two machines ... then it looses client quorum and everything should get blocked. Then (in my cases because it were just watchdog reboots but bad timing) the machines both came up again and so quorum was there again ... vms were blocked still.

So best way:
- kill two machines ... you should see that filesystem goes to readonly in the VMs or such
- start machnes again to get glusterfs back
- try to get vms restarting

BTW: One case I also had was that one VM crashed and "rebootet" because of the fs topic and then it went into a boot loop because fs was readonly ... but also after the glusterfs was back working.
So another thing to try could be "start a VM while the two machines are down" ...

For me it was pve 5 so far ... I now prepare to move to 6 in the next time
I set up a three node cluster with a Gluster volume with replication. One node has two VMs on this volume. Stopping the other two nodes lead to I/O errors and a read-only filesystem when I wanted to do something in the VMs. However, after starting the nodes and stopping and restarting the VMs everything seemed to be back to normal.

Could you please post
gluster volume info
gluster volume get YOUR_VOLUME all | grep quorum
Could it be that the updated Qemu, kvm and such in PVE 6 reacts better to such cases as the ones included in PVE 5?

root@pm1:~# gluster volume info
Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x 3 = 15
Transport-type: tcp
Options Reconfigured:
storage.fips-mode-rchecksum: on
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 4
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: diff
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
cluster.granular-entry-heal: enable

root@pm1:~# gluster volume get gv0 all | grep quorum
cluster.quorum-type                     auto                                   
cluster.quorum-count                    (null)                                 
cluster.server-quorum-type              server                                 
cluster.server-quorum-ratio             0                                       
cluster.quorum-reads                    no
PS: My problem was that after getting the i/o errors a stop of the KVM did not succeeded
I am still using type "Replicated" instead of "Distributed-Replicated", but the same quorum settings for my Gluster cluster. So far, stopping the VMs still works perfectly when the filesystem gets read-only and the I/O errors happen.
Could it be that the updated Qemu, kvm and such in PVE 6 reacts better to such cases as the ones included in PVE 5?
Could be.
I've already upgraded 3 of my 7 nodes to PVE6 ... rest to come the next days ... then I could check it again maybe ...
  • Like
Reactions: Dominic


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!