PVE with glusterfs as shored storage and "Filesystem gets read only"/On Boot

[Apollon77]

Hi All,

I use PVE 5.x (upgrade to 6.x pending till the corosync probles are solved) with a 7 node HA cluster. On the nodes beside PVE also glusterfs (3-way-replicated) is installed as shared fs.
In general it works well. I get problematic situations if the glusterfs gets readonly for whatever reason. Then the VMs just hang and also a shutdown or such is not really possible. I tried all PVE options like Reset and such that I know. Basically I end up with the "hard way" to click stop and then manually kill the kvm process on the machine.

Is there any chance to also get such a "kill" function also in PVE? Or is there any setting for KVM/Qemu I can do to handle such "underlaying FS got read only" in a better way? Especially also it stayed read-only after GlusterFS was back working. A solution that automatically fixes such status would also help a lot in an HA featureset

Ingo

 

[Dominic]

That means you clicked stop on your host but it didn't work (like shutdown) and that is why you had to kill the process?

 

[Apollon77]

That means you clicked stop on your host but it didn't work (like shutdown) and that is why you had to kill the process?

Yes correct. I tried "Stop" ... also "Reset" after some time ... but the process was not ended till I manually killed it (or 20-30 mins was not enough time)

And BTW: I use HA feature, so HA manager is involved too on stopping the vm

 

[Dominic]

Is there a chance that you could narrow this

gets readonly for whatever reason

down?

I installed GlusterFS on a virtual 3-node PVE 6 cluster and started a VM that is stored on a Gluster volume. Then I set

gluster volume set gv0 features.read-only on

While shutting down the VM via GUI did not work

TASK ERROR: VM quit/powerdown failed - got timeout

clicking stop did really stop my VM and the KVM process was gone, too.

 

[Apollon77]

Is there a chance that you could narrow this

down?

Sure: Make sure the glusterfs looses client-quorum

So when you have 3 node cluster (setup as replica 2 or such) and then turn off two machines ... then it looses client quorum and everything should get blocked. Then (in my cases because it were just watchdog reboots but bad timing) the machines both came up again and so quorum was there again ... vms were blocked still.

So best way:
- kill two machines ... you should see that filesystem goes to readonly in the VMs or such
- start machnes again to get glusterfs back
- try to get vms restarting

BTW: One case I also had was that one VM crashed and "rebootet" because of the fs topic and then it went into a boot loop because fs was readonly ... but also after the glusterfs was back working.
So another thing to try could be "start a VM while the two machines are down" ...

For me it was pve 5 so far ... I now prepare to move to 6 in the next time

 

[Dominic]

I set up a three node cluster with a Gluster volume with replication. One node has two VMs on this volume. Stopping the other two nodes lead to I/O errors and a read-only filesystem when I wanted to do something in the VMs. However, after starting the nodes and stopping and restarting the VMs everything seemed to be back to normal.

Could you please post

gluster volume info
gluster volume get YOUR_VOLUME all | grep quorum

 

[Apollon77]

Could it be that the updated Qemu, kvm and such in PVE 6 reacts better to such cases as the ones included in PVE 5?

root@pm1:~# gluster volume info
 
Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x 3 = 15
Transport-type: tcp
Bricks:
Brick1: 192.168.178.50:/gluster/brick1/gv0
Brick2: 192.168.178.76:/gluster/brick1/gv0
Brick3: 192.168.178.96:/gluster/brick2/gv0
Brick4: 192.168.178.50:/gluster/brick2/gv0
Brick5: 192.168.178.81:/gluster/brick1/gv0
Brick6: 192.168.178.94:/gluster/brick1/gv0
Brick7: 192.168.178.50:/gluster/brick3/gv0
Brick8: 192.168.178.82:/gluster/brick1/gv0
Brick9: 192.168.178.94:/gluster/brick2/gv0
Brick10: 192.168.178.50:/gluster/brick4/gv0
Brick11: 192.168.178.95:/gluster/brick1/gv0
Brick12: 192.168.178.94:/gluster/brick3/gv0
Brick13: 192.168.178.50:/gluster/brick5/gv0
Brick14: 192.168.178.96:/gluster/brick1/gv0
Brick15: 192.168.178.94:/gluster/brick4/gv0
Options Reconfigured:
storage.fips-mode-rchecksum: on
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 4
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: diff
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
cluster.granular-entry-heal: enable


root@pm1:~# gluster volume get gv0 all | grep quorum
cluster.quorum-type                     auto                                   
cluster.quorum-count                    (null)                                 
cluster.server-quorum-type              server                                 
cluster.server-quorum-ratio             0                                       
cluster.quorum-reads                    no

 

[Apollon77]

PS: My problem was that after getting the i/o errors a stop of the KVM did not succeeded

 

[Dominic]

I am still using type "Replicated" instead of "Distributed-Replicated", but the same quorum settings for my Gluster cluster. So far, stopping the VMs still works perfectly when the filesystem gets read-only and the I/O errors happen.

Could it be that the updated Qemu, kvm and such in PVE 6 reacts better to such cases as the ones included in PVE 5?

Could be.

 

[Apollon77]

I've already upgraded 3 of my 7 nodes to PVE6 ... rest to come the next days ... then I could check it again maybe ...