Proxmox GlusterFS performance tests

[ricardoj]

Hi,

This message is to ask you guys if anybody else can share gluster experience as a Proxmox shared storage.

I decided to give GlusterFS a try and the performance was really bad when I started.

As I have only one environment where I can test it, all the tests have the same machines and I only change configurations.

There are 4 physical machines : 2 Proxmox nodes and 2 NAS

PVE-01 PVE-02 NAS-01 NAS-02

I normally run VMs "hosted" in NAS-02 which is newer and faster and NAS-01 which is old and slower is a backup server

Both NAS are running OMV ( Open Media Vault 4.X ) configured to provide NFS and CIFS

PVE-01 and PVE-02 are connected to NAS-02 using NFS and VM performance is good.

I installed GlusterFS on both NAS machines and configure it as replicated ( it was really easy ).

There is a separate network with jumbo frames enabled to interconnect NAS-01 NAS-02 gluster "brick" traffic.

I know there is no 3rd node ( arbiter ) but this is a test environment in a local LAN.

Next step I configure gluster storage from PVE-01 Datacenter and restore a VM backup to this gluster storage.

VM works but very slow.

After some more tests I decided to change NAS configuration and created an OMV NFS on top of "local" GlusterFS on NAS-01.

I removed Gluster configuration from PVE-01 Datacenter and create a NFS Storage pointing to NAS-02.

Again restored a VM backup and it works as fast as the previous NFS share accessing local disks.

Then I decided to test again gluster storage from PVE-01 Datacenter and the performance was really bad.

I returned to GlusterFS over NFS ( not NFS Ganesha ) as I'm running OMV NFS and performance is back to normal.

Can anyone comment / shed a light on what else can be done to diagnose why GlusterFS client connection seems to be so slow ?

Direct connection using gluster-client its supposed to be faster or as fast as the NFS connection.

A very important point here is that I'm running GlusterFS version 6.3.1 and I upgraded gluster-client and gluster-common on both PVE nodes.

So when I created a gluster storage from PVE-01 Datacenter it was running on the new version of gluster-client.

By default PVE uses version 3.8.8 for gluster-client which is not compatible with gluster-server version 6.3.1.

It is also important to note that I always checked that the files were created in glusterfs. This is true for PVE direct connection and PVE NFS connection.


===============================

GlusterFS is using shards

root@nas-02:~# gluster volume info
Volume Name: gluster01
Type: Replicate
Volume ID: c82432fc-41ec-422f-8898-9b9fa3ce3b3f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gpeer-01.XYZ
Brick2: gpeer-02.XYZ
Options Reconfigured:
features.shard-block-size: 32MB
features.shard: enable
client.event-threads: 2
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
===============================

===============================
This is the mount command used by OMV ( remote mount package ) to access gluster on local node

gpeer-02:/gluster01 on /srv/9733531f-71aa-42d1-b456-0d00e9942a89 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072,_netdev)
===============================


Thank you for your time and attention reading this.

Refrences :

https://access.redhat.com/documenta...ration_guide/sect-creating_replicated_volumes

https://www.linux-kvm.org/images/5/51/03x04-Martin_Sivak-oVirt_and_gluster_hyperconverged.pdf

https://lenovopress.com/lp1148.pdf

Regards,

Ricardo Jorge


In no way I'm associated with any companies and / or projects depicted in this message.

 

[ricardoj]

Hi,

GlusterFS releases :

https://www.gluster.org/release-schedule/

5 Maintained 2018-10-23
6 Current 2019-03-25

Regards,

Ricardo Jorge

 

[ricardoj]

Hi,

I'm still running my tests over gluster and today I tested a container and the performance was very good.

I think this is good point to be noted as some time ago people noticed performance problems with gluster + containers.

See here : https://forum.proxmox.com/threads/container-on-gluster-volume-not-possible.40889/

Regards,

Ricardo Jorge

 

[ricardoj]

Hi,

Searching the threads, I found this one :

https://forum.proxmox.com/threads/bug-proxmoxs-qemu-and-debian-8-virtio-drivers-as-vm-guest.22882/

It seems that this problem (GlusterFS performance) has been around for a long time without any action.

Regards,

Ricardo Jorge

 

[Dominic]

There are some discussions about Gluster performance on the gluster-users mailing list that might be interesting:

• Regarding version 3.8 vs newer ones
• Regarding NFS-Ganesha

 

[ricardoj]

Hi,

Thank you @Dominic for the update.

At least for me the way GlusterFS "balance" the nodes is too slow to use it as an online storage.

For VM backup and other backup needs its a good price / security option especially when you consider remote nodes ( geographically distributed nodes ).

Its also easy to configure, maintain and upgrade.

Regards,

Ricardo Jorge

 

[Dominic]

At least for me the way GlusterFS "balance" the nodes is too slow to use it as an online storage.

Have you tried Ceph already? We did a benchmark with it not too long ago.

 

[ricardoj]

Hi,

I think Ceph could be an option, but hardware requirements are out of my reach at the moment.

Regards,

Ricardo Jorge

 

[DANILO MONTAGNA]

I used GlusterFS last year on one of ours OpenStack Cluster, the performance is good while there is no problem on any brick node... my experience says that if you are going to use glusterfs with large RAW files, be aware that it will freeze your brain when a file-healing occurs at one of your brick nodes running glusterfs... it will take DAYS depending the number of VMs and the size of the guest disk images you have to glusterfs recover from a split-brain or node failure. Also, dont use any kind of file storage over network if you don't have a reliable network (10GBit or faster)..

If I could give you an advice... I would say, move to a BLOCK storage filesystem instead, like CEPH RDB.