Proxmox slow glusterfs

[rickygm]

Hi forum , I have been using GlusterFS to provide file synchronisation over two networked servers , I have mounted three VMs that occupy a total of 400gb, and from the beginning I have performance problems (very slow) , each server has 32gb ram, Core i7, 1GB dual port network card, replication is done with a 1tb disk (7200RPM) on each server.

pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-4-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-7
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-38
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-10
libpve-storage-perl: 5.0-28
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-34
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

 

[LnxBil]

What is "very slow"? How fast is your network connection? One 7.2 krpm is also not that fast... Please provide numbers (search for fio to create a proper benchmark)

 

[Dark26]

Can you tell us more of the gluster configuration ?

 

[rickygm]

my network speed is 1GB with cisco switch 2960g , I'm not an expert in glusterfs, but this is the default configuration :
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
# option transport.address-family inet6
# option base-port 49152
end-volume

 

[tschanness]

You have in total 3x 1 TB 7200RPM HDDs is your Glustercluster. You only have two gigabit speed NICs.
Of course that's slow. Is it too slow? We don't know. We don't even have metrics.

 

[Dark26]

For the gluster, you have a replicate of 3 ? do you have arbitrary, disperse ?


The gluster server is on the proxmox node , or separate servers ?

On my infrastructure, with gluster, the performance is divided by 2 with gluster vs the normal storage, and the cpu utilization is very high

Can you do a gluster volume info ?

root@prox8:~# gluster volume info

Volume Name: Emmc
Type: Distributed-Replicate
Volume ID: 01b5c009-9076-4cea-bc0a-81f0c78dd54e
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x (2 + 1) = 9
Transport-type: tcp
Bricks:
Brick1: 192.168.1.101:/Data/Emmc/Brick1
Brick2: 192.168.1.108:/Data/Emmc/Brick1
Brick3: 192.168.1.106:/Data/Emmc/Brick1_a (arbiter)
Brick4: 192.168.1.101:/Data/Emmc/Brick2
Brick5: 192.168.1.106:/Data/Emmc/Brick2
Brick6: 192.168.1.108:/Data/Emmc/Brick2_a (arbiter)
Brick7: 192.168.1.108:/Data/Emmc/Brick3
Brick8: 192.168.1.106:/Data/Emmc/Brick3
Brick9: 192.168.1.101:/Data/Emmc/Brick3_a (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
auth.allow: /(192.168.1.*,/lxc(192.168.1.*)
server.event-threads: 3
client.event-threads: 3
performance.cache-size: 128MB
performance.io-thread-count: 32

 

[rickygm]

I'm going to build the cluster again and look for faster disks, and I want to upgrade to 10gb ethernet, then I'll post to ask for help and put together a good configuration.