GlusterFS very slow write speed - replication

rootus

New Member
Dec 15, 2014
6
0
1
Germany
www.huza.ro
Hi guys, I'm planning to move everything from SolusVM to Proxmox, due to the fact that SolusVM does not support HA.
Everything went fine until I ended up testing GlusterFS as shared storage, I'm using replication over 3 nodes and when I started creating containers I noticed that the process was really slow, then I decided to test the speed of read/write and this is what I got:

Code:
#### Read from Gluster/Write to Gluster - 3.55 MB/s

s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso
sending incremental file list
debian-7.7.0-amd64-netinst.iso
   232783872 100%    3.60MB/s    0:01:01 (xfer#1, to-check=0/1)

sent 232812383 bytes  received 31 bytes  3724998.62 bytes/sec
total size is 232783872  speedup is 1.00



#### Read from Gluster/Write to ext3 - 148.01 MB/s

root@s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso /root/tmp.iso
sending incremental file list
rsync: link_stat "/mnt/pve/gluster/template/iso/tmp.iso" failed: No such file or directory (2)
debian-7.7.0-amd64-netinst.iso
   232783872 100%  148.18MB/s    0:00:01 (xfer#1, to-check=0/1)

sent 232812385 bytes  received 31 bytes  155208277.33 bytes/sec




#### Read from ext3/Write to Gluster - 3.02 MB/s

root@s014:/mnt/pve/gluster/template/iso# rsync -av --progress /root/tmp.iso tmpx.iso
sending incremental file list
tmp.iso
   232783872 100%    3.07MB/s    0:01:12 (xfer#1, to-check=0/1)

sent 232812360 bytes  received 31 bytes  3167515.52 bytes/sec
total size is 232783872  speedup is 1.00


#### Read from ext3/Write to the ext3 formatted disk on top of which Gluster is set up - 88.81 MB/s

root@s014:/gluster# rsync -av --progress /root/tmp.iso tmpx.iso
sending incremental file list
tmp.iso
   232783872 100%  119.72MB/s    0:00:01 (xfer#1, to-check=0/1)

sent 232812360 bytes  received 31 bytes  93124956.40 bytes/sec
total size is 232783872  speedup is 1.00


##################################################################

/dev/mapper/pve-root on / type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sda2 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/mapper/glustervg-gluster on /gluster type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
storage014:s01 on /mnt/pve/gluster type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)


##################################################################

Volume Name: s01
Type: Replicate
Volume ID: 105cf651-3353-43b6-a207-33f95592a23h
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: storage014:/gluster/s
Brick2: storage015:/gluster/s
Brick3: storage024:/gluster/s

Read speed of 150MB/s is OK, but 3MB/s to write is really unacceptable, any idea where to start looking at least? I tested the disks on all systems and all seem to be OK, as you may see above, writing to Gluster is the issue here and I'm clueless on what could be the cause.
 
Hi guys, I'm planning to move everything from SolusVM to Proxmox, due to the fact that SolusVM does not support HA.
Everything went fine until I ended up testing GlusterFS as shared storage, I'm using replication over 3 nodes and when I started creating containers I noticed that the process was really slow, then I decided to test the speed of read/write and this is what I got:

Code:
#### Read from Gluster/Write to Gluster - 3.55 MB/s

s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso
sending incremental file list
debian-7.7.0-amd64-netinst.iso
   232783872 100%    3.60MB/s    0:01:01 (xfer#1, to-check=0/1)

sent 232812383 bytes  received 31 bytes  3724998.62 bytes/sec
total size is 232783872  speedup is 1.00
[/quote]
Hi,
rsync is not the best tool for measuring read/write speed! Because of overhead for encryption, syncing? and so on.

Slightly better is dd and much better things like fio.
[quote]

#### Read from Gluster/Write to ext3 - 148.01 MB/s

root@s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso /root/tmp.iso
sending incremental file list
rsync: link_stat "/mnt/pve/gluster/template/iso/tmp.iso" failed: No such file or directory (2)
debian-7.7.0-amd64-netinst.iso
   232783872 100%  148.18MB/s    0:00:01 (xfer#1, to-check=0/1)

sent 232812385 bytes  received 31 bytes  155208277.33 bytes/sec
[/quote]
if you don't flush the buffer before, you have only meassure the write-to-ext3 speed with de/encryption (CPU)!
Because the file /mnt/pve/gluster/template/iso/tmp.iso are allready in the buffer (or partly).

Try to messure writespeed something like this (fill an file with 4GB of zeros):
[code]
dd if=/dev/zero of=/mnt/pve/gluster/bigfile bs=1024k count=4096 conv=fdatasync
Clear the buffer
Code:
echo 3 > /proc/sys/vm/drop_caches
After that, you can read the file again
Code:
dd if=/mnt/pve/gluster/bigfile of=/dev/null bs=1024k
But I'm not an gluster expert - so I can't help for further tuning for gluster.

You can use iperf to control the network-speed between the nodes, perhaps there is something wrong.

Udo
 
Something to measure against (SSD disks and 20GB Infiniband using XFS as gluster file system)

root@esx1:~# dd if=/dev/zero of=/mnt/pve/gfs1/dd.img bs=1024K count=4096 conv=fdatasync
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 19.4307 s, 221 MB/s
root@esx1:~# echo 3 > /proc/sys/vm/drop_caches
root@esx1:~# dd if=/mnt/pve/gfs1/dd.img of=/dev/null bs=1024K
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 17.859 s, 240 MB/s
 
Thank you very much for your replies. I always performed the echo 3 > /proc/sys/vm/drop_caches, so the tests you saw were "real" sync speed. I wanted to use a real file with actual content, not something with zeros - transfer speed proved to be ~30% higher when transferring a "zero" file over infiniband with rdma, this is why I avoid them.
Anyway, the issue just went away, now I get like 90+ MB/s. Absolutely nothing was changed since the tests were performed and today I just saw that write speed looks normal. I suspect one of the disks from one of the servers to be the cause, anyone can suggest a good disk testing tool? I can't understand what happened, nobody else has access to the systems and it's curious why out of a sudden things began to work, I don't know Gluster to have some kind of "warm up" or sync time periods when performance drops - like hw raid sync / rebuild.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!