Hi guys, I'm planning to move everything from SolusVM to Proxmox, due to the fact that SolusVM does not support HA.
Everything went fine until I ended up testing GlusterFS as shared storage, I'm using replication over 3 nodes and when I started creating containers I noticed that the process was really slow, then I decided to test the speed of read/write and this is what I got:
Read speed of 150MB/s is OK, but 3MB/s to write is really unacceptable, any idea where to start looking at least? I tested the disks on all systems and all seem to be OK, as you may see above, writing to Gluster is the issue here and I'm clueless on what could be the cause.
Everything went fine until I ended up testing GlusterFS as shared storage, I'm using replication over 3 nodes and when I started creating containers I noticed that the process was really slow, then I decided to test the speed of read/write and this is what I got:
Code:
#### Read from Gluster/Write to Gluster - 3.55 MB/s
s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso
sending incremental file list
debian-7.7.0-amd64-netinst.iso
232783872 100% 3.60MB/s 0:01:01 (xfer#1, to-check=0/1)
sent 232812383 bytes received 31 bytes 3724998.62 bytes/sec
total size is 232783872 speedup is 1.00
#### Read from Gluster/Write to ext3 - 148.01 MB/s
root@s014:/mnt/pve/gluster/template/iso# rsync -av --progress debian-7.7.0-amd64-netinst.iso tmp.iso /root/tmp.iso
sending incremental file list
rsync: link_stat "/mnt/pve/gluster/template/iso/tmp.iso" failed: No such file or directory (2)
debian-7.7.0-amd64-netinst.iso
232783872 100% 148.18MB/s 0:00:01 (xfer#1, to-check=0/1)
sent 232812385 bytes received 31 bytes 155208277.33 bytes/sec
#### Read from ext3/Write to Gluster - 3.02 MB/s
root@s014:/mnt/pve/gluster/template/iso# rsync -av --progress /root/tmp.iso tmpx.iso
sending incremental file list
tmp.iso
232783872 100% 3.07MB/s 0:01:12 (xfer#1, to-check=0/1)
sent 232812360 bytes received 31 bytes 3167515.52 bytes/sec
total size is 232783872 speedup is 1.00
#### Read from ext3/Write to the ext3 formatted disk on top of which Gluster is set up - 88.81 MB/s
root@s014:/gluster# rsync -av --progress /root/tmp.iso tmpx.iso
sending incremental file list
tmp.iso
232783872 100% 119.72MB/s 0:00:01 (xfer#1, to-check=0/1)
sent 232812360 bytes received 31 bytes 93124956.40 bytes/sec
total size is 232783872 speedup is 1.00
##################################################################
/dev/mapper/pve-root on / type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sda2 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/mapper/glustervg-gluster on /gluster type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
storage014:s01 on /mnt/pve/gluster type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
##################################################################
Volume Name: s01
Type: Replicate
Volume ID: 105cf651-3353-43b6-a207-33f95592a23h
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: storage014:/gluster/s
Brick2: storage015:/gluster/s
Brick3: storage024:/gluster/s
Read speed of 150MB/s is OK, but 3MB/s to write is really unacceptable, any idea where to start looking at least? I tested the disks on all systems and all seem to be OK, as you may see above, writing to Gluster is the issue here and I'm clueless on what could be the cause.