low fsyncs with GlusterFS

[gbr]

Hi,

I had an iSCSI/DRBD setup which I switched over to GlusterFS. First I tried NFS/DRBD, and although it worked, throughput went fron 118 MB/s to 35 MB/s with the secondary DRBD attached.

Now with gluster, I'm getting the expected throughput, 58MB/s (about 1/2 of GigE due to Gluster replication), but I'm getting really bad fsyncs. I know it's not the underlying hatrdware, since it worked beautifully under iSCSI.

root@proxmox-3:/etc/pve/qemu-server# pveperf /mnt/pve/RAID6-Gluster/
CPU BOGOMIPS: 19999.36
REGEX/SECOND: 973072
HD SIZE: 5586.67 GB (192.168.20.1:RAID6)
FSYNCS/SECOND: 37.73
DNS EXT: 36.40 ms
DNS INT: 0.69 ms


The reason I'd like to switch from iSCSI to NFS/Gluster is to get snapshotting capabilities.

When I create or move a disk to the Gluster storage, PVE times out. The file gets created, but it takes a long time.

Gerald


ps: with iSCSI i was getting fsyncs/s of over 1000

 

[spirit]

glusterfs through fuse is not super fast.
If you use kvm, you can use glusterfs storage type, which access directly gluster like a block device. (but you can't snapshot).

I don't known what it your hardware, but maybe an ssd for journal could improve speed.

 

[gbr]

Hi,

I used the glusterfs mount option through the Proxmox GUI, which I assumed used the gluster library.

Unfortunately, it's snapshots I want.

Gerald

 

[spirit]

Hi,

I used the glusterfs mount option through the Proxmox GUI, which I assumed used the gluster library.

Unfortunately, it's snapshots I want.

Gerald

Proxmox use only the mountpoint for create/delete volume, but qemu directly access to gluster.
So it's faster in your vm than from proxmox host.
Maybe can you try some benchmark from your guest vm.

(But snapshot are not supported yet by gluster)

 

[gbr]

Hi,

Ahhh! That's how it works. Unfortunately, I get timeouts on creating a disk.

Gerald