I have a three-node cluster, all ZFS. One node is small used just for quorum.
I set up two of the nodes to hold gluster data in a ZFS data set on each. The set up and configuration was pretty straight forward, just gluster peer probe each other after installing the glusterfs-server package.
The volume was created as follows:
The gluster wiki virt-store-usecase settings are:
I moved a VM with a 32GB virtual disk from being on a ZFS volume to the glusterfs. The read/write performance became horribly, horribly slow. About 1/4 to 1/3 the speed on writes, and nearly infinity slower on the reads.
Specifically, a simple dd test run on the original ZFS backed VM:
The speed is pretty variable, ranging from 50 to 100 seconds, but not so bad.
Running the same test on the glusterfs backed VM takes upwards of 260 seconds or more:
some other test runs show even slower, when turning off atime on the underlying ZFS dataset gluster is using (tank/gluster).
The VM uses no-cache on the disk. I tried write-through but that was so slow I just killed the test.
I also tried to run an MD5 checksum on my "bigfile" file, and after an hour I killed it. There's no reason an md5 checksum on that file should take over an hour, especially after I just wrote it. On direct hardware this takes just a few seconds.
Now, I expected some slowdown due to the client-based replication, but this is downright unusable.
My questions are:
1) Is this expected slowdown?
2) If not, what can I do about it?
I set up two of the nodes to hold gluster data in a ZFS data set on each. The set up and configuration was pretty straight forward, just gluster peer probe each other after installing the glusterfs-server package.
The volume was created as follows:
Code:
zfs create tank/gluster
mkdir /tank/gluster/brick
gluster volume create datastore replica 2 transport tcp pve1-cluster:/tank/gluster/brick pve2-cluster:/tank/gluster/brick
gluster volume start datastore
gluster volume info
# create /var/lib/glusterd/groups/virt as per Virt-store-usecase wiki page.
gluster volume set datastore group virtThe gluster wiki virt-store-usecase settings are:
Code:
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=serverI moved a VM with a 32GB virtual disk from being on a ZFS volume to the glusterfs. The read/write performance became horribly, horribly slow. About 1/4 to 1/3 the speed on writes, and nearly infinity slower on the reads.
Specifically, a simple dd test run on the original ZFS backed VM:
Code:
/usr/bin/time dd if=/dev/urandom of=bigfile bs=8k count=204800
204800+0 records in
204800+0 records out
1677721600 bytes transferred in 72.410429 secs (23169613 bytes/sec)
72.41 real 0.00 user 5.60 sysThe speed is pretty variable, ranging from 50 to 100 seconds, but not so bad.
Running the same test on the glusterfs backed VM takes upwards of 260 seconds or more:
Code:
1677721600 bytes transferred in 262.989241 secs (6379431 bytes/sec)some other test runs show even slower, when turning off atime on the underlying ZFS dataset gluster is using (tank/gluster).
Code:
1677721600 bytes transferred in 403.081872 secs (4162235 bytes/sec)The VM uses no-cache on the disk. I tried write-through but that was so slow I just killed the test.
I also tried to run an MD5 checksum on my "bigfile" file, and after an hour I killed it. There's no reason an md5 checksum on that file should take over an hour, especially after I just wrote it. On direct hardware this takes just a few seconds.
Now, I expected some slowdown due to the client-based replication, but this is downright unusable.
My questions are:
1) Is this expected slowdown?
2) If not, what can I do about it?
