SSD HW Raid 10 or ZFS Raid 10 SSD

[Whatever]

Ok guys thanks.

I setup 2 servers just to test ZFS performance and put a few very busy VPS servers on it.

First server HW RAID 10 SATA Enterprise 7200rpm disks (x 6 ) with BBU (writeback) and second server ZFS (same disks also x6 in ZRAID 10)

VPS servers run fine on HW RAID but when I move them to server with ZFS they slow down alot.

Is ZFS only good with SSD as I see most here with SSDs? We want to host max 25 smallish VPS Servers per server - or could this CPU be the bottle neck? Intel Xeon E5-1620 3.5GHz

Also note each server had 64GB ECC Memory and gave 32GB of memory to ZFS and ARC.

You have forgotten 2 very important tune options for ZFS configuration that might be to be used for proper comparison:
1) use additional SSD for ZIL (with good synchronous write performance - Intel 3500/3700 series for example)
2) set "zfs set sync=disabled tank/dataset" with UPS (this options completely disable ZIL functionality for given dataset and dramatically improve write performance especially with non-SSD disks)

Note: you can't use 1) and 2) at the same time

 

[mir]

set "zfs set sync=disabled tank/dataset"

This is generally a bad advice if your data is important.

with UPS

An UPS cannot safe the impact of a bad disk.

 

[Whatever]

This is generally a bad advice if your data is important.

As far as I know this option is not safe in cases of unexpected power loss. It can cause a data corruption from an application point of view it doesn't impact ZFS on-disk consistency

An UPS cannot safe the impact of a bad disk.

Neither could do ZIL (it's just transaction log) or what you mean by "bad disk" ?

 

[sahostking]

I thought the same thing about disabling sync.

Anyway tell me about this.

I setup 2 servers - exact same specs

server 1: I setup using RAIDZ2 during setup with 6 disks (500GB Crucial MX200) - I know they not the best disks but jsut for testing.

server 2: I set up in using ZRAID10 during setup with 6 disks (same as above)

Now when I run performance tests I get this:

server1 in raidz2:

root@vz-jhb-2:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0

errors: No known data errors
root@vz-jhb-2:~# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 1462 MB in 3.00 seconds = 487.09 MB/sec
root@vz-jhb-2:~# dd if=/dev/zero of=/temp.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.224252 s, 4.7 GB/s

Server 2 in RAID10 ZFS:

root@vz-jhb-3:~# dd if=/dev/zero of=/temp.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.24305 s, 4.3 GB/s
root@vz-jhb-3:~# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 1466 MB in 3.00 seconds = 488.32 MB/sec
root@vz-jhb-3:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0

errors: No known data errors


Speed is so close - So are we saying using RAIDz2 should be better as it gives more space with basically same performance which is weird unless I'm doing the wrong tests

Also as this is pure SSD do I actually need ZIL and ARC Disk for it. Hence 5 disks in RAIDZ2 and the other for zil and arc ? better or not necessary?

 

[Whatever]

Speed is so close - So are we saying using RAIDz2 should be better as it gives more space with basically same performance which is weird unless I'm doing the wrong tests

your tests (using hdparm or dd with options you set) could not be used for performance comparision. Use fio with appropriate profile

Also as this is pure SSD do I actually need ZIL and ARC Disk for it. Hence 5 disks in RAIDZ2 and the other for zil and arc ? better or not necessary?

in case when your SSDs has good synchronous write performance there is no big difference whether to use additional SSD for ZIL or not but in case of non-enterprise SSDs (like your Crucial) using additional SSD for ZIL would give you noticeable write performance gain (once again: for ZIL it's very important to ensure that SSD has good write performance in synchronous mode)

 

[Whatever]

For everyone interesting whether it's safe to disable ZIL or not read this:
http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/47947

However probably there could be some special cases (crushes) when disabling ZIL would finally corrupt the data itself

 

[sahostking]

ok thanks these will be cPanel KVM VPS Servers which basically are mail,web and mysql database queries. Shared hosting. Does anyone know if FSYNCS are really important in these types of environments.

Also we had a scenario where Hardware raid failed. IF that happens we in worse situation than ZFS having 30seconds of data loss.

 

[mir]

ok thanks these will be cPanel KVM VPS Servers which basically are mail,web and mysql database queries. Shared hosting. Does anyone know if FSYNCS are really important in these types of environments.

Also we had a scenario where Hardware raid failed. IF that happens we in worse situation than ZFS having 30seconds of data loss.

When you have mail - and database servers you cannot use sync=disabled.

 

[BuffaloBrent]

I haven't done any comparison testing with the syncing option, but I can say that there is a dramatic difference overall if you use the cache and log option in ZFS. I have a single SSD that I use for the boot/OS and created a 8G partition for log and 32G for cache on it as well. 32G of cache is a bit overkill for my setup, but without giving you a bunch of boring numbers, it's well worth the small 40G of space to use these features. I have 14 spinning disks in a RAID10 setup, and I'm getting write speed that are just a hair under what my SSD is giving. Read speeds are actually faster than my SSD, but I'm sure part of that is the RAID setup. Of course the IOPS can't compare, but for heavy database use, I'm getting better performance than when I had a dedicated machine with a single SSD.