HW raid-6 to ZFS Raid1

[Alessandro 123]

Stupid question, but I admit, I was never been able to measure IOPS and read/write throuhgput.

I have a Xen node with 8x SAS 15k disks in hardware RAID-6 (512MB write cache) that I would like to replace with a Proxmox server.

My idea is: can I replace this server with a new one by using ZFS 3-way RAID-1 (SAS 7.2 disks) and an SSD as ZIL/L2ARC ?

Performance would be much lower ? How many IOPS should I expect with a ZFS 3way mirror VS an HW RAID6 with 8 disks?

 

[Alessandro 123]

No suggestions?

 

[tom]

No suggestions?

Just run benchmarks with fio on your hardware and you will see. Performance depends on the disk model, the controller settings, the SSD model, etc.

 

[Alessandro 123]

Just run benchmarks with fio on your hardware and you will see. Performance depends on the disk model, the controller settings, the SSD model, etc.

I'm asking because I don't have a new server to benchmark.

 

[tom]

I'm asking because I don't have a new server to benchmark.

at least, provide the benchmark from the old one, include the details of your benchmark tools/config (I suggest fio) so maybe other can run the benchmark on their hosts for a comparison.

 

[Alessandro 123]

I don't have fio installed on XenServer and I prefere to not install additional software on this junk environment.
I have "sysstat", thus I can provide you "iostat"

 

[tom]

For reliable benchmarks, I recommend fio.

 

[udo]

Stupid question, but I admit, I was never been able to measure IOPS and read/write throuhgput.

I have a Xen node with 8x SAS 15k disks in hardware RAID-6 (512MB write cache) that I would like to replace with a Proxmox server.

My idea is: can I replace this server with a new one by using ZFS 3-way RAID-1 (SAS 7.2 disks) and an SSD as ZIL/L2ARC ?

Performance would be much lower ? How many IOPS should I expect with a ZFS 3way mirror VS an HW RAID6 with 8 disks?

Hi,
you will get much much worser performance!
8 SAS 15k disks in raid 6 are normaly fast for reading and the speed for writing is pimped by the 512MB write cache.

To get an reliable speed with 7.2 disks (ok, sas - but in real live near line) in raid-1 (from performance view an bad raid level) you need an very good SSD (see Sebastian Han list of good ceph journal ssds) and an workload, which benefit from read-cache. If you often read new, uncached data, your speed will be very low.

I use in production four server with SSDs in zfs raid-1 (2*1.6TB Intel SSD), and as failover an pool with 4*Sata 7.2k as raid-10.
The speed of the ssd-pool is ok, but not so good, like the sas-raid (also 8 disk if i remember right), where the images stay before.
The raid-10 sata-pool is for emergency only - with the normal work load you can't realy use that (if one node die - i move the images back on the sas-raid).

Use an good nvme-disk as cache and/or SSD-only setup. If you can, use raid-10 and not raid1!

Udo

 

[kobuki]

Striped RAID1 vdevs x3 (~RAID10) and SSD slog + L2ARC on SSD vs. HWRAID RAID6 with a relatively small 512 MB cache. Why would the former be objectively slower? For what type of workload? What is the target/acceptable IOPS? The only thing to be wary of usually is the higher IOWAIT with ZFS, when using direct attach SAS/SATA disks because there is no controller to hide away the wait states in IO behind the cache. The rest entirely depends on other requirements (like FS features) and the real workload. The ARC does a great job for read caching and SSD as SLOG usually handles random writes pretty well. Some more info on the workload and the planned usage pattern would help in deciding anyway.

 

[udo]

The only thing to be wary of usually is the higher IOWAIT with ZFS

right, and higher iowait say: slower, because the processes must wait for IO.

And yes, you are right 512MB cache is not much, but look at the cache of the sata disk (used for simulate speed) they are not realy big (like 16MB) and must switched of for data security with zfs.
sso the only writespeed is done with the SSD cache.

My comparision between LVM-SAS-Raid-6 and ZFS-SSD-Raid-1 shows an zeroing unsused space inside an VM Windows guest of app. 400MB/s on the SAS-Raid and (if I remember right) app. 150MB/s with the ZFS-Raid-1.

Udo

 

[kobuki]

right, and higher iowait say: slower, because the processes must wait for IO.

That's not really accurate. The general iowait the kernel shows you is not necessarily tied to your actual workload, most often it has nothing to do with it (I mean, it they are connected but not directly because of coalescing, used queuing principle and depth, etc.). It's a general measure globally or per device. When you have an ssd slog, it's just as fast or sometimes even faster than a raid card with bbwc for random writes, when chosen carefully. Write latency on spinning disks, generally creating that bigger iowait, play no role here. For streaming workloads it's always the bottom layer (spinning disks) is what counts, they are going to pose the real bottleneck.

My comparision between LVM-SAS-Raid-6 and ZFS-SSD-Raid-1 shows an zeroing unsused space inside an VM Windows guest of app. 400MB/s on the SAS-Raid and (if I remember right) app. 150MB/s with the ZFS-Raid-1.
Udo

The 2 mentioned setups are a bit like apples and oranges. A raid6 topology with ~6 disks will give you several hundred MB/s for sequential writes, 400 sounds about right. OTOH, writing to a single zfs mirror is like writing to a single disk. That's about 150 MB/s nowadays. Sequentially writing to a single hw-backed mirror would provide about the same speed.

 

[Alessandro 123]

My use case are web hosting VMs. There is almost 0 sequential write like in any web hosting VM.

More or less, I have to migrate 4-5 VM, with about 400-500 sites each, that's why an L2ARC could be usefull, where ZFS will store most read files from the VM. (if ZFS is able to cache VM blocks when a visitor will access the site, I don't know)

 

[kobuki]

Average web sites are mostly random read (but of course it depends), you would benefit greatly from a large ARC. L2ARC might or might not worth it, I tend to see low utilisation with them. If you can afford it, put more ram in the server for the ARC instead. I'd suggest going with striped mirror vdevs (raid10-like) for faster initial random access.

 

[Alessandro 123]

Using 4x2TB disks in a RAID10 is not an issue, but I really hate any 2-way mirror (and a RAID10 has 2 2way mirrors inside).
I had multiple full data-loss when using mirror, now all my server are on RAID-6 or at least 3way mirrors. That's why i've talked about 3way mirror

 

[kobuki]

Your alternative would be raidz2 which is similar to raid6, with good sequential read/write, but your read iops might suffer (until you hit the arc). You could be good to go, though, with that kind of workload.

 

[Alessandro 123]

AFAIK, any mirror (RAID1, RAID10, ....) is much faster than any parity RAID. So, a 3way mirror should be better than a RAID-Z2 (and also cheaper, as I need 1 disk less)

 

[kobuki]

They differ in speed, depending on the workload (yeah, again)... More independent spindles = more iops; more stripe sets = more sequential speed. And their various mixes, of course. You can add more than 2 disks to a mirror, if that's your thing, it can increase your random reads too, as reads are supposed to be balanced between them.