ZFS RAID Types and Speed

[matthew]

I currently have several KVMs running inside a simple proxmox host that do a lot of random disk io. The proxmox host is using two sata drives in a RAID 1 type array. I was thinking of moving to 6 drives in a RAID 6 type array. Would that improve disk io speed significantly or not?

I found this on ZFS performance.

https://calomel.org/zfs_raid_speed_capacity.html

I always heard that RAID 5 and 6 performed poorly compared too RAID 10 but this indicates differently. I like the redundancy that RAID 6 offers.

 

[Knuuut]

I always heard that RAID 5 and 6 performed poorly compared too RAID 10 but this indicates differently. I like the redundancy that RAID 6 offers.

Just add another 2 drives and create a striped RAIDZ2 pool. But, with 8 drives you may also create a 4-way striped mirror with a lot more performance by identical capacity and almost identical redundance.

 

[Klaus Steinberger]

much more important than the max read/write speeds for you deployment is the IOPS.

Common sense for ZFS is that one VDEV is as fast as one of the drives in terms of IOPS for random reads/writes.

So for your deployment I would recommend a setup of 3 mirrored VDEV in one pool instead of a RaidZ2 setup. This setup will give you three times the IIOPS speed than your current setup.
Maybe you can add a small fast SSD as a ZIL device. This will speed up sync writes tremendously.

 

[matthew]

What would you recommend as a ZIL device? Does it need to be mirrored as well? What happens if it fails?

 

[wolfgang]

What would you recommend as a ZIL device?

A small SSD with good 4k write properties, like the Intel Optane

Does it need to be mirrored as well?

This is a personal decision on how important are your data.
I think 99% of the Data are not that worth.

What happens if it fails?

If the Server is running normal, nothing will happen. The ZIL will never be read if no power outage happens.
If a power outage happens and at this moment your ZIL fails your data on the ZIL what is not written to the HDD are lost.

 

[Knuuut]

If the Server is running normal, nothing will happen. The ZIL will never be read if no power outage happens.
If a power outage happens and at this moment your ZIL fails your data on the ZIL what is not written to the HDD are lost.

And this would be the last 5 seconds of writes in worst case. Data that already has been written is not affected.

 

[matthew]

If you were putting 4 7200 rpm sata drives in a proxmox zfs array and cared most about random i/o reads and writes and redundancy how should you set them up? What about 6 drives or 8 drives?

 

[udo]

If you were putting 4 7200 rpm sata drives in a proxmox zfs array and cared most about random i/o reads and writes and redundancy how should you set them up? What about 6 drives or 8 drives?

Hi,
I know it's not very helpfull, but...

If you care about "random i/o reads and writes and redundancy" you don't use 4 sata 7200 hdds!

Udo

 

[guletz]

Hi,
As Klaus said before, one good ideea for you is to use a 3x mirror vdev.

Another option is to use 2 x 3 way mirror (much better read iops, better redundancy, but you lose usable raw storage capacity)

Anyway, for zfs random write is not a big problem, because in moste cases zfs will convert this random writes io in to sequential io. Also the NCQ from hdd will help.

 

[matthew]

Think I am going to bump to 6 drives and 2 x 3 way mirror as suggested.

Would this be a good ZIL device?
https://ark.intel.com/products/9754...-Series-16GB-M_2-80mm-PCIe-3_0-20nm-3D-Xpoint

It states it requires a i7 processor. Will it work with an E5 processor and "M.2 PCIe to PCIe 3.0 x4 Adapter"?

 

[Knuuut]

Afaik a M.2 NvME SSD will work with a PCIe-Adaptor even on an old PCIe 1.0 Boards, but with less performance...
Anyway, you schould watch out for an Enterprise-Grade SSD with Power-Loss-Protection.

 

[matthew]

So perhaps one of these would be a better choice?
https://ark.intel.com/products/74949/Intel-SSD-DC-S3500-Series-80GB-2_5in-SATA-6Gbs-20nm-MLC

 

[Knuuut]

No, bad IO performance on writes and low endurance.

Example fpr NvME: Seagate Nytro 5000 Mixed-Workload 1.5DWPD 400GB, M.2 (XP400HE30002)
Example for PCIe: Intel SSD DC P3700 400GB, PCIe 3.0 x4 (SSDPEDMD400G401)
Example for SATA: Intel SSD DC S4600 480GB, 2.5", SATA (SSDSC2KG480G701)

NvME/PCIe is 1st choice, sata 2nd...