ZFS bad Performance!

[6uellerbpanda]

arc_summary output pl

with a mirrored vdev you get the write performance of one hdd and for reads it can read from both.

your seagate hdd are also not very fast you will get max raw 60 IOPS (IOPS = 1000/(Seek Latency + Rotational Latency) and when I look at your fio stuff it looks fine regarding your hardware.

what's your workload on these vm's when you experience io delay ?
I'm pretty sure when you make heavy seq io writes you'll always get io delay with that zpool.

 

[ramrot]

We have increased value Raw_Read_Error_Rate 0x000f 076 064 006 Pre-fail Always - 35880624 and Raw_Read_Error_Rate 0x000f 076 064 006 Pre-fail Always - 35887408 for the

root@De-Vre-Prox13:~# smartctl -a /dev/sdc
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM005-2DP166
Serial Number:    ZDH1BF2Y

root@De-Vre-Prox13:~# smartctl -a /dev/sdd
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM005-2DP166
Serial Number:    ZGY00T9Z

Can you please check the cabling of the disks?

 

[Daniel Luyckx]

arc_summary output pl

with a mirrored vdev you get the write performance of one hdd and for reads it can read from both.

your seagate hdd are also not very fast you will get max raw 60 IOPS (IOPS = 1000/(Seek Latency + Rotational Latency) and when I look at your fio stuff it looks fine regarding your hardware.

what's your workload on these vm's when you experience io delay ?
I'm pretty sure when you make heavy seq io writes you'll always get io delay with that zpool.

Hello,
Here the Output: https://pastebin.com/nG3daevg

We have increased value Raw_Read_Error_Rate 0x000f 076 064 006 Pre-fail Always - 35880624 and Raw_Read_Error_Rate 0x000f 076 064 006 Pre-fail Always - 35887408 for the

root@De-Vre-Prox13:~# smartctl -a /dev/sdc
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM005-2DP166
Serial Number:    ZDH1BF2Y

root@De-Vre-Prox13:~# smartctl -a /dev/sdd
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM005-2DP166
Serial Number:    ZGY00T9Z

Can you please check the cabling of the disks?

The Server is a Dell R710 with SAS Cables on the HBA. In my experience, a failure in that connection is resulting in very bad Performance, what i cant explain to myself with fio.

Greetings

 

[6uellerbpanda]

arc_summary looks fine.
is there any reason why you didn't add a slog ?

 

[Daniel Luyckx]

arc_summary looks fine.
is there any reason why you didn't add a slog ?

I dont even know what slog is

Im gonna search about it.

Do i need SAS Drives for ZFS ?`

I just testet the File transfer this morning and i can transfer a 10 GB File with 100 MB / s now, i didnt testet with small files or even greater files this morning.

 

[Daniel Luyckx]

By the way, thanks very much for your fast replys, never had such a good Support in a Community Forum

 

[ramrot]

Do i need SAS Drives for ZFS ?`

I have installations of zfs on HDD ( WD RED ) and SDD ( Samsung PRO ), mirors and zraid2, which are all consumer grade. In my opinion, If you don't have real production needs, then those SATA drives would be enough

 

[Daniel Luyckx]

I have installations of zfs on HDD ( WD RED ) and SDD ( Samsung PRO ), mirors and zraid2, which are all consumer grade. In my opinion, If you don't have real production needs, then those SATA drives would be enough

Alright, yes, its just a Homelab with later about 30 TB Space and various "Gaming" Servers and RDP / Terminal, Nextcloud Servers..

With Hyper V and NTFS i could use Consumer grade SSDs... But now the IO Delay was peaking to death with them until the Server got unresponsive. So i am running now only Samsung P883 SSDs in my Servers.

 

[mailinglists]

Well, sorry to disappoint, but 2 x HHDs in mirros on ZFS will always be painfully slow in my experience.
The first usable config with HDDs in my use cases is at minimum of 10 HDDs.

However what you could try is:
1. Enable SLOG for pool (create partitions on ssds and add them as log device for pool, no need to mirror them: zpoll add pool log /dev/sda5 /dev/sdb5 - or use partition IDs), l2arc is not needed when you have so much RAM (BTW you can set min size of ARC).
2. Disable ZIL for pool, but be careful, as you can loose data in RAM, not yet written to diss, without underlying guest knowig about it, on power loss. You can do this dangerous operation with zfs set sync=disabled

There are many other tweaks that can be made, like zfs set logbias=throughput which might hep your use case. Tuning ZFS can be hard and time consuming, but if you start by testig these two and understanding what they do, you _might_ find a suitable solution for you.

 

[Daniel Luyckx]

Well, sorry to disappoint, but 2 x HHDs in mirros on ZFS will always be painfully slow in my experience.
The first usable config with HDDs in my use cases is at minimum of 10 HDDs.

However what you could try is:
1. Enable SLOG for pool (create partitions on ssds and add them as log device for pool, no need to mirror them: zpoll add pool log /dev/sda5 /dev/sdb5 - or use partition IDs), l2arc is not needed when you have so much RAM (BTW you can set min size of ARC).
2. Disable ZIL for pool, but be careful, as you can loose data in RAM, not yet written to diss, without underlying guest knowig about it, on power loss. You can do this dangerous operation with zfs set sync=disabled

There are many other tweaks that can be made, like zfs set logbias=throughput which might hep your use case. Tuning ZFS can be hard and time consuming, but if you start by testig these two and understanding what they do, you _might_ find a suitable solution for you.

Thanks for your Answer. It is very Helpful to know now, that it is normal that the Array performs that bad.

As it is no Option for me to buy a new Server for Homelab and put 10 Disks in it, i would rather suggest to go full SSD now here, since the Prices for SSDs are low as never before. I want to know your Opinion:

An (Usually ZFS RAID 10?) Array, 8x 500 GB Samsung Evo or Kingston A400.. How would them Perform ?
That would be enough Space for my homelab and it *should* be very fast. But ZFS has its own rules.. My 2x Kingston A400 as Mirror were enormous slow..

Greetings

 

[mailinglists]

I am not familiar with Kingston A400, but Samsung EVO / PRO will be faster than HDDs but still painfully slow, as well pose a significant threat for data loss on power failure. Take a look at intel D3 SSD line of disks and similar. Just look at the enterprise SSDs from any provider really. They are really cheap nowdays also.

Based on price / performance / space / durability ratio I would suggest two or four of SSDSC2KB960G801. I get them for around 160 EUR + VAT. But seems like the price has gone up to 170 EUR in the last month, due to popularity and scarcity.

And yes for IOPS do ZFS RAID 10, but in you choose these SSDs and really need more space and are willing to expand the array further, you can test first also with RAIDZ+2. Might be fast enought for your home lab and more secure as any two of the disks can fail before you loose your data, opos to RAID 10 where only two not in mirror part can fail at the same time.

 

[Daniel Luyckx]

I am not familiar with Kingston A400, but Samsung EVO / PRO will be faster than HDDs but still painfully slow, as well pose a significant threat for data loss on power failure. Take a look at intel D3 SSD line of disks and similar. Just look at the enterprise SSDs from any provider really. They are really cheap nowdays also.

Based on price / performance / space / durability ratio I would suggest two or four of SSDSC2KB960G801. I get them for around 160 EUR + VAT. But seems like the price has gone up to 170 EUR in the last month, due to popularity and scarcity.

And yes for IOPS do ZFS RAID 10, but in you choose these SSDs and really need more space and are willing to expand the array further, you can test first also with RAIDZ+2. Might be fast enought for your home lab and more secure as any two of the disks can fail before you loose your data, opos to RAID 10 where only two not in mirror part can fail at the same time.

Thank you very much for your recommendation. I will have a look at it

 

[LnxBil]

Do i need SAS Drives for ZFS ?`

The main difference is that in general, the firmware of SAS drives are optimized for enterprise, so that errors are faster reported (guaranteed response time) than in ordinary consumer hardware (SATA). Yet this is a disk firmware setting, not a SAS/SATA problem, but SATA are often consumer grade firmware, where errors do not need to be reported so quickly and it is tried to read the data multiple times in order to increase the probability to read it.

Another thing you can check if it is slow: Look at the per disk io response time with iostat -x 5 and look at the await statistics.