ZFS bad Performance!

[Daniel Luyckx]

Hello and thanks for Hosting this Forum!

I am using Proxmox since Years now, but i am testing ZFS in my homelab.

I had Consumer SSDs with the Proxmox Host in Mirror 1 at the Beginning. Very Bad idea! I bought Samsung P883 and the Problem with IO Delay seems to be better.

I bought 2x 4 TB Seagate Barracuda now and put them in Mirror. But i am experiencing very high IO Delay again! The Writes are going very down and the Server gets a little bit unresponsive.

I cant explain the Problems.. There is much RAM on the System and Obviosly Proxmox is allocating very much, the VMs are just about 16 GB RAM together.
I discussed very much in Facebook Groups with other Proxmox Users but we didnt found the Problem.

I will attach some Screenshots.

With Greetings from Germany,
Daniel Luyckx

 

[mir]

If you want performance as in IO the only way to go is using RAID 10 (striped mirrors). More stripes means higher performance. The explanation given here is very god: https://www.youtube.com/watch?v=GuUh3bkzaKE

 

[Daniel Luyckx]

If you want performance as in IO the only way to go is using RAID 10 (striped mirrors). More stripes means higher performance. The explanation given here is very god: https://www.youtube.com/watch?v=GuUh3bkzaKE

Hello, thanks for your answer.
The Problem is, that the Performance is FAR under the Performance of single Disks and that is a Fail in my opinion, i think it should be far better than single disk, because of the ARC and L2ARC for example.

We are not talking about a Upgrade to the single Disc Configuration, i would be happy with a Write of 50 MB / s what is absolutely not much. But i am effective at a write of 10-30 MB / s.

 

[Daniel Luyckx]

By the way: The LED on the Drive cages at the Servers (Dell R710) are just lighting for one second, then they are one second turned off, so i assume that the Filesystem writes just half the Time on the Disks. The Servers got
16 x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2 Sockets)
and
24 x Intel(R) Xeon(R) CPU X5670 @ 2.93GHz (2 Sockets)

CPUs and one 90 GB and the other 160 GB RAM. Bot Host VMs dont use more than 50% of the System RAM, Linux (or ZFS) allocates almost the entire rest.

 

[ramrot]

I would check two things for the zfs:

1. What is the configured ashift. You can find it with zdb | grep ashift
2. Do you have enabled the LZ4 compression on zfs?

Also on what governor cpu frequency are you running?

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

could you also share zpool status?

 

[Daniel Luyckx]

I would check two things for the zfs:

1. What is the configured ashift. You can find it with zdb | grep ashift
2. Do you have enabled the LZ4 compression on zfs?

Also on what governor cpu frequency are you running?

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

could you also share zpool status?

Hello, thanks for your Reply. I got a ashift of 12.

Compression is on everywhere:

root@De-Vre-Prox13:~# zfs get compression
NAME PROPERTY VALUE SOURCE
De-Vre-Prox13Mirror4TB compression on local
De-Vre-Prox13Mirror4TB/vm-101-disk-0 compression on inherited from De-Vre-Prox13Mirror4TB
rpool compression on local
rpool/ROOT compression on inherited from rpool
rpool/ROOT/pve-1 compression on inherited from rpool
rpool/data compression on inherited from rpool
rpool/data/vm-101-disk-0 compression on inherited from rpool
rpool/data/vm-101-disk-1 compression on inherited from rpool
rpool/data/vm-104-disk-0 compression on inherited from rpool
rpool/data/vm-104-disk-1 compression on inherited from rpool
rpool/data/vm-105-disk-0 compression on inherited from rpool
rpool/data/vm-105-disk-1 compression on inherited from rpool
rpool/data/vm-106-disk-0 compression on inherited from rpool
rpool/data/vm-108-disk-0 compression on inherited from rpool

Ashift is 12:

root@De-Vre-Prox13:~# zdb | grep ashift
ashift: 12
ashift: 12

CPU frequency should be around 2.6 ghz, its pending from 1.8 ghz to 2.6 when i open it again and again.

root@De-Vre-Prox13:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
cat: /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors: No such file or directory
root@De-Vre-Prox13:~# cd /sys/devices/system/cpu
root@De-Vre-Prox13:/sys/devices/system/cpu# cd cpu0
root@De-Vre-Prox13:/sys/devices/system/cpu/cpu0# cat /proc/cpuinfo | less
root@De-Vre-Prox13:/sys/devices/system/cpu/cpu0# cat /proc/cpuinfo | less
root@De-Vre-Prox13:/sys/devices/system/cpu/cpu0# cat /proc/cpuinfo | less
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
stepping : 2
microcode : 0x1a
cpu MHz : 2527.089
cache size : 12288 KB
physical id : 1
siblings : 8
core id : 0
cpu cores : 4
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
bogomips : 4788.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:


Should i turn off compression or even set the Power Management in BIOS to maximum? It is on OS control right now, and i think maximum would pull very much more Power for the Server.

Greetings

 

[ramrot]

Those look pretty good. We could also try adding a little load on the disks and check the metrics. You can install the fio tool with apt install fio and run the following command

fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

This will create a 4G file named test on the current directory and perform read/writes on it. Please run this command on a directory inside the zfs storage.

Also it would be usefull while running the command to give as the output of vmstat 1 and zpool iostat -v 1. If you don't mind also share the output of zpool status

 

[Daniel Luyckx]

Those look pretty good. We could also try adding a little load on the disks and check the metrics. You can install the fio tool with apt install fio and run the following command

fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

This will create a 4G file named test on the current directory and perform read/writes on it. Please run this command on a directory inside the zfs storage.

Also it would be usefull while running the command to give as the output of vmstat 1 and zpool iostat -v 1. If you don't mind also share the output of zpool status

Alright, im gonna do that! But i will let some Backups roll full back on the Disk first (50 GB Backup Now one hour restoring over 1 Gbit connection)

I will send the Results here!

 

[Daniel Luyckx]

Those look pretty good. We could also try adding a little load on the disks and check the metrics. You can install the fio tool with apt install fio and run the following command

fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

This will create a 4G file named test on the current directory and perform read/writes on it. Please run this command on a directory inside the zfs storage.

Also it would be usefull while running the command to give as the output of vmstat 1 and zpool iostat -v 1. If you don't mind also share the output of zpool status

Here is the Output Sir:
https://pastebin.com/SVdkrzbG

 

[ramrot]

The fio metrics are pretty poor. Also something does not seem right with the vmstat and zpool iostat commands. Just for clarification, were those two commands also running while you were running the fio tool?

 

[Daniel Luyckx]

The fio metrics are pretty poor. Also something does not seem right with the vmstat and zpool iostat commands. Just for clarification, were those two commands also running while you were running the fio tool?

No, i ran them one after the another, as it stands in the Pastebin :=)

I have really no idea what i am doing wrong.. Its basically the Proxmox configuration, i didnt changed anything...

 

[ramrot]

Could you please try to run those commands concurrently (you can open more terminals with ssh to the host or your can use a terminal multiplexer), as we want to check those metrics while the fio tool is running. I'm sure that you haven't made any misconfiguration. I think that maybe the hardware is acting strange.

Also let's check the cpu frequncy that is set properly. Please install cpufreq with apt install cpufrequtils and share the output of cpufreq-info

 

[Daniel Luyckx]

Sir, i would like to test another Power Mode in the Server (not OS control in the Dell, i want to test system maximum performance)

 

[ramrot]

Sir, i would like to test another Power Mode in the Server (not OS control in the Dell, i want to test system maximum performance)

Yes that is surely an option that we can try and see if that fixes the performance

 

[Daniel Luyckx]

Yes that is surely an option that we can try and see if that fixes the performance

Alright, sadly the Power was already set to maximum in the Dell, ive set it to OS control now. Im wondering, why the CPU Frequency was not always maximum as i testet it.

Here the Output sir: https://pastebin.com/pEg3J89n

vmstat: https://pastebin.com/uKxJbW9W

FIO: https://pastebin.com/8i3pNbKE

iostat: https://pastebin.com/iJCgj2NN

All three at the same time.

Power management is now on OS control on the Server, no more on maximum.

 

[ramrot]

Im wondering, why the CPU Frequency was not always maximum as i testet it.

From the first link with the frequtil the governor is set to ondemand, which means that the kernel can decide on which power mode to put the cpu. We can leave it to ondemand since the most of the time the cpu is working at full speed.

I noticed that you have run fio in /root/ and /tmp and i assume that the De-Vre-Prox13Mirror4TB zfs pool is mounted there. One more thing that i would like to ask is, if you have the disks managed by a disk controller or if those are managed as devices by the zfs?

 

[Daniel Luyckx]

From the first link with the frequtil the governor is set to ondemand, which means that the kernel can decide on which power mode to put the cpu. We can leave it to ondemand since the most of the time the cpu is working at full speed.

I noticed that you have run fio in /root/ and /tmp and i assume that the De-Vre-Prox13Mirror4TB zfs pool is mounted there. One more thing that i would like to ask is, if you have the disks managed by a disk controller or if those are managed as devices by the zfs?

The Disks are managed by ZFS, they are directly on a HBA.

The Host is on the 500 GB SSD Mirror with Samsung SM883a SSDs.

I just testet again, the IO delay peeks..

 

[ramrot]

Do you also have the smartmontools installed? Can you share the output of smartctl -a /dev/sdX for each of the disks to also check those info?

 

[ramrot]

I would also check iDrac or IPMI for any hardware logs regarding cpu, power supply and disk controller

 

[Daniel Luyckx]

Do you also have the smartmontools installed? Can you share the output of smartctl -a /dev/sdX for each of the disks to also check those info?

SMART: https://pastebin.com/ktZf635q

Ive looked in Idrac. There is no Problem, all Settings are green and Online.

CPU temps seems finde to me:
root@De-Vre-Prox13:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +27.0°C (high = +85.0°C, crit = +95.0°C)
Core 1: +26.0°C (high = +85.0°C, crit = +95.0°C)
Core 9: +25.0°C (high = +85.0°C, crit = +95.0°C)
Core 10: +27.0°C (high = +85.0°C, crit = +95.0°C)

coretemp-isa-0001
Adapter: ISA adapter
Core 0: +23.0°C (high = +85.0°C, crit = +95.0°C)
Core 1: +20.0°C (high = +85.0°C, crit = +95.0°C)
Core 9: +26.0°C (high = +85.0°C, crit = +95.0°C)
Core 10: +22.0°C (high = +85.0°C, crit = +95.0°C)

I really want that ZFS to get working.. I need it for my Configuration... :=)

Greetings