Poor performance with ZFS

chalan

Member
Mar 16, 2015
119
2
16
if i have poor r/w with zfs raid 10 4x1TB sata2 disks, i should buy another 1TB SSD disk and add it for ZIL? :(
 

chalan

Member
Mar 16, 2015
119
2
16
yes proxmox 5 fresh install with 4 1TB WD RED sata2 disks and the performace is very poor. copiing from one hp sata2 drive to rpool 20MB/s :( direct in pve not in VM!!!

Code:
  pool: rpool
 state: ONLINE
  scan: none requested
config:

    NAME                                                STATE     READ WRITE CKSUM
    rpool                                               ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J2021886-part2  ONLINE       0     0     0
        ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U6546808-part2  ONLINE       0     0     0
      mirror-1                                          ONLINE       0     0     0
        ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J2AK75T9        ONLINE       0     0     0
        ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1JE0SFR        ONLINE       0     0     0

errors: No known data errors
 

cpzengel

Active Member
Nov 12, 2015
210
10
38
Aschaffenburg, Germany
zfs.rocks
What says arcstat?
Enable Drive Writecache
Try to offline one drive after another
Install Smartmontools

smartctl --test=short /dev/sda
smartctl --test=short /dev/sdb
smartctl --test=short /dev/sdc
smartctl --test=short /dev/sdd


—— wait three minutes

clear
smartctl --all /dev/sda | grep Short
smartctl --all /dev/sdb | grep Short
smartctl --all /dev/sdc | grep Short
smartctl --all /dev/sdd | grep Short
 

chalan

Member
Mar 16, 2015
119
2
16
Code:
root@pve-klenova:~# smartctl --all /dev/sda | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26667         -
root@pve-klenova:~# smartctl --all /dev/sdb | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     43111         -
# 4  Short offline       Completed without error       00%     15243         -
root@pve-klenova:~# smartctl --all /dev/sdc | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26577         -
# 3  Short offline       Completed without error       00%     17443         -
root@pve-klenova:~# smartctl --all /dev/sdd | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26573         -
# 4  Short offline       Completed without error       00%     17443         -
root@pve-klenova:~# smartctl --all /dev/sde | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%       465         -
root@pve-klenova:~# smartctl --all /dev/sdf | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%       465         -
 

LnxBil

Famous Member
Feb 21, 2015
6,660
878
163
Saarland, Germany
I'd also monitor the average io completion time with iostat on heavy load. Normally one bad (non-enterprise-grade) drive can slow down the whole pool.
 

chalan

Member
Mar 16, 2015
119
2
16
the drives are 4x WD RED 1TB NAS... how can i check every single hdd from the zfs pool to find out which one is slow?
 

LnxBil

Famous Member
Feb 21, 2015
6,660
878
163
Saarland, Germany
the drives are 4x WD RED 1TB NAS... how can i check every single hdd from the zfs pool to find out which one is slow?

Stress your pool and look at the iostat times, here an example:

Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,06    0,00    0,09   17,38    0,00   82,47

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
...
sda               0,00     0,00    0,00    0,60     0,00     2,40     8,00     0,00    0,00    0,00    0,00   0,00   0,00
sdb               0,20     0,00    0,60    0,00   245,60     0,00   818,67     0,02   25,33   25,33    0,00   9,33   0,56
sdc               0,20     0,00    0,60    0,00   302,40     0,00  1008,00     0,01   16,00   16,00    0,00   9,33   0,56
sde               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdd               0,20     0,00    0,60    0,00   254,40     0,00   848,00     0,00    6,67    6,67    0,00   5,33   0,32
sdq               0,00     0,00    1,40    0,00     7,20     0,00    10,29     0,01    9,14    9,14    0,00   9,14   1,28
sdn               0,00     0,00    1,00    0,00     6,40     0,00    12,80     0,00    2,40    2,40    0,00   2,40   0,24
sdk               0,00     0,00    6,60    0,00    85,60     0,00    25,94     0,00    0,61    0,61    0,00   0,24   0,16
sdh               0,00     0,00    2,80    0,00    66,40     0,00    47,43     0,00    0,86    0,86    0,00   0,57   0,16
sdo               0,00     0,00    4,00    0,00    92,00     0,00    46,00     0,01    2,60    2,60    0,00   2,40   0,96
sdf               0,00     0,00    3,00    0,00    66,40     0,00    44,27     0,01    3,20    3,20    0,00   3,20   0,96
sdi               0,00     0,00    0,20    0,00     0,80     0,00     8,00     0,00    0,00    0,00    0,00   0,00   0,00
sdl               0,00     0,00    4,00    0,00    92,80     0,00    46,40     0,01    3,40    3,40    0,00   1,40   0,56
sdj               0,00     0,00    5,00    0,00    83,20     0,00    33,28     0,01    2,40    2,40    0,00   0,80   0,40
sdg               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdm               0,00     0,00    7,60    0,00   187,20     0,00    49,26     0,07    9,16    9,16    0,00   3,47   2,64
sdp               0,00     0,00    3,40   13,00   117,60   342,40    56,10    14,73 1005,85 1588,47  853,48  60,98 100,00
...

The drive sdp was the bottleneck and took too much time to complete operation. It still did, but the performance was very, very bad.
 

chalan

Member
Mar 16, 2015
119
2
16
iostat during vm clooning

Code:
Every 1.0s: iostat                         pve-klenova: Sun Oct 15 00:15:46 2017

Linux 4.10.17-2-pve (pve-klenova)       10/15/2017      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.32    0.00    3.69    1.68    0.00   79.30

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              47.76      1082.84       456.83  685286010  289110920
sdb              48.15      1082.20       456.83  684883290  289110920
sdc               1.36        21.45        79.39   13572168   50245208
sdd               1.39        21.41        79.55   13550700   50346472
sde              52.14      1085.24       477.21  686810800  302009308
sdf              51.87      1085.39       477.21  686903024  302009308
sdh               0.00         0.02         0.00      15176          0
zd0              50.54       237.90       145.61  150560533   92152132
zd16              0.00         0.00         0.00         20          0
zd32             46.41         1.92       181.67    1216540  114970932
zd48              1.92         1.36         6.58     862116    4164876
zd64              2.18         4.04         4.68    2558668    2959672
zd80             18.69        18.06        73.96   11427076   46808004

is it ok?
 

GadgetPig

Member
Apr 26, 2016
138
25
18
53
Hi Chalan, you need to install package "sysstat" first
#apt-get install sysstat
run a clone operation and then run:
#iostat -x 5
then watch for a disk with any unusually high values under these columns
| await | r_await | w_await | % util
 

chalan

Member
Mar 16, 2015
119
2
16
sda
sdb
sdf
sdg

are the drives in raid 10 seems to be similar

Code:
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdb               1,40     0,00  210,60  153,80 13179,20  9620,80   125,14     4,44   12,74   20,03    2,75   2,58  94,00
sdc               1,20     0,00  119,40  173,80  8560,00 11289,60   135,40     5,51   19,56   43,46    3,13   3,38  99,12
sdd               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sde               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdf               1,20     0,00  164,40  222,60 10978,40 18026,40   149,90     2,61    6,76   14,16    1,29   1,89  73,20
sdg               0,80     0,00  136,00  223,80  7923,20 18026,40   144,24     2,13    5,97   14,01    1,09   1,86  67,04
zd0               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
zd16              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
zd32              0,00     0,00    0,00  146,40     0,00   584,80     7,99     0,98    6,77    0,00    6,77   6,68  97,76
zd48              0,00     0,00    0,00    3,00     0,00     9,60     6,40     0,19   62,93    0,00   62,93  62,93  18,88
zd64              0,00     0,00    0,00    2,00     0,00     8,00     8,00     1,08  426,00    0,00  426,00 220,40  44,08
zd80              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

root@pve-klenova:~# arcstat
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
14:15:05 30 14 46 1 5 13 100 1 8 4.0G 4.0G
 

GadgetPig

Member
Apr 26, 2016
138
25
18
53

chalan

Member
Mar 16, 2015
119
2
16
so why i have such slow perf?

root@pve-klenova:~# pveperf
CPU BOGOMIPS: 38401.52
REGEX/SECOND: 430906
HD SIZE: 654.48 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 53.92
DNS EXT: 196.53 ms
DNS INT: 18.91 ms (elson.sk)
 

Nemesiz

Well-Known Member
Jan 16, 2009
678
48
48
Lithuania
Example - SSD server

ZFS sync standard

# pveperf
CPU BOGOMIPS: 19154.70
REGEX/SECOND: 1491489
HD SIZE: 33.58 GB (rpool/debian)
FSYNCS/SECOND: 2553.13
DNS EXT: 54.12 ms

ZFS sync disabled
# pveperf
CPU BOGOMIPS: 19154.70
REGEX/SECOND: 1501062
HD SIZE: 33.58 GB (rpool/debian)
FSYNCS/SECOND: 16839.19
DNS EXT: 54.74 ms
 

chalan

Member
Mar 16, 2015
119
2
16
1.) i have 4xWD RED 1TB drives in raid10, do i really need another SSD drive? If so how BIG?

2.) Is the problem in my server LOW RAM or is THIS a standard performance?

3.) when I disable ZFS SYNC, can I expect an unstable system or data loss in case of power failure and so on?

4.) if i disable sync how can i enable it again? and how can i see if sync is enabled or disabled?
 
Last edited:

Nemesiz

Well-Known Member
Jan 16, 2009
678
48
48
Lithuania
1.) i have 4xWD RED 1TB drives in raid10, do i really need another SSD drive? If so how BIG?

Can you print? #zpool status

If you use SSD as ZFS ZIL you need big as = 5 seconds * SSD write speed
Like 5s*500MiB/s = 2.5 GiB

Don`t forget ZFS write data to ZIL linear from beginning every time. IT can eat SSD fast.

2.) Is the problem in my server LOW RAM or is THIS a standard performance?

It`s depends on what your server is doing with data.

3.) when I disable ZFS SYNC, can I expect an unstable system or data loss in case of power failure and so on?

ZFS is very stable and protected with data but more you can read here
https://forum.proxmox.com/threads/p...-ssd-drives-sync-parameter.31130/#post-155543
https://forum.proxmox.com/threads/high-io-load-txg_sync.35566/#post-174995

4.) if i disable sync how can i enable it again? and how can i see if sync is enabled or disabled?

You can set sync option in every zvol.

To see sync parameter everywhere -> # zfs get sync
To specific zvol/file system -> # zfs get sync pool/filesystem
To change -> # zfs set sync=[ standard | always | disabled ] pool/filesystem

Almost all ZFS parameters are tuneable live.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!