Poor performance with ZFS

if i have poor r/w with zfs raid 10 4x1TB sata2 disks, i should buy another 1TB SSD disk and add it for ZIL? :(
 
yes proxmox 5 fresh install with 4 1TB WD RED sata2 disks and the performace is very poor. copiing from one hp sata2 drive to rpool 20MB/s :( direct in pve not in VM!!!

Code:
  pool: rpool
 state: ONLINE
  scan: none requested
config:

    NAME                                                STATE     READ WRITE CKSUM
    rpool                                               ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J2021886-part2  ONLINE       0     0     0
        ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U6546808-part2  ONLINE       0     0     0
      mirror-1                                          ONLINE       0     0     0
        ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J2AK75T9        ONLINE       0     0     0
        ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1JE0SFR        ONLINE       0     0     0

errors: No known data errors
 
What says arcstat?
Enable Drive Writecache
Try to offline one drive after another
Install Smartmontools

smartctl --test=short /dev/sda
smartctl --test=short /dev/sdb
smartctl --test=short /dev/sdc
smartctl --test=short /dev/sdd


—— wait three minutes

clear
smartctl --all /dev/sda | grep Short
smartctl --all /dev/sdb | grep Short
smartctl --all /dev/sdc | grep Short
smartctl --all /dev/sdd | grep Short
 
Code:
root@pve-klenova:~# smartctl --all /dev/sda | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26667         -
root@pve-klenova:~# smartctl --all /dev/sdb | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     43111         -
# 4  Short offline       Completed without error       00%     15243         -
root@pve-klenova:~# smartctl --all /dev/sdc | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26577         -
# 3  Short offline       Completed without error       00%     17443         -
root@pve-klenova:~# smartctl --all /dev/sdd | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%     26573         -
# 4  Short offline       Completed without error       00%     17443         -
root@pve-klenova:~# smartctl --all /dev/sde | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%       465         -
root@pve-klenova:~# smartctl --all /dev/sdf | grep Short
Short self-test routine
# 1  Short offline       Completed without error       00%       465         -
 
I'd also monitor the average io completion time with iostat on heavy load. Normally one bad (non-enterprise-grade) drive can slow down the whole pool.
 
the drives are 4x WD RED 1TB NAS... how can i check every single hdd from the zfs pool to find out which one is slow?
 
the drives are 4x WD RED 1TB NAS... how can i check every single hdd from the zfs pool to find out which one is slow?

Stress your pool and look at the iostat times, here an example:

Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,06    0,00    0,09   17,38    0,00   82,47

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
...
sda               0,00     0,00    0,00    0,60     0,00     2,40     8,00     0,00    0,00    0,00    0,00   0,00   0,00
sdb               0,20     0,00    0,60    0,00   245,60     0,00   818,67     0,02   25,33   25,33    0,00   9,33   0,56
sdc               0,20     0,00    0,60    0,00   302,40     0,00  1008,00     0,01   16,00   16,00    0,00   9,33   0,56
sde               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdd               0,20     0,00    0,60    0,00   254,40     0,00   848,00     0,00    6,67    6,67    0,00   5,33   0,32
sdq               0,00     0,00    1,40    0,00     7,20     0,00    10,29     0,01    9,14    9,14    0,00   9,14   1,28
sdn               0,00     0,00    1,00    0,00     6,40     0,00    12,80     0,00    2,40    2,40    0,00   2,40   0,24
sdk               0,00     0,00    6,60    0,00    85,60     0,00    25,94     0,00    0,61    0,61    0,00   0,24   0,16
sdh               0,00     0,00    2,80    0,00    66,40     0,00    47,43     0,00    0,86    0,86    0,00   0,57   0,16
sdo               0,00     0,00    4,00    0,00    92,00     0,00    46,00     0,01    2,60    2,60    0,00   2,40   0,96
sdf               0,00     0,00    3,00    0,00    66,40     0,00    44,27     0,01    3,20    3,20    0,00   3,20   0,96
sdi               0,00     0,00    0,20    0,00     0,80     0,00     8,00     0,00    0,00    0,00    0,00   0,00   0,00
sdl               0,00     0,00    4,00    0,00    92,80     0,00    46,40     0,01    3,40    3,40    0,00   1,40   0,56
sdj               0,00     0,00    5,00    0,00    83,20     0,00    33,28     0,01    2,40    2,40    0,00   0,80   0,40
sdg               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdm               0,00     0,00    7,60    0,00   187,20     0,00    49,26     0,07    9,16    9,16    0,00   3,47   2,64
sdp               0,00     0,00    3,40   13,00   117,60   342,40    56,10    14,73 1005,85 1588,47  853,48  60,98 100,00
...

The drive sdp was the bottleneck and took too much time to complete operation. It still did, but the performance was very, very bad.
 
iostat during vm clooning

Code:
Every 1.0s: iostat                         pve-klenova: Sun Oct 15 00:15:46 2017

Linux 4.10.17-2-pve (pve-klenova)       10/15/2017      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.32    0.00    3.69    1.68    0.00   79.30

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              47.76      1082.84       456.83  685286010  289110920
sdb              48.15      1082.20       456.83  684883290  289110920
sdc               1.36        21.45        79.39   13572168   50245208
sdd               1.39        21.41        79.55   13550700   50346472
sde              52.14      1085.24       477.21  686810800  302009308
sdf              51.87      1085.39       477.21  686903024  302009308
sdh               0.00         0.02         0.00      15176          0
zd0              50.54       237.90       145.61  150560533   92152132
zd16              0.00         0.00         0.00         20          0
zd32             46.41         1.92       181.67    1216540  114970932
zd48              1.92         1.36         6.58     862116    4164876
zd64              2.18         4.04         4.68    2558668    2959672
zd80             18.69        18.06        73.96   11427076   46808004

is it ok?
 
Hi Chalan, you need to install package "sysstat" first
#apt-get install sysstat
run a clone operation and then run:
#iostat -x 5
then watch for a disk with any unusually high values under these columns
| await | r_await | w_await | % util
 
sda
sdb
sdf
sdg

are the drives in raid 10 seems to be similar

Code:
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdb               1,40     0,00  210,60  153,80 13179,20  9620,80   125,14     4,44   12,74   20,03    2,75   2,58  94,00
sdc               1,20     0,00  119,40  173,80  8560,00 11289,60   135,40     5,51   19,56   43,46    3,13   3,38  99,12
sdd               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sde               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdf               1,20     0,00  164,40  222,60 10978,40 18026,40   149,90     2,61    6,76   14,16    1,29   1,89  73,20
sdg               0,80     0,00  136,00  223,80  7923,20 18026,40   144,24     2,13    5,97   14,01    1,09   1,86  67,04
zd0               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
zd16              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
zd32              0,00     0,00    0,00  146,40     0,00   584,80     7,99     0,98    6,77    0,00    6,77   6,68  97,76
zd48              0,00     0,00    0,00    3,00     0,00     9,60     6,40     0,19   62,93    0,00   62,93  62,93  18,88
zd64              0,00     0,00    0,00    2,00     0,00     8,00     8,00     1,08  426,00    0,00  426,00 220,40  44,08
zd80              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

root@pve-klenova:~# arcstat
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
14:15:05 30 14 46 1 5 13 100 1 8 4.0G 4.0G
 
so why i have such slow perf?

root@pve-klenova:~# pveperf
CPU BOGOMIPS: 38401.52
REGEX/SECOND: 430906
HD SIZE: 654.48 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 53.92
DNS EXT: 196.53 ms
DNS INT: 18.91 ms (elson.sk)
 
1.) i have 4xWD RED 1TB drives in raid10, do i really need another SSD drive? If so how BIG?

2.) Is the problem in my server LOW RAM or is THIS a standard performance?

3.) when I disable ZFS SYNC, can I expect an unstable system or data loss in case of power failure and so on?

4.) if i disable sync how can i enable it again? and how can i see if sync is enabled or disabled?
 
Last edited:
1.) i have 4xWD RED 1TB drives in raid10, do i really need another SSD drive? If so how BIG?

Can you print? #zpool status

If you use SSD as ZFS ZIL you need big as = 5 seconds * SSD write speed
Like 5s*500MiB/s = 2.5 GiB

Don`t forget ZFS write data to ZIL linear from beginning every time. IT can eat SSD fast.

2.) Is the problem in my server LOW RAM or is THIS a standard performance?

It`s depends on what your server is doing with data.

3.) when I disable ZFS SYNC, can I expect an unstable system or data loss in case of power failure and so on?

ZFS is very stable and protected with data but more you can read here
https://forum.proxmox.com/threads/p...-ssd-drives-sync-parameter.31130/#post-155543
https://forum.proxmox.com/threads/high-io-load-txg_sync.35566/#post-174995

4.) if i disable sync how can i enable it again? and how can i see if sync is enabled or disabled?

You can set sync option in every zvol.

To see sync parameter everywhere -> # zfs get sync
To specific zvol/file system -> # zfs get sync pool/filesystem
To change -> # zfs set sync=[ standard | always | disabled ] pool/filesystem

Almost all ZFS parameters are tuneable live.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!