Poor performance with ZFS

FSYNCS/SEC have nothing to do with ARC. Those should go straight to slog device (your SSD, part1)

The backups arn't written to ZFS but to another EXT3 drive. So only the reads are from ZFS. I believe ZFS tries to put the most frequently used data in ARC, so when a backup is made from every VM each night, maybe this ends up in ARC, poluting the normal reads?? Just a tought
 
I think we are talking two different things here.

You state "But my performance is way slower:" and you show pveperf output. In that output the only FS performance parameter is "FSYNCS/SECOND: 128.89

That parameter is pretty low at ~129 for a sync activity using a slog device.

Other than that you don't present any evidence on slow FS performance, so there is nothing that can be debugged, sorry.



 
I think we are talking two different things here.

You state "But my performance is way slower:" and you show pveperf output. In that output the only FS performance parameter is "FSYNCS/SECOND: 128.89

That parameter is pretty low at ~129 for a sync activity using a slog device.

Other than that you don't present any evidence on slow FS performance, so there is nothing that can be debugged, sorry.




Yes 129 is pretty low, compared to the 2999.24 on -almost- the same system in the post of tom on 03-25:
Code:
root@proxmox:~# pveperf /space/
CPU BOGOMIPS:      57595.44
REGEX/SECOND:      1804289
HD SIZE:           14601.65 GB (space)
FSYNCS/SECOND:     2999.24
DNS EXT:           35.32 ms
DNS INT:           0.52 ms (proxmox.com)

I seen more threads with this problem and if I read them it looks like they all use ZFS as root system.
Or can we drop any comparisson with pvepref's FSYNCS/SEC and ZFS?
 
I have a post above where I suggest running a iostat command during pveperf and see where is the bottleneck.
You mean this:
@sxlderek: open two terminal windows.
In one of them type zpool iostat -v 1
In the other one start the pveperf and check if there is any slog device activity (the one you added).

When done, stop the first command and type iostat -kx 1. Start "pveperf" again in second.

Please paste the output of the first terminal during pveperf runs here.

I can create the two windows, "zpool iostat -v 1" runs in one of them, but when I run "iostat -kx 1" iostat command doesn't exist nor is "zpool iostat -kx 1" (invalid option "k")
 
apt-get install sysstat (that package provides iostat)

And you need to run pveperf along each of "zpool iostat" and "iostat", not the iostats together.

Basically, you need to check with "zpool iostat" and/or "iostat" what disk(s) take the pveperf fsync load.
 
apt-get install sysstat (that package provides iostat)

And you need to run pveperf along each of "zpool iostat" and "iostat", not the iostats together.

Basically, you need to check with "zpool iostat" and/or "iostat" what disk(s) take the pveperf fsync load.

Thank you. Was already searching for a package. Will try that tomorrow and post the results here.
 
apt-get install sysstat (that package provides iostat)

And you need to run pveperf along each of "zpool iostat" and "iostat", not the iostats together.

Basically, you need to check with "zpool iostat" and/or "iostat" what disk(s) take the pveperf fsync load.

Here is the first output, FSYNC was even worse.

Code:
                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     68     93  6.41M  1.65M
  raidz2                                              885G  15.4T     68     78  6.41M   989K
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     14     11  1.04M   381K
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -     15     11  1.12M   379K
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     16     11  1.22M   379K
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     14     11  1.03M   381K
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -     15     11  1.12M   379K
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     16     11  1.22M   379K
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   4.04M  37.2G      0     15      4   703K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E      1     10  88.8K   970K
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     14      0   210K  7.99K
  raidz2                                              885G  15.4T     14      0   210K      0
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -      4      0  40.0K      0
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      1      0  20.0K      0
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      5      0  52.0K      0
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -      1      0  28.0K      0
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      2      0  32.0K      0
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     10      0  79.9K      0
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   4.04M  37.2G      0      0      0  7.99K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     17     25   133K   201K
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     55    293   237K  1.58M
  raidz2                                              885G  15.4T     55    282   237K  1.45M
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -      4     41  36.0K   843K
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      1     49  24.0K   891K
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     28     49   136K   891K
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -      7     50  48.0K   891K
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      3     41  20.0K   859K
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     24     41   104K   843K
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   6.85M  37.2G      0     10      0   124K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     61     24   215K   285K
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     22      0   116K      0
  raidz2                                              885G  15.4T     22      0   116K      0
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -      2      0  12.0K      0
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      1      0  7.99K      0
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     12      0  60.0K      0
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -      1      0  16.0K      0
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      1      0  7.99K      0
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     13      0  56.0K      0
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   6.85M  37.2G      0      0      0      0
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     26     33   116K   197K
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T      0      0  2.50K      0
  raidz2                                              885G  15.4T      0      0  2.50K      0
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -      0      0      0      0
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      0      0      0      0
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      0      0  4.00K      0
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -      0      0      0      0
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      0      0      0      0
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      0      0      0      0
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   6.85M  37.2G      0      0      0      0
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E      0      6  3.00K  37.0K
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     52      5  1.20M   116K
  raidz2                                              885G  15.4T     52      0  1.20M      0
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     43      0   384K      0
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -     29      0   280K      0
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     26      0   248K      0
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     41      0   296K      0
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -     13      0  99.9K      0
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      8      0  75.9K      0
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   6.85M  37.2G      0      5      0   116K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     26      0   460K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     43    364   536K  2.98M
  raidz2                                              885G  15.4T     43    359   536K  2.92M
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     18     39   112K  1.35M
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      8     37  48.0K  1.36M
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     13     38  75.9K  1.36M
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     19     37   156K  1.35M
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -     10     39  87.9K  1.37M
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     10     39  83.9K  1.36M
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   6.85M  37.2G      0      4      0  56.0K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     31      0   257K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     28    292   259K   803K
  raidz2                                              885G  15.4T     28    261   259K   584K
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     16     52  99.9K   588K
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      3     53  36.0K   616K
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      3     52  28.0K   608K
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     17     53  99.9K   612K
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      5     52  40.0K   600K
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      3     52  32.0K   588K
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1    260K  37.2G      0     30      0   220K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E      6      0  69.5K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     27    564  78.9K  1.37M
  raidz2                                              885G  15.4T     27    564  78.9K  1.37M
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     12    119  56.0K  1.28M
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      1    128  12.0K  1.31M
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      0    126  7.99K  1.29M
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     16    126  72.0K  1.29M
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      0    124  4.00K  1.31M
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      0    119  4.00K  1.28M
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1    708K  37.2G      0      0      0      0
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     10      0   165K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     41    606  48.5K  1.62M
  raidz2                                              885G  15.4T     41    421  48.5K   920K
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     11     87  48.0K   971K
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      0     83      0   979K
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      6     83  28.0K   959K
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     11     83  48.0K   959K
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      0     87  4.00K   983K
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     10     87  44.0K   975K
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1    260K  37.2G      0    184      0   739K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E      3      0  28.5K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     64    245   341K   731K
  raidz2                                              885G  15.4T     64    149   341K   295K
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     23     32   132K   336K
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -      5     29  59.9K   336K
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -     16     29  95.9K   332K
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     25     29   108K   332K
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      6     32  32.0K   344K
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -     10     32  79.9K   336K
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1    136K  37.2G      0     95      0   436K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     21      0   245K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     77    477   425K  1.64M
  raidz2                                              885G  15.4T     77    466   425K  1.31M
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     39     72   180K  1.09M
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -     16     87  79.9K  1.12M
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      8     87  40.0K  1.11M
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     43     87   196K  1.11M
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -     13     72  68.0K  1.09M
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      7     72  36.0K  1.09M
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1   1.79M  37.2G      0     10      0   332K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E     39      0   341K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----

                                                        capacity     operations    bandwidth
pool                                                 alloc   free   read  write   read  write
---------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                 885G  15.4T     42    570   334K  1.52M
  raidz2                                              885G  15.4T     42    565   334K  1.44M
    ata-ST3000DM001-1ER166_W500V5P2-part3                -      -     28    111   128K  1.34M
    ata-ST3000DM001-1ER166_W500V5P4-part3                -      -     11    119  56.0K  1.34M
    ata-ST3000DM001-1ER166_W500TF38-part3                -      -      2    116  20.0K  1.32M
    ata-ST3000DM001-1ER166_W500TF0G-part3                -      -     22    118   120K  1.31M
    ata-ST3000DM001-1ER166_W500THXN-part3                -      -      9    109  64.0K  1.35M
    ata-ST3000DM001-1ER166_W500TEVZ-part3                -      -      7    109  44.0K  1.32M
logs                                                     -      -      -      -      -      -
  ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1    260K  37.2G      0      4      0  79.9K
cache                                                    -      -      -      -      -      -
  scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2   278G  16.0E      4      0   162K      0
---------------------------------------------------  -----  -----  -----  -----  -----  -----


CPU BOGOMIPS:      57600.72
REGEX/SECOND:      1879422
HD SIZE:           10876.31 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     99.93
 
And the second:
avg-cpu: %user %nice %system %iowait %steal %idle
5.10 0.00 3.44 2.33 0.00 89.13

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.50 0.20 1.75 34.35 90.67 1672.84 97.70 0.07 2.05 2.68 2.02 1.68 6.07
sdb 0.01 1477.81 1.52 13.35 8.75 5964.65 803.16 2.41 162.09 12.08 179.17 5.31 7.90
sdc 1.16 0.00 13.21 12.35 1060.22 380.90 112.75 0.14 5.67 7.01 4.24 3.40 8.70
sdd 1.16 0.00 14.29 12.31 1151.17 379.45 115.09 0.16 5.96 7.14 4.58 3.52 9.36
sde 1.36 0.00 14.94 12.30 1247.36 378.82 119.38 0.16 5.88 6.99 4.53 3.45 9.40
sdf 1.17 0.00 13.15 12.35 1058.49 380.88 112.90 0.15 5.92 7.18 4.58 3.59 9.14
sdg 1.15 0.00 14.28 12.31 1148.04 379.44 114.91 0.16 5.91 7.11 4.51 3.49 9.27
sdh 1.37 0.00 14.97 12.30 1245.68 378.81 119.13 0.16 5.79 6.85 4.50 3.42 9.32
zd0 0.00 0.00 0.00 0.00 0.00 63.26 96894.09 0.00 12.33 0.24 16.43 0.37 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
3.03 0.00 2.44 3.19 0.00 91.34

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 14.00 23.00 178.50 439.50 33.41 0.09 2.35 3.00 1.96 1.78 6.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 3.00 0.00 24.00 0.00 172.00 0.00 14.33 0.13 5.38 5.38 0.00 2.54 6.10
sdd 0.00 0.00 11.00 0.00 60.00 0.00 10.91 0.06 5.45 5.45 0.00 3.91 4.30
sde 0.00 0.00 9.00 0.00 52.00 0.00 11.56 0.15 16.56 16.56 0.00 5.22 4.70
sdf 6.00 0.00 22.00 0.00 192.00 0.00 17.45 0.13 5.91 5.91 0.00 3.82 8.40
sdg 2.00 0.00 18.00 0.00 116.00 0.00 12.89 0.13 7.78 7.78 0.00 4.78 8.60
sdh 1.00 0.00 14.00 0.00 84.00 0.00 12.00 0.08 5.71 5.71 0.00 3.36 4.70
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
2.59 0.00 1.84 0.50 0.00 95.06

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 3.00 19.00 28.50 139.00 15.23 0.06 2.73 3.00 2.68 2.73 6.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 1.00 0.00 4.00 0.00 8.00 0.00 3.00 3.00 0.00 3.00 0.30
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
3.90 0.00 2.57 2.74 0.00 90.78

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 17.00 12.00 342.50 188.00 36.59 0.05 1.76 1.94 1.50 1.62 4.70
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 25.00 0.00 188.00 0.00 15.04 0.38 15.00 15.00 0.00 3.20 8.00
sdd 0.00 0.00 15.00 0.00 68.00 0.00 9.07 0.14 9.27 9.27 0.00 4.40 6.60
sde 0.00 0.00 6.00 0.00 32.00 0.00 10.67 0.04 5.83 5.83 0.00 4.17 2.50
sdf 0.00 0.00 21.00 0.00 172.00 0.00 16.38 0.34 16.19 16.19 0.00 3.62 7.60
sdg 0.00 0.00 12.00 0.00 128.00 0.00 21.33 0.14 11.58 11.58 0.00 3.50 4.20
sdh 0.00 0.00 8.00 0.00 104.00 0.00 26.00 0.14 17.62 17.62 0.00 5.50 4.40
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
4.21 0.00 2.73 7.85 0.00 85.21

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1.00 105.00 10.00 552.00 252.00 13.98 0.32 2.75 2.87 1.50 2.70 31.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 11.00 0.00 76.00 0.00 13.82 0.18 16.45 16.45 0.00 16.45 18.10
sdd 0.00 0.00 6.00 0.00 48.00 0.00 16.00 0.06 9.83 9.83 0.00 8.83 5.30
sde 0.00 0.00 33.00 0.00 164.00 0.00 9.94 0.24 7.06 7.06 0.00 7.21 23.80
sdf 0.00 0.00 8.00 0.00 48.00 0.00 12.00 0.10 12.50 12.50 0.00 12.50 10.00
sdg 0.00 0.00 9.00 0.00 68.00 0.00 15.11 0.11 12.67 12.67 0.00 12.67 11.40
sdh 0.00 0.00 43.00 0.00 220.00 0.00 10.23 0.28 6.58 6.58 0.00 6.05 26.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
5.67 0.00 2.50 10.34 0.00 81.48

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 62.00 59.00 649.00 849.00 24.76 0.34 2.83 2.90 2.75 2.69 32.50
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 5.00 57.00 24.00 1520.00 49.81 0.27 4.34 6.60 4.14 4.31 26.70
sdd 0.00 0.00 2.00 55.00 8.00 1548.00 54.60 0.43 7.51 83.50 4.75 5.53 31.50
sde 0.00 0.00 25.00 57.00 108.00 1544.00 40.29 0.38 4.72 6.24 4.05 4.45 36.50
sdf 0.00 0.00 7.00 55.00 36.00 1544.00 50.97 0.33 5.26 13.86 4.16 5.03 31.20
sdg 0.00 0.00 5.00 57.00 20.00 1524.00 49.81 0.68 11.00 56.20 7.04 6.69 41.50
sdh 0.00 0.00 24.00 57.00 100.00 1524.00 40.10 0.37 4.57 5.54 4.16 3.99 32.30
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
11.29 0.00 2.70 3.27 0.00 82.73

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 36.00 20.00 951.00 272.00 43.68 0.13 2.32 2.78 1.50 2.14 12.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 37.00 0.00 248.00 0.00 13.41 0.16 4.43 4.43 0.00 3.76 13.90
sdd 0.00 0.00 13.00 0.00 72.00 0.00 11.08 0.08 6.08 6.08 0.00 5.38 7.00
sde 0.00 0.00 8.00 0.00 44.00 0.00 11.00 0.07 8.38 8.38 0.00 8.38 6.70
sdf 0.00 0.00 39.00 0.00 280.00 0.00 14.36 0.23 5.85 5.85 0.00 5.21 20.30
sdg 0.00 0.00 25.00 0.00 192.00 0.00 15.36 0.12 4.72 4.72 0.00 4.16 10.40
sdh 0.00 0.00 20.00 0.00 160.00 0.00 16.00 0.10 5.15 5.15 0.00 4.55 9.10
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
10.15 0.00 1.43 1.01 0.00 87.42

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 7.00 2.00 123.00 12.00 30.00 0.02 2.44 3.00 0.50 2.33 2.10
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 7.00 0.00 92.00 0.00 26.29 0.04 5.71 5.71 0.00 5.00 3.50
sdd 0.00 0.00 6.00 0.00 68.00 0.00 22.67 0.03 4.33 4.33 0.00 3.83 2.30
sde 0.00 0.00 5.00 0.00 64.00 0.00 25.60 0.04 9.00 9.00 0.00 8.40 4.20
sdf 0.00 0.00 9.00 0.00 96.00 0.00 21.33 0.05 5.11 5.11 0.00 4.89 4.40
sdg 0.00 0.00 3.00 0.00 24.00 0.00 16.00 0.02 6.00 6.00 0.00 6.00 1.80
sdh 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.02 16.00 16.00 0.00 16.00 1.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
7.64 0.00 1.60 3.36 0.00 87.41

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 25.00 8.00 387.00 64.00 27.33 0.09 2.67 3.04 1.50 2.64 8.70
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 12.00 0.00 124.00 0.00 20.67 0.08 6.33 6.33 0.00 4.67 5.60
sdd 0.00 0.00 5.00 0.00 32.00 0.00 12.80 0.03 6.40 6.40 0.00 6.40 3.20
sde 0.00 0.00 11.00 0.00 72.00 0.00 13.09 0.11 10.27 10.27 0.00 10.27 11.30
sdf 1.00 0.00 16.00 0.00 176.00 0.00 22.00 0.11 7.19 7.19 0.00 6.56 10.50
sdg 1.00 0.00 12.00 0.00 144.00 0.00 24.00 0.07 5.58 5.58 0.00 5.33 6.40
sdh 0.00 0.00 11.00 0.00 120.00 0.00 21.82 0.09 7.91 7.91 0.00 7.00 7.70
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
3.15 0.00 2.74 9.70 0.00 84.41

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1.00 39.00 19.00 1178.00 696.00 64.62 0.07 1.17 0.87 1.79 0.69 4.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 13.00 0.00 112.00 0.00 17.23 0.06 4.31 4.31 0.00 3.62 4.70
sdd 0.00 0.00 9.00 0.00 72.00 0.00 16.00 0.04 5.00 5.00 0.00 4.00 3.60
sde 0.00 0.00 3.00 0.00 48.00 0.00 32.00 0.14 45.33 45.33 0.00 24.00 7.20
sdf 0.00 0.00 10.00 0.00 104.00 0.00 20.80 0.05 4.60 4.60 0.00 4.30 4.30
sdg 0.00 0.00 6.00 0.00 52.00 0.00 17.33 0.03 5.00 5.00 0.00 5.00 3.00
sdh 0.00 0.00 3.00 0.00 32.00 0.00 21.33 0.03 9.00 9.00 0.00 9.00 2.70
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
4.11 0.00 6.82 9.79 0.00 79.28

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 8.00 15.00 36.00 156.00 16.70 0.04 1.96 3.00 1.40 1.96 4.50
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 11.00 75.00 88.00 2460.00 59.26 0.43 5.01 6.91 4.73 4.64 39.90
sdd 0.00 0.00 6.00 80.00 56.00 2504.00 59.53 0.39 4.50 10.00 4.09 3.95 34.00
sde 0.00 0.00 3.00 77.00 36.00 2488.00 63.10 0.33 4.06 3.00 4.10 4.06 32.50
sdf 0.00 0.00 8.00 76.00 76.00 2488.00 61.05 0.34 4.08 5.75 3.91 4.08 34.30
sdg 0.00 0.00 3.00 76.00 36.00 2500.00 64.20 0.50 6.33 21.33 5.74 5.57 44.00
sdh 0.00 0.00 3.00 75.00 28.00 2460.00 63.79 0.32 4.05 3.00 4.09 4.01 31.30
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
8.26 0.00 13.71 8.73 0.00 69.30

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 10.00 25.00 141.00 132.00 15.60 0.07 2.00 3.00 1.60 2.00 7.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 27.00 47.00 132.00 656.00 21.30 0.37 5.03 6.44 4.21 4.88 36.10
sdd 0.00 0.00 14.00 46.00 72.00 668.00 24.67 0.35 5.82 7.71 5.24 4.95 29.70
sde 0.00 0.00 7.00 46.00 44.00 652.00 26.26 0.44 5.21 14.71 3.76 6.83 36.20
sdf 0.00 0.00 26.00 47.00 128.00 648.00 21.26 0.63 8.60 15.73 4.66 6.48 47.30
sdg 0.00 0.00 14.00 47.00 76.00 664.00 24.26 0.28 4.51 6.21 4.00 4.46 27.20
sdh 0.00 0.00 17.00 47.00 80.00 660.00 23.12 0.54 8.33 10.41 7.57 8.23 52.70
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
4.80 0.00 3.28 13.30 0.00 78.62

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 2.00 11.00 125.00 90.50 376.00 6.86 0.21 1.55 2.45 1.47 1.46 19.90
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 55.00 134.00 296.00 1440.00 18.37 1.16 6.08 10.91 4.10 3.86 72.90
sdd 0.00 0.00 16.00 125.00 104.00 1500.00 22.75 0.70 4.87 7.12 4.58 4.33 61.10
sde 0.00 0.00 15.00 125.00 100.00 1472.00 22.46 0.75 6.52 13.67 5.66 4.65 65.10
sdf 2.00 0.00 47.00 124.00 336.00 1472.00 21.15 1.11 6.42 12.68 4.04 4.39 75.10
sdg 2.00 0.00 24.00 143.00 196.00 1512.00 20.46 0.77 4.53 7.67 4.00 3.74 62.50
sdh 0.00 0.00 25.00 133.00 156.00 1444.00 20.25 1.11 6.97 21.44 4.26 4.08 64.50
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
2.92 0.00 2.59 8.94 0.00 85.55

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 1.00 2.00 6.50 0.00 4.33 0.00 1.00 3.00 0.00 1.00 0.30
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 19.00 92.00 84.00 984.00 19.24 0.56 5.14 6.95 4.77 4.56 50.60
sdd 0.00 0.00 5.00 100.00 28.00 1052.00 20.57 0.53 4.64 2.80 4.73 4.94 51.90
sde 0.00 0.00 2.00 93.00 12.00 1020.00 21.73 0.47 4.35 10.50 4.22 4.71 44.70
sdf 0.00 0.00 14.00 95.00 76.00 1020.00 20.11 0.59 5.56 6.79 5.38 5.02 54.70
sdg 0.00 0.00 8.00 100.00 44.00 1044.00 20.15 0.87 8.18 3.00 8.59 7.96 86.00
sdh 0.00 0.00 5.00 90.00 28.00 988.00 21.39 0.44 4.77 3.00 4.87 4.55 43.20
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
3.27 0.00 2.68 10.55 0.00 83.50

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 12.00 0.00 29.50 0.00 4.92 0.04 3.00 3.00 0.00 3.00 3.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 28.00 115.00 128.00 1328.00 20.36 0.77 5.41 9.32 4.46 4.78 68.40
sdd 0.00 0.00 5.00 118.00 20.00 1396.00 23.02 0.70 6.15 19.20 5.59 5.00 61.50
sde 0.00 0.00 3.00 118.00 12.00 1356.00 22.61 0.73 6.44 16.00 6.19 5.59 67.60
sdf 0.00 0.00 21.00 117.00 124.00 1360.00 21.51 0.73 5.32 9.67 4.54 4.67 64.40
sdg 0.00 0.00 10.00 113.00 80.00 1364.00 23.48 0.83 4.76 5.70 4.67 5.43 66.80
sdh 0.00 0.00 10.00 113.00 64.00 1336.00 22.76 0.59 4.84 5.50 4.78 4.56 56.10
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
2.84 0.00 2.75 11.26 0.00 83.15

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 18.00 3.00 216.00 8.00 21.33 0.06 2.76 3.06 1.00 2.76 5.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 24.00 100.00 112.00 1060.00 18.90 0.64 5.14 7.00 4.69 4.40 54.50
sdd 0.00 0.00 11.00 98.00 72.00 1060.00 20.77 0.79 7.22 17.09 6.11 5.71 62.20
sde 0.00 0.00 3.00 98.00 16.00 1056.00 21.23 0.57 5.67 25.67 5.06 5.28 53.30
sdf 0.00 0.00 29.00 98.00 160.00 1056.00 19.15 0.76 5.91 9.14 4.95 4.72 59.90
sdg 0.00 0.00 26.00 101.00 144.00 1080.00 19.28 0.97 9.65 16.31 7.93 5.67 72.00
sdh 0.00 0.00 11.00 100.00 68.00 1064.00 20.40 0.62 5.55 10.36 5.02 4.65 51.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
2.83 0.00 2.99 9.14 0.00 85.04

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 5.00 3.00 53.50 32.00 21.38 0.02 2.25 3.00 1.00 2.25 1.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 32.00 103.00 140.00 1256.00 20.68 0.61 4.51 5.34 4.25 3.92 52.90
sdd 0.00 0.00 14.00 103.00 92.00 1264.00 23.18 0.61 5.18 9.14 4.64 4.51 52.80
sde 0.00 0.00 19.00 104.00 388.00 1232.00 26.34 1.08 8.78 18.74 6.96 6.15 75.60
sdf 2.00 0.00 27.00 104.00 160.00 1232.00 21.25 0.98 7.50 16.30 5.22 5.58 73.10
sdg 3.00 0.00 24.00 103.00 220.00 1296.00 23.87 0.73 5.72 11.17 4.45 4.33 55.00
sdh 2.00 0.00 22.00 104.00 256.00 1260.00 24.06 0.83 6.25 14.64 4.48 4.14 52.20
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
5.39 0.00 3.07 12.12 0.00 79.42

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1.00 20.00 38.00 254.00 221.50 16.40 0.11 1.84 2.65 1.42 1.64 9.50
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 40.00 118.00 252.00 1660.00 24.20 0.98 6.19 10.65 4.68 4.19 66.20
sdd 0.00 0.00 12.00 115.00 84.00 1712.00 28.28 0.71 5.56 9.67 5.13 4.90 62.20
sde 0.00 0.00 13.00 116.00 80.00 1700.00 27.60 0.96 7.43 17.23 6.33 5.84 75.40
sdf 3.00 0.00 42.00 115.00 288.00 1700.00 25.32 1.01 6.40 5.26 6.82 5.39 84.60
sdg 2.00 0.00 22.00 118.00 192.00 1692.00 26.91 0.79 5.63 8.64 5.07 4.58 64.10
sdh 1.00 0.00 31.00 115.00 440.00 1668.00 28.88 1.12 7.82 16.29 5.53 4.97 72.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
5.15 0.00 2.99 16.27 0.00 75.60

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 33.00 17.00 559.50 97.00 26.26 0.13 2.68 2.97 2.12 2.60 13.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 22.00 69.00 204.00 724.00 20.40 1.54 16.97 40.23 9.55 7.60 69.20
sdd 0.00 0.00 12.00 71.00 80.00 724.00 19.37 0.67 8.10 20.42 6.01 6.13 50.90
sde 0.00 0.00 17.00 71.00 116.00 704.00 18.64 0.59 6.70 11.18 5.63 5.43 47.80
sdf 1.00 0.00 25.00 71.00 240.00 704.00 19.67 0.79 8.22 16.40 5.34 5.08 48.80
sdg 1.00 0.00 25.00 69.00 220.00 740.00 20.43 0.74 7.87 14.64 5.42 5.07 47.70
sdh 1.00 0.00 19.00 69.00 188.00 724.00 20.73 0.63 7.44 11.84 6.23 5.40 47.50
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

CPU BOGOMIPS: 57600.72
REGEX/SECOND: 1882571
HD SIZE: 10876.31 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 11.18
 
To me it doesn't look like your slog device is used at all. I assume that the system was completely idle while doing these benchmarks right?
Using sync writes and a slog device should show steady I/O on slog device and bursts on main drives.
 
To me it doesn't look like your slog device is used at all. I assume that the system was completely idle while doing these benchmarks right?
Using sync writes and a slog device should show steady I/O on slog device and bursts on main drives.

No the system isn't idle. I have VM's running, but as you can see they don't use much IO
Do you have any idea why the slog device isn't used? Strange thing is that I added the SLOG with:
"scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part1" and now it shows up as:
"ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1"
 
this is the expected performance. if you want to use zfs for VM storage, add a SSD for cache (ZIL and L2ARC) and you will get MUCH better experience:

http://pve.proxmox.com/wiki/ZFS#Add_Cache_and_Log_to_existing_pool

For best SSD reliablity, I can recommend one of these:

Intel SSD DC S3710 Series 200GB, 2.5", SATA 6Gb/s (SSDSC2BA200G401)
Intel SSD DC S3700 Series 200GB, 2.5", SATA 6Gb/s (SSDSC2BA200G3)


Here are pveperf results from one of our testlab hosts (raidz2 with 6 x 4 TB Seagate SATA and one Intel SSD DC S3710 Series 200GB for cache)

Code:
root@proxmox:~# zpool status
  pool: space
 state: ONLINE
  scan: scrub repaired 0 in 2h31m with 0 errors on Sun Mar 22 09:18:29 2015
config:

        NAME                                                 STATE     READ WRITE CKSUM
        space                                                ONLINE       0     0     0
          raidz2-0                                           ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z3M15Q               ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z3N1H3               ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z1S6VN               ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z1RBLP               ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z1PEP7               ONLINE       0     0     0
            scsi-SATA_ST4000NM0033-9Z_Z1Z3LT1C               ONLINE       0     0     0
        logs
          scsi-SATA_INTEL_SSDSC2BA2BTTV439401WX200GGN-part1  ONLINE       0     0     0
        cache
          scsi-SATA_INTEL_SSDSC2BA2BTTV439401WX200GGN-part2  ONLINE       0     0     0


errors: No known data errors

root@proxmox:~# pveperf /space/
CPU BOGOMIPS:      57595.44
REGEX/SECOND:      1804289
HD SIZE:           14601.65 GB (space)
FSYNCS/SECOND:     2999.24
DNS EXT:           35.32 ms
DNS INT:           0.52 ms (proxmox.com)

How exactly do you partition the SSD? How do you determine the size of each partition should be?
 
Seems to be working. My FSYNCS were still pretty slow though even with the SSD cache/logs drive at around 400.

Actual performance seems to be okay, busy restoring a backup from a USB 2 external drive. Will only know when I start working with the VMs.

While copying from external:

Code:
root@proxmox0:~# zpool iostat -v
                                                     capacity     operations    bandwidth
pool                                              alloc   free   read  write   read  write
------------------------------------------------  -----  -----  -----  -----  -----  -----
zfs0                                               285G  3.35T      0  5.12K    408  29.7M
  raidz2                                           285G  3.35T      0  5.12K     56  29.7M
    scsi-SATA_ST1000VX000-1CU_Z1D8RSG0                -      -      0    219    503  26.3M
    scsi-SATA_ST1000VX000-1CU_Z1D8RRRT                -      -      0    219    327  26.4M
    scsi-SATA_ST1000VX000-1CU_Z1D8SBR9                -      -      0    220    359  26.4M
    scsi-SATA_ST1000DM003-9YN_S1D1V9LC                -      -      0    218    340  26.4M
logs                                                  -      -      -      -      -      -
  scsi-SATA_Samsung_SSD_850S21TNSAG223774B-part1     4K  18.5G      0      0    352  2.78K
cache                                                 -      -      -      -      -      -
  scsi-SATA_Samsung_SSD_850S21TNSAG223774B-part2  8.04G  66.5G      0     26    117  2.95M
------------------------------------------------  -----  -----  -----  -----  -----  -----
 
The most dangerous aspect of sync=disabled is that the client guest can issue a write barrier if it is doing metadata-related things on that FS, and won't continue until the write is ACK'ed. But here we are lying and saying 'we wrote it!'. It's entirely possible to cause filesystem corruption (for the guest OS, not ZFS...)
 
  • Like
Reactions: chrone
I think FSYNCS measurement are just wrong with zfs .

zfs 'raid-10' shows a lot lower then 3ware raid-10 system. like 15-20 times less .

However for our main business application the speed is close to the same.

Currently we use zfs / drbd / lvm with one drbd per kvm. Our goal is high data availability with users having zero keyboard lags and zero unscheduled downtime.

I figure there will be improvements to pve/zfs in the future , just like with ceph and other storage.

Our mission critical app uses openvz. I find it is more reliable then kvm for that system. Our main app runs old debian etch. most other systems use jessie and have no kvm issues.
 
If you have a raid controller with writeback cache, this is not surprising.
 
Would you mind elaborating a bit?
Do you mean that HW raid is supposed to be faster or that the test is skewed?

If you are measuring fsync's then having a cache on your RAID controller (assuming it is battery backed up and gets actually used safely) is going to make it faster. Having an SSD SLOG device on your ZFS is the way to speed up that operation in ZFS.
 
  • Like
Reactions: Ovidiu

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!