[SOLVED] Langsame writes/reads mit ZFS raidz1

svh79 · Aug 16, 2020

Hallo Forum,

ich habe ein neues raidz1 mit drei USB 3.0 Festplatten (PMR) eingerichtet und erreiche nur äußerst magere Lese-/Schreibwerte. Ich hätte hierbei mit mindestens dem 10-fachen Durchsatz gerechnet, da die Festplatten im Einzelbetrieb die erwarteten Durchsätze liefern.

Mein System ist ein Intel i5-10210U mit 16 GB RAM und aktuellster Proxmox Version. Weitere Details habe ich beigefügt.

Wo könnte die Ursache liegen, wie kann ich sie finden und beseitigen? Könnt ihr mir Tipps geben?

Viele Grüße,
Sven

Bash:

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/3p, 5000M
            |__ Port 1: Dev 7, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 3: Dev 5, If 0, Class=Hub, Driver=hub/3p, 5000M
            |__ Port 1: Dev 8, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/3p, 5000M
            |__ Port 1: Dev 9, If 0, Class=Mass Storage, Driver=uas, 5000M

Bash:

# zpool create -f -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa tank raidz /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV66N1J /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV67M4L /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV6A7M4

Bash:

# zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///root/zfs_key tank/vault

Bash:

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  27.3T  4.85T  22.4T        -         -     0%    17%  1.00x    ONLINE  -

Bash:

# zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

    NAME                                   STATE     READ WRITE CKSUM
    tank                                   ONLINE       0     0     0
      raidz1-0                             ONLINE       0     0     0
        ata-ST10000DM0004-2GR11L_ZJV66N1J  ONLINE       0     0     0
        ata-ST10000DM0004-2GR11L_ZJV67M4L  ONLINE       0     0     0
        ata-ST10000DM0004-2GR11L_ZJV6A7M4  ONLINE       0     0     0

errors: No known data errors

Bash:

# zpool iostat
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        4.85T  22.4T      1     99   260K  21.8M

Bash:

# lsblk -o NAME,PHY-SeC
NAME                          PHY-SEC
sdb                              4096
├─sdb1                           4096
└─sdb9                           4096
sdc                              4096
├─sdc1                           4096
└─sdc9                           4096
sdd                              4096
├─sdd1                           4096
└─sdd9                           4096

Bash:

# fdisk -l
Disk /dev/sdb: 9.1 TiB, 10000831348224 bytes, 19532873727 sectors
Disk model: Backup+ Hub BK  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 90E25874-6DC0-DA4F-AF16-B3AF391979F6

Device           Start         End     Sectors  Size Type
/dev/sdb1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sdb9  19532855296 19532871679       16384    8M Solaris reserved 1


Disk /dev/sdc: 9.1 TiB, 10000831348224 bytes, 19532873727 sectors
Disk model: Backup+ Hub BK  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: FDA6F74E-67F6-4040-93E8-F8924C797794

Device           Start         End     Sectors  Size Type
/dev/sdc1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sdc9  19532855296 19532871679       16384    8M Solaris reserved 1


Disk /dev/sdd: 9.1 TiB, 10000831348224 bytes, 19532873727 sectors
Disk model: Backup+ Hub BK  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7EEA4374-861F-C14E-8306-04B91E49EB69

Device           Start         End     Sectors  Size Type
/dev/sdd1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sdd9  19532855296 19532871679       16384    8M Solaris reserved 1

H4R0 · Aug 16, 2020

zfs iostat ist nicht wirklich aussagend, mal mal ein richtigen Benchmark mit fio und poste die Ergebnisse.

Dein setup ist alles andere als gut, usb bandbreite ist zwischen allen platten geteilt, hinzu kommt du hast extremen overhead bei zfs checksums, parity berechnung, authentifizierte 256bit verschlüsselung (gcm), komprimierung... Das ganze bei einer 15 Watt TDP Notebook CPU.

Hast du mal nebenbei CPU, RAM, Temperatur, CPU Clockspeed, etc. gemonitort ?!

svh79 · Aug 16, 2020

Die Temperatur schient mir in Ordnung zu sein (~55 °C).

Ist der USB-Hub wirklich limitierend? Ich ging davon aus, dass dieser am 3.1 Gen2 Anschluss maximal 900 MB/s schaffen sollte. Ich wäre ja bereits mit 150 MB/s zufrieden. Testweise habe ich die drei HDDs mal jeweils direkt an den PC angeschlossen - die Geschwindigkeit bleibt unverändert.

Bash:

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/tank/vault/random_rw.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][99.9%][r=4864KiB/s,w=1680KiB/s][r=1216,w=420 IOPS][eta 00m:05s]  
test: (groupid=0, jobs=1): err= 0: pid=577203: Sun Aug 16 18:20:03 2020
  read: IOPS=93, BW=376KiB/s (385kB/s)(3070MiB/8363015msec)
   bw (  KiB/s): min=    7, max= 6280, per=100.00%, avg=599.25, stdev=704.57, samples=10488
   iops        : min=    1, max= 1570, avg=149.76, stdev=176.15, samples=10488
  write: IOPS=31, BW=126KiB/s (129kB/s)(1026MiB/8363015msec); 0 zone resets
   bw (  KiB/s): min=    7, max= 2048, per=100.00%, avg=203.33, stdev=238.26, samples=10328
   iops        : min=    1, max=  512, avg=50.77, stdev=59.57, samples=10328
  cpu          : usr=0.02%, sys=0.55%, ctx=140476, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=376KiB/s (385kB/s), 376KiB/s-376KiB/s (385kB/s-385kB/s), io=3070MiB (3219MB), run=8363015-8363015msec
  WRITE: bw=126KiB/s (129kB/s), 126KiB/s-126KiB/s (129kB/s-129kB/s), io=1026MiB (1076MB), run=8363015-8363015msec

Bash:

# sensors
Package id 0:  +58.0°C  (high = +100.0°C, crit = +100.0°C)
Core 0:        +58.0°C  (high = +100.0°C, crit = +100.0°C)
Core 1:        +54.0°C  (high = +100.0°C, crit = +100.0°C)
Core 2:        +55.0°C  (high = +100.0°C, crit = +100.0°C)
Core 3:        +56.0°C  (high = +100.0°C, crit = +100.0°C)

Bash:

# lshw -c cpu
  *-cpu                    
       description: CPU
       product: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
       vendor: Intel Corp.
       physical id: 45
       bus info: cpu@0
       version: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
       serial: To Be Filled By O.E.M.
       slot: U3E1
       size: 3899MHz
       capacity: 4200MHz
       width: 64 bits
       clock: 100MHz
       capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities cpufreq
       configuration: cores=4 enabledcores=4 threads=8

UdoB · Aug 16, 2020

Hallo,

Blechplatten bringen nun einmal nicht mehr als 93+31=120 IOPS. Das sind nur 8ms pro Zugriff, wie sollte das schneller gehen???

Wenn es nennenswert mehr sync-writes werden sollen, könnte eine (kleine!) SSD als SLOG helfen.

Just my 2€¢

H4R0 · Aug 18, 2020

Ein mixed readwrite test bringt dir nicht viel bei HDD's, mach mal ein sequentiellen write test:

Code:

fio --name=seqwrite --filename=
/tank/vault/fio_seqwrite.fio --refill_buffers --rw=write --direct=1 --loops=5 --ioengine=libaio --bs=1m --size=1G --runtime=60 --group_reporting

Und anschließend teste mal die verschiedenen sektor größen durch.

Code:

fio --loops=5 --size=1G --filename=
/tank/vault/benchmark.fio --stonewall --ioengine=libaio --direct=1 \
  --name=Seqread --bs=1m --rw=read \
  --name=Seqwrite --bs=1m --rw=write \
  --name=512Kread --bs=512k --rw=randread \
  --name=512Kwrite --bs=512k --rw=randwrite \
  --name=4kQD32read --bs=4k --iodepth=32 --rw=randread \
  --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite

Um herauszufinden wo dein bottleneck ist, während der tests nebenbei "atop 1" und "watch -n 0.2 sensors" laufen lassen, dabei dann load, cpu auslastung, clock speed, festplatten auslastung und temperatur notieren.

raidz hat einiges an overhead und skaliert nicht gut, bei dir wäre ein raid1 mirror besser oder bei 4 festplatten ein stripped vdev mirror, dabei hast du wesentlich mehr performance. 2x schreib und 4x lese rate ohne parity overhead, allerdings weniger speicher insgesamt.

Hast du ZFS arc begrenzt oder darf der standardmäßig seine 50% ram nehmen ?

Hast du zufällig deduplication aktiviert ?

svh79 · Aug 19, 2020

Die Deduplication ist deaktiviert. RAM-Settings habe ich nicht angepasst, d.h. es ist alles auf Standardwerte gestellt.

Während der fio Runs war die Temperatur recht unauffällig (~60 °C) und der Lüfter sprang nur selten an. Es lief "atop 1" nebenher, jedoch fiel mir auch dort nichts Besonderes auf. Die z_wr_* Prozesse lagen stets bei < 2 %. Die drei raidz1 HDDs sind derzeit sd[cde].

Bash:

# atop 1
PRC | sys    0.15s |  user   0.22s | #proc    548 |  #trun      4 | #tslpi   742 |  #tslpu     2 | #zombie    0 |  clones    24 |              |  #exit     23 |
CPU | sys       6% |  user     12% | irq       2% |  idle    659% | wait    127% |  guest     2% | ipc     0.89 |  cycl  182MHz | curf 3.89GHz |  curscal  92% |
CPL | avg1    3.57 |  avg5    4.58 | avg15   4.90 |               |              |  csw    10753 | intr    5253 |               |              |  numcpu     8 |
MEM | tot    15.4G |  free    3.5G | cache   3.1G |  buff    1.0G | slab  752.5M |  shmem 469.6M | shrss 400.4M |  vmbal   0.0M | hptot   0.0M |  hpuse   0.0M |
SWP | tot     8.0G |  free    7.7G |              |               |              |               |              |               | vmcom   9.9G |  vmlim  15.7G |
DSK |          sdc |  busy     17% | read       2 |  write     37 | KiB/r      0 |  KiB/w     37 | MBr/s    0.0 |  MBw/s    1.3 | avq    21.43 |  avio 4.51 ms |
DSK |          sdd |  busy     17% | read       5 |  write     32 | KiB/r      0 |  KiB/w     62 | MBr/s    0.0 |  MBw/s    2.0 | avq    14.21 |  avio 4.54 ms |
DSK |          sde |  busy     19% | read       2 |  write     38 | KiB/r      0 |  KiB/w     44 | MBr/s    0.0 |  MBw/s    1.6 | avq    17.04 |  avio 4.70 ms |

Bash:

# fio --name=seqwrite --filename=/tank/vault/fio_seqwrite.fio --refill_buffers --rw=write --direct=1 --loops=5 --ioengine=libaio --bs=1m --size=1G --runtime=60 --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
seqwrite: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=4096KiB/s][w=4 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=166247: Tue Aug 18 21:29:47 2020
  write: IOPS=2, BW=2875KiB/s (2944kB/s)(169MiB/60191msec); 0 zone resets
    slat (msec): min=205, max=790, avg=355.98, stdev=115.41
    clat (nsec): min=2268, max=16801, avg=3069.31, stdev=1834.66
     lat (msec): min=205, max=790, avg=355.99, stdev=115.40
    clat percentiles (nsec):
     |  1.00th=[ 2288],  5.00th=[ 2416], 10.00th=[ 2480], 20.00th=[ 2544],
     | 30.00th=[ 2608], 40.00th=[ 2672], 50.00th=[ 2800], 60.00th=[ 2864],
     | 70.00th=[ 2960], 80.00th=[ 3056], 90.00th=[ 3280], 95.00th=[ 3536],
     | 99.00th=[16320], 99.50th=[16768], 99.90th=[16768], 99.95th=[16768],
     | 99.99th=[16768]
   bw (  KiB/s): min= 2043, max= 6144, per=100.00%, avg=2940.20, stdev=1054.62, samples=117
   iops        : min=    1, max=    6, avg= 2.79, stdev= 1.06, samples=117
  lat (usec)   : 4=97.63%, 10=0.59%, 20=1.78%
  cpu          : usr=0.07%, sys=0.07%, ctx=345, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,169,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2875KiB/s (2944kB/s), 2875KiB/s-2875KiB/s (2944kB/s-2944kB/s), io=169MiB (177MB), run=60191-60191msec

svh79 · Aug 19, 2020

Der zweite fio Lauf dauerte einige Stunden und ich habe ihn bei einer eta von 11 Stunden abgebrochen. Ich hoffe, dass die Ergebnisse dennoch aussagekräftig sind. Ansonsten lasse ich es nochmal - und dann bis zum Ende - durchlaufen.

Bash:

# fio --loops=5 --size=1G --filename=/tank/vault/benchmark.fio --stonewall --ioengine=libaio --direct=1 \
  --name=Seqread --bs=1m --rw=read \
  --name=Seqwrite --bs=1m --rw=write \
  --name=512Kread --bs=512k --rw=randread \
  --name=512Kwrite --bs=512k --rw=randwrite \
  --name=4kQD32read --bs=4k --iodepth=32 --rw=randread \
  --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite
  
Seqread: (groupid=0, jobs=1): err= 0: pid=170784: Wed Aug 19 06:53:51 2020
  read: IOPS=5316, BW=5317MiB/s (5575MB/s)(5120MiB/963msec)
    slat (usec): min=138, max=1479, avg=186.70, stdev=38.43
    clat (nsec): min=276, max=18507, avg=508.78, stdev=756.57
     lat (usec): min=138, max=1482, avg=187.44, stdev=38.62
    clat percentiles (nsec):
     |  1.00th=[  306],  5.00th=[  330], 10.00th=[  346], 20.00th=[  370],
     | 30.00th=[  386], 40.00th=[  406], 50.00th=[  430], 60.00th=[  458],
     | 70.00th=[  494], 80.00th=[  556], 90.00th=[  652], 95.00th=[  772],
     | 99.00th=[ 1080], 99.50th=[ 1240], 99.90th=[13632], 99.95th=[18048],
     | 99.99th=[18560]
   bw (  MiB/s): min= 5312, max= 5312, per=99.93%, avg=5312.99, stdev= 0.00, samples=1
   iops        : min= 5312, max= 5312, avg=5312.00, stdev= 0.00, samples=1
  lat (nsec)   : 500=71.43%, 750=22.99%, 1000=4.24%
  lat (usec)   : 2=1.09%, 20=0.25%
  cpu          : usr=0.31%, sys=99.58%, ctx=12, majf=0, minf=265
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seqwrite: (groupid=1, jobs=1): err= 0: pid=170785: Wed Aug 19 06:53:51 2020
  write: IOPS=2, BW=2829KiB/s (2897kB/s)(5120MiB/1853516msec); 0 zone resets
    slat (msec): min=202, max=887, avg=362.01, stdev=118.28
    clat (nsec): min=2058, max=30663, avg=3438.61, stdev=2645.13
     lat (msec): min=202, max=887, avg=362.01, stdev=118.28
    clat percentiles (nsec):
     |  1.00th=[ 2384],  5.00th=[ 2576], 10.00th=[ 2640], 20.00th=[ 2736],
     | 30.00th=[ 2800], 40.00th=[ 2896], 50.00th=[ 2960], 60.00th=[ 3056],
     | 70.00th=[ 3184], 80.00th=[ 3280], 90.00th=[ 3440], 95.00th=[ 3728],
     | 99.00th=[19328], 99.50th=[21376], 99.90th=[22144], 99.95th=[23680],
     | 99.99th=[30592]
   bw (  KiB/s): min= 1488, max= 4704, per=75.99%, avg=2148.96, stdev=811.97, samples=3641
   iops        : min=    1, max=    4, avg= 1.57, stdev= 0.81, samples=3641
  lat (usec)   : 4=96.35%, 10=0.90%, 20=1.76%, 50=1.00%
  cpu          : usr=0.01%, sys=0.11%, ctx=10292, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kread: (groupid=2, jobs=1): err= 0: pid=208467: Wed Aug 19 06:53:51 2020
  read: IOPS=11.7k, BW=5845MiB/s (6129MB/s)(5120MiB/876msec)
    slat (usec): min=35, max=303, avg=84.40, stdev=18.09
    clat (nsec): min=279, max=33148, avg=445.96, stdev=941.14
     lat (usec): min=36, max=304, avg=85.00, stdev=18.23
    clat percentiles (nsec):
     |  1.00th=[  298],  5.00th=[  306], 10.00th=[  314], 20.00th=[  326],
     | 30.00th=[  338], 40.00th=[  342], 50.00th=[  354], 60.00th=[  370],
     | 70.00th=[  398], 80.00th=[  458], 90.00th=[  556], 95.00th=[  636],
     | 99.00th=[  860], 99.50th=[ 1032], 99.90th=[18048], 99.95th=[19584],
     | 99.99th=[29056]
   bw (  MiB/s): min= 5877, max= 5877, per=100.00%, avg=5877.52, stdev= 0.00, samples=1
   iops        : min=11755, max=11755, avg=11755.00, stdev= 0.00, samples=1
  lat (nsec)   : 500=84.47%, 750=13.53%, 1000=1.46%
  lat (usec)   : 2=0.24%, 10=0.02%, 20=0.22%, 50=0.05%
  cpu          : usr=4.00%, sys=95.77%, ctx=26, majf=0, minf=136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kwrite: (groupid=3, jobs=1): err= 0: pid=208484: Wed Aug 19 06:53:51 2020
  write: IOPS=4, BW=2427KiB/s (2486kB/s)(5120MiB/2159879msec); 0 zone resets
    slat (msec): min=109, max=1202, avg=210.92, stdev=108.42
    clat (nsec): min=1826, max=31825, avg=3373.04, stdev=2461.67
     lat (msec): min=109, max=1202, avg=210.92, stdev=108.42
    clat percentiles (nsec):
     |  1.00th=[ 2320],  5.00th=[ 2544], 10.00th=[ 2608], 20.00th=[ 2704],
     | 30.00th=[ 2768], 40.00th=[ 2896], 50.00th=[ 2992], 60.00th=[ 3088],
     | 70.00th=[ 3152], 80.00th=[ 3248], 90.00th=[ 3440], 95.00th=[ 3664],
     | 99.00th=[18560], 99.50th=[21120], 99.90th=[21888], 99.95th=[22400],
     | 99.99th=[27008]
   bw (  KiB/s): min=  738, max= 3141, per=74.69%, avg=1812.64, stdev=851.76, samples=4301
   iops        : min=    1, max=    6, avg= 2.99, stdev= 1.68, samples=4301
  lat (usec)   : 2=0.02%, 4=96.80%, 10=0.71%, 20=1.76%, 50=0.71%
  cpu          : usr=0.01%, sys=0.11%, ctx=20514, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQD32read: (groupid=4, jobs=1): err= 0: pid=259787: Wed Aug 19 06:53:51 2020
  read: IOPS=68.3k, BW=267MiB/s (280MB/s)(5120MiB/19203msec)
    slat (usec): min=2, max=277, avg=13.85, stdev= 7.85
    clat (nsec): min=787, max=1059.2k, avg=454426.96, stdev=86395.23
     lat (usec): min=15, max=1253, avg=468.36, stdev=88.60
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  273], 10.00th=[  359], 20.00th=[  412],
     | 30.00th=[  433], 40.00th=[  449], 50.00th=[  465], 60.00th=[  482],
     | 70.00th=[  498], 80.00th=[  515], 90.00th=[  545], 95.00th=[  570],
     | 99.00th=[  627], 99.50th=[  652], 99.90th=[  717], 99.95th=[  742],
     | 99.99th=[  824]
   bw (  KiB/s): min=172257, max=256614, per=68.79%, avg=187826.68, stdev=23256.36, samples=37
   iops        : min=43064, max=64153, avg=46956.38, stdev=5814.07, samples=37
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=4.21%
  lat (usec)   : 500=68.03%, 750=27.70%, 1000=0.04%
  lat (msec)   : 2=0.01%
  cpu          : usr=5.83%, sys=94.03%, ctx=308, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQD32write: (groupid=5, jobs=1): err= 0: pid=260157: Wed Aug 19 06:53:51 2020
  write: IOPS=18, BW=72.0KiB/s (74.7kB/s)(2098MiB/29426268msec); 0 zone resets
    slat (msec): min=21, max=1228, avg=54.79, stdev=75.14
    clat (usec): min=5, max=4741.5k, avg=1698734.78, stdev=1099287.08
     lat (msec): min=24, max=4796, avg=1753.53, stdev=1110.37
    clat percentiles (msec):
     |  1.00th=[  776],  5.00th=[  776], 10.00th=[  776], 20.00th=[  802],
     | 30.00th=[  818], 40.00th=[  852], 50.00th=[  911], 60.00th=[ 1552],
     | 70.00th=[ 2433], 80.00th=[ 3239], 90.00th=[ 3440], 95.00th=[ 3540],
     | 99.00th=[ 3708], 99.50th=[ 3809], 99.90th=[ 4077], 99.95th=[ 4178],
     | 99.99th=[ 4597]
   bw (  KiB/s): min=    7, max=  264, per=100.00%, avg=72.99, stdev=61.29, samples=58775
   iops        : min=    1, max=   66, avg=18.18, stdev=15.32, samples=58775
  lat (usec)   : 10=0.01%, 20=0.01%
  lat (msec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 1000=53.68%
  cpu          : usr=0.02%, sys=0.21%, ctx=1082970, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,536967,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=5317MiB/s (5575MB/s), 5317MiB/s-5317MiB/s (5575MB/s-5575MB/s), io=5120MiB (5369MB), run=963-963msec

Run status group 1 (all jobs):
  WRITE: bw=2829KiB/s (2897kB/s), 2829KiB/s-2829KiB/s (2897kB/s-2897kB/s), io=5120MiB (5369MB), run=1853516-1853516msec

Run status group 2 (all jobs):
   READ: bw=5845MiB/s (6129MB/s), 5845MiB/s-5845MiB/s (6129MB/s-6129MB/s), io=5120MiB (5369MB), run=876-876msec

Run status group 3 (all jobs):
  WRITE: bw=2427KiB/s (2486kB/s), 2427KiB/s-2427KiB/s (2486kB/s-2486kB/s), io=5120MiB (5369MB), run=2159879-2159879msec

Run status group 4 (all jobs):
   READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=5120MiB (5369MB), run=19203-19203msec

Run status group 5 (all jobs):
  WRITE: bw=72.0KiB/s (74.7kB/s), 72.0KiB/s-72.0KiB/s (74.7kB/s-74.7kB/s), io=2098MiB (2199MB), run=29426268-29426268msec

H4R0 · Aug 20, 2020

Hmm irgendwas stimmt da nicht. Die Festplatten sollten definitiv schneller sein.

Kannst du den Pool mal ohne Komprimierung, Verschlüsselung und ACL anlegen, besteht das Problem dann immer noch ?

Wenn nicht, eins nach dem anderen wieder aktivieren und testen. Ansonsten mal ein mirror mit nur 2 Festplatten testen.

Irgendwas scheint Probleme zu machen.

svh79 · Aug 20, 2020

Ich habe den Pool neu angelegt - ohne Verschlüsselung; siehe da, auf einmal geht's flott.

Allerdings wundert mich dies: bei aktivierter Verschlüsselung ist die CPU nicht nennenswert ausgelastet, dennoch ist die Leistung schlecht. Gibt's nach Anlegen des verschlüsselten ZFS noch weitere Settings zu beachten?

H4R0 · Aug 20, 2020

Verschlüsselung ist bei Zol afaik noch in der Beta Phase. Du könntest mal CCM statt GCM testen.

Aber die Verschlüsselung bringt dir nicht viel solange der key auf einer unverschlüsselten Festplatte liegt..

Wie performt der sequentielle write test denn jetzt ?

svh79 · Aug 20, 2020

...und nochmal den Pool angelegt:

Bash:

# zpool create -f -o ashift=12 -O acltype=posixacl -O dnodesize=auto -O encryption=aes-256-gcm -O keylocation=file:///root/zfs_key -O keyformat=raw -O compression=lz4 -O xattr=sa tank raidz /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV66N1J /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV67M4L /dev/disk/by-id/ata-ST10000DM0004-2GR11L_ZJV6A7M4
# zfs create tank/vault
# fio --name=seqwrite --filename=/tank/vault/fio_seqwrite.fio --refill_buffers --rw=write --direct=1 --loops=5 --ioengine=libaio --bs=1m --size=1G --runtime=60 --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
seqwrite: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=509MiB/s][w=508 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=2477385: Thu Aug 20 14:28:53 2020
  write: IOPS=556, BW=557MiB/s (584MB/s)(5120MiB/9197msec); 0 zone resets
    slat (usec): min=132, max=3695, avg=1652.83, stdev=629.51
    clat (nsec): min=288, max=29922, avg=770.77, stdev=1174.75
     lat (usec): min=133, max=3701, avg=1653.90, stdev=629.63
    clat percentiles (nsec):
     |  1.00th=[  346],  5.00th=[  394], 10.00th=[  422], 20.00th=[  462],
     | 30.00th=[  498], 40.00th=[  548], 50.00th=[  612], 60.00th=[  676],
     | 70.00th=[  748], 80.00th=[  844], 90.00th=[ 1048], 95.00th=[ 1368],
     | 99.00th=[ 2608], 99.50th=[ 3088], 99.90th=[19328], 99.95th=[21632],
     | 99.99th=[29824]
   bw (  KiB/s): min=454656, max=1869824, per=100.00%, avg=570710.89, stdev=325111.48, samples=18
   iops        : min=  444, max= 1826, avg=557.28, stdev=317.50, samples=18
  lat (nsec)   : 500=31.04%, 750=38.93%, 1000=18.77%
  lat (usec)   : 2=8.91%, 4=1.89%, 10=0.08%, 20=0.31%, 50=0.08%
  cpu          : usr=8.97%, sys=7.01%, ctx=35198, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=557MiB/s (584MB/s), 557MiB/s-557MiB/s (584MB/s-584MB/s), io=5120MiB (5369MB), run=9197-9197msec

svh79 · Aug 20, 2020

...und noch der größere Benchmark:

Bash:

Seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seqwrite: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
512Kread: (g=2): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
512Kwrite: (g=3): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
4kQD32read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
4kQD32write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 6 processes
Seqread: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(5),w(1)][100.0%][w=776MiB/s][w=199k IOPS][eta 00m:00s]                         
Seqread: (groupid=0, jobs=1): err= 0: pid=2225212: Thu Aug 20 13:17:18 2020
  read: IOPS=3158, BW=3159MiB/s (3312MB/s)(5120MiB/1621msec)
    slat (usec): min=136, max=40095, avg=314.70, stdev=696.63
    clat (nsec): min=312, max=29990, avg=741.83, stdev=959.53
     lat (usec): min=136, max=40099, avg=315.75, stdev=696.98
    clat percentiles (nsec):
     |  1.00th=[  322],  5.00th=[  354], 10.00th=[  374], 20.00th=[  414],
     | 30.00th=[  450], 40.00th=[  490], 50.00th=[  532], 60.00th=[  604],
     | 70.00th=[  700], 80.00th=[  884], 90.00th=[ 1224], 95.00th=[ 1624],
     | 99.00th=[ 2800], 99.50th=[ 2960], 99.90th=[17792], 99.95th=[18304],
     | 99.99th=[30080]
   bw (  MiB/s): min= 1152, max= 5551, per=90.85%, avg=2869.65, stdev=2352.57, samples=3
   iops        : min= 1152, max= 5551, avg=2869.33, stdev=2352.71, samples=3
  lat (nsec)   : 500=42.54%, 750=31.21%, 1000=10.39%
  lat (usec)   : 2=12.23%, 4=3.36%, 10=0.02%, 20=0.23%, 50=0.02%
  cpu          : usr=0.43%, sys=94.38%, ctx=452, majf=0, minf=265
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seqwrite: (groupid=1, jobs=1): err= 0: pid=2225223: Thu Aug 20 13:17:18 2020
  write: IOPS=5187, BW=5187MiB/s (5439MB/s)(5120MiB/987msec); 0 zone resets
    slat (usec): min=80, max=7311, avg=189.99, stdev=305.88
    clat (nsec): min=268, max=304566, avg=734.69, stdev=4317.54
     lat (usec): min=80, max=7315, avg=191.02, stdev=306.40
    clat percentiles (nsec):
     |  1.00th=[   286],  5.00th=[   294], 10.00th=[   302], 20.00th=[   322],
     | 30.00th=[   338], 40.00th=[   370], 50.00th=[   446], 60.00th=[   548],
     | 70.00th=[   660], 80.00th=[   836], 90.00th=[  1320], 95.00th=[  1880],
     | 99.00th=[  2800], 99.50th=[  3280], 99.90th=[ 13120], 99.95th=[ 18048],
     | 99.99th=[305152]
   bw (  MiB/s): min= 2560, max= 2560, per=49.36%, avg=2560.63, stdev= 0.00, samples=1
   iops        : min= 2560, max= 2560, avg=2560.00, stdev= 0.00, samples=1
  lat (nsec)   : 500=55.16%, 750=20.57%, 1000=8.71%
  lat (usec)   : 2=11.52%, 4=3.83%, 10=0.04%, 20=0.16%, 500=0.02%
  cpu          : usr=5.98%, sys=67.75%, ctx=1629, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kread: (groupid=2, jobs=1): err= 0: pid=2225441: Thu Aug 20 13:17:18 2020
  read: IOPS=26.7k, BW=13.1GiB/s (14.0GB/s)(5120MiB/383msec)
    slat (usec): min=31, max=246, avg=36.46, stdev= 6.65
    clat (nsec): min=276, max=22176, avg=377.00, stdev=620.49
     lat (usec): min=31, max=248, avg=36.94, stdev= 6.73
    clat percentiles (nsec):
     |  1.00th=[  290],  5.00th=[  298], 10.00th=[  306], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  342], 60.00th=[  350],
     | 70.00th=[  362], 80.00th=[  382], 90.00th=[  414], 95.00th=[  466],
     | 99.00th=[  596], 99.50th=[  676], 99.90th=[13120], 99.95th=[18304],
     | 99.99th=[19840]
  lat (nsec)   : 500=96.95%, 750=2.71%, 1000=0.17%
  lat (usec)   : 2=0.04%, 20=0.13%, 50=0.01%
  cpu          : usr=1.83%, sys=97.38%, ctx=10, majf=0, minf=136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kwrite: (groupid=3, jobs=1): err= 0: pid=2225564: Thu Aug 20 13:17:18 2020
  write: IOPS=20.2k, BW=9.84GiB/s (10.6GB/s)(5120MiB/508msec); 0 zone resets
    slat (usec): min=41, max=156, avg=48.66, stdev= 7.15
    clat (nsec): min=270, max=19603, avg=369.43, stdev=574.75
     lat (usec): min=42, max=158, avg=49.14, stdev= 7.24
    clat percentiles (nsec):
     |  1.00th=[  278],  5.00th=[  290], 10.00th=[  294], 20.00th=[  306],
     | 30.00th=[  314], 40.00th=[  322], 50.00th=[  330], 60.00th=[  338],
     | 70.00th=[  350], 80.00th=[  366], 90.00th=[  398], 95.00th=[  482],
     | 99.00th=[  732], 99.50th=[  844], 99.90th=[12992], 99.95th=[18048],
     | 99.99th=[18560]
  lat (nsec)   : 500=95.52%, 750=3.67%, 1000=0.58%
  lat (usec)   : 2=0.10%, 10=0.01%, 20=0.13%
  cpu          : usr=11.64%, sys=88.17%, ctx=6, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQD32read: (groupid=4, jobs=1): err= 0: pid=2225645: Thu Aug 20 13:17:18 2020
  read: IOPS=338k, BW=1319MiB/s (1383MB/s)(5120MiB/3883msec)
    slat (nsec): min=1563, max=195638, avg=2455.02, stdev=1536.97
    clat (nsec): min=672, max=462461, avg=92043.42, stdev=14680.71
     lat (usec): min=2, max=465, avg=94.55, stdev=14.96
    clat percentiles (usec):
     |  1.00th=[   79],  5.00th=[   82], 10.00th=[   83], 20.00th=[   85],
     | 30.00th=[   86], 40.00th=[   86], 50.00th=[   87], 60.00th=[   88],
     | 70.00th=[   90], 80.00th=[  100], 90.00th=[  111], 95.00th=[  122],
     | 99.00th=[  145], 99.50th=[  155], 99.90th=[  206], 99.95th=[  239],
     | 99.99th=[  285]
   bw (  MiB/s): min= 1126, max= 1299, per=88.20%, avg=1162.98, stdev=61.73, samples=7
   iops        : min=288435, max=332666, avg=297722.71, stdev=15802.37, samples=7
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=80.77%, 250=19.19%, 500=0.04%
  cpu          : usr=15.95%, sys=83.98%, ctx=105, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQD32write: (groupid=5, jobs=1): err= 0: pid=2226061: Thu Aug 20 13:17:18 2020
  write: IOPS=197k, BW=768MiB/s (805MB/s)(5120MiB/6668msec); 0 zone resets
    slat (usec): min=3, max=293, avg= 4.50, stdev= 2.33
    clat (nsec): min=717, max=736073, avg=157904.98, stdev=23327.69
     lat (usec): min=4, max=741, avg=162.47, stdev=23.77
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  143], 10.00th=[  143], 20.00th=[  145],
     | 30.00th=[  147], 40.00th=[  147], 50.00th=[  149], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  169], 90.00th=[  184], 95.00th=[  200],
     | 99.00th=[  241], 99.50th=[  277], 99.90th=[  379], 99.95th=[  416],
     | 99.99th=[  465]
   bw (  KiB/s): min=771472, max=807192, per=99.95%, avg=785880.00, stdev=8991.44, samples=13
   iops        : min=192868, max=201798, avg=196470.00, stdev=2247.86, samples=13
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=99.19%
  lat (usec)   : 500=0.80%, 750=0.01%
  cpu          : usr=12.22%, sys=87.69%, ctx=84, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3159MiB/s (3312MB/s), 3159MiB/s-3159MiB/s (3312MB/s-3312MB/s), io=5120MiB (5369MB), run=1621-1621msec

Run status group 1 (all jobs):
  WRITE: bw=5187MiB/s (5439MB/s), 5187MiB/s-5187MiB/s (5439MB/s-5439MB/s), io=5120MiB (5369MB), run=987-987msec

Run status group 2 (all jobs):
   READ: bw=13.1GiB/s (14.0GB/s), 13.1GiB/s-13.1GiB/s (14.0GB/s-14.0GB/s), io=5120MiB (5369MB), run=383-383msec

Run status group 3 (all jobs):
  WRITE: bw=9.84GiB/s (10.6GB/s), 9.84GiB/s-9.84GiB/s (10.6GB/s-10.6GB/s), io=5120MiB (5369MB), run=508-508msec

Run status group 4 (all jobs):
   READ: bw=1319MiB/s (1383MB/s), 1319MiB/s-1319MiB/s (1383MB/s-1383MB/s), io=5120MiB (5369MB), run=3883-3883msec

Run status group 5 (all jobs):
  WRITE: bw=768MiB/s (805MB/s), 768MiB/s-768MiB/s (805MB/s-805MB/s), io=5120MiB (5369MB), run=6668-6668msec

svh79 · Aug 20, 2020

Ich mag die Zahlen falsch lesen, aber die Raten empfand ich als zu hoch. Daher habe ich nochmal mit 2G (statt 1G) getestet:

Bash:

Jobs: 1 (f=1): [_(5),w(1)][100.0%][w=12.5MiB/s][w=3194 IOPS][eta 00m:03s]          
Seqread: (groupid=0, jobs=1): err= 0: pid=65242: Thu Aug 20 21:43:53 2020
  read: IOPS=3708, BW=3709MiB/s (3889MB/s)(10.0GiB/2761msec)
    slat (usec): min=136, max=988, avg=268.06, stdev=204.58
    clat (nsec): min=276, max=31044, avg=623.03, stdev=863.24
     lat (usec): min=136, max=991, avg=268.97, stdev=205.00
    clat percentiles (nsec):
     |  1.00th=[  302],  5.00th=[  330], 10.00th=[  346], 20.00th=[  370],
     | 30.00th=[  394], 40.00th=[  422], 50.00th=[  458], 60.00th=[  516],
     | 70.00th=[  628], 80.00th=[  764], 90.00th=[ 1004], 95.00th=[ 1224],
     | 99.00th=[ 1848], 99.50th=[ 2448], 99.90th=[18048], 99.95th=[18560],
     | 99.99th=[19072]
   bw (  MiB/s): min= 1406, max= 5570, per=89.02%, avg=3301.68, stdev=2076.47, samples=5
   iops        : min= 1406, max= 5570, avg=3301.00, stdev=2076.34, samples=5
  lat (nsec)   : 500=57.89%, 750=21.17%, 1000=10.77%
  lat (usec)   : 2=9.41%, 4=0.50%, 20=0.24%, 50=0.01%
  cpu          : usr=0.69%, sys=99.06%, ctx=106, majf=0, minf=264
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seqwrite: (groupid=1, jobs=1): err= 0: pid=65293: Thu Aug 20 21:43:53 2020
  write: IOPS=20, BW=20.0MiB/s (21.0MB/s)(10.0GiB/488680msec); 0 zone resets
    slat (usec): min=94, max=183867k, avg=47718.48, stdev=2309406.96
    clat (nsec): min=300, max=4155.3k, avg=1718.13, stdev=41112.90
     lat (usec): min=94, max=183867k, avg=47720.81, stdev=2309407.02
    clat percentiles (nsec):
     |  1.00th=[   358],  5.00th=[   434], 10.00th=[   474], 20.00th=[   532],
     | 30.00th=[   596], 40.00th=[   692], 50.00th=[   860], 60.00th=[  1192],
     | 70.00th=[  1656], 80.00th=[  2224], 90.00th=[  2672], 95.00th=[  2960],
     | 99.00th=[  3440], 99.50th=[  3664], 99.90th=[ 14912], 99.95th=[ 16064],
     | 99.99th=[201728]
   bw (  KiB/s): min= 6059, max=2974547, per=100.00%, avg=342040.58, stdev=640271.64, samples=45
   iops        : min=    5, max= 2904, avg=333.51, stdev=625.19, samples=45
  lat (nsec)   : 500=13.93%, 750=30.15%, 1000=10.94%
  lat (usec)   : 2=21.20%, 4=23.48%, 10=0.17%, 20=0.11%, 50=0.02%
  lat (usec)   : 250=0.01%
  lat (msec)   : 10=0.01%
  cpu          : usr=0.05%, sys=0.50%, ctx=40813, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kread: (groupid=2, jobs=1): err= 0: pid=99466: Thu Aug 20 21:43:53 2020
  read: IOPS=355, BW=178MiB/s (186MB/s)(10.0GiB/57670msec)
    slat (usec): min=31, max=55359k, avg=2814.49, stdev=386828.60
    clat (nsec): min=261, max=18399, avg=462.86, stdev=527.10
     lat (usec): min=31, max=55359k, avg=2815.14, stdev=386828.63
    clat percentiles (nsec):
     |  1.00th=[  286],  5.00th=[  294], 10.00th=[  302], 20.00th=[  318],
     | 30.00th=[  330], 40.00th=[  342], 50.00th=[  354], 60.00th=[  374],
     | 70.00th=[  418], 80.00th=[  506], 90.00th=[  684], 95.00th=[  932],
     | 99.00th=[ 1640], 99.50th=[ 2288], 99.90th=[ 7392], 99.95th=[13248],
     | 99.99th=[18048]
   bw (  MiB/s): min=  492, max= 5245, per=100.00%, avg=1885.75, stdev=1914.42, samples=5
   iops        : min=  984, max=10490, avg=3770.80, stdev=3829.10, samples=5
  lat (nsec)   : 500=79.56%, 750=12.24%, 1000=3.96%
  lat (usec)   : 2=3.58%, 4=0.52%, 10=0.04%, 20=0.10%
  cpu          : usr=0.05%, sys=3.68%, ctx=1314, majf=0, minf=135
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kwrite: (groupid=3, jobs=1): err= 0: pid=105784: Thu Aug 20 21:43:53 2020
  write: IOPS=34, BW=17.1MiB/s (17.9MB/s)(10.0GiB/599765msec); 0 zone resets
    slat (usec): min=44, max=157584k, avg=29280.18, stdev=1667238.03
    clat (nsec): min=286, max=127774, avg=956.68, stdev=1855.11
     lat (usec): min=44, max=157584k, avg=29281.54, stdev=1667238.09
    clat percentiles (nsec):
     |  1.00th=[  326],  5.00th=[  354], 10.00th=[  382], 20.00th=[  430],
     | 30.00th=[  466], 40.00th=[  516], 50.00th=[  604], 60.00th=[  740],
     | 70.00th=[  972], 80.00th=[ 1352], 90.00th=[ 2064], 95.00th=[ 2544],
     | 99.00th=[ 3152], 99.50th=[ 3440], 99.90th=[ 9024], 99.95th=[16064],
     | 99.99th=[96768]
   bw (  KiB/s): min= 8359, max=1712432, per=100.00%, avg=316647.71, stdev=359506.80, samples=45
   iops        : min=   16, max= 3344, avg=617.98, stdev=702.14, samples=45
  lat (nsec)   : 500=36.66%, 750=23.96%, 1000=10.27%
  lat (usec)   : 2=18.33%, 4=10.49%, 10=0.20%, 20=0.06%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.05%, sys=0.41%, ctx=46200, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQD32read: (groupid=4, jobs=1): err= 0: pid=143108: Thu Aug 20 21:43:53 2020
  read: IOPS=24.5k, BW=95.9MiB/s (101MB/s)(10.0GiB/106823msec)
    slat (nsec): min=1642, max=68940M, avg=39948.74, stdev=42579301.73
    clat (nsec): min=689, max=68956M, avg=1263491.81, stdev=237095371.78
     lat (usec): min=2, max=68956k, avg=1303.53, stdev=240890.04
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  277], 10.00th=[  322], 20.00th=[  367],
     | 30.00th=[  400], 40.00th=[  424], 50.00th=[  449], 60.00th=[  469],
     | 70.00th=[  490], 80.00th=[  515], 90.00th=[  553], 95.00th=[  586],
     | 99.00th=[  840], 99.50th=[ 1090], 99.90th=[ 1958], 99.95th=[ 2147],
     | 99.99th=[ 2507]
   bw (  KiB/s): min= 4716, max=347145, per=100.00%, avg=203234.36, stdev=47984.11, samples=77
   iops        : min= 1179, max=86786, avg=50808.25, stdev=11995.98, samples=77
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.04%
  lat (usec)   : 250=3.56%, 500=70.21%, 750=24.81%, 1000=0.78%
  lat (msec)   : 2=0.51%, 4=0.09%, 10=0.01%, 20=0.01%
  cpu          : usr=2.15%, sys=33.07%, ctx=3462, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQD32write: (groupid=5, jobs=1): err= 0: pid=153476: Thu Aug 20 21:43:53 2020
  write: IOPS=292, BW=1172KiB/s (1200kB/s)(10.0GiB/8949918msec); 0 zone resets
    slat (usec): min=3, max=96855k, avg=3412.47, stdev=437605.79
    clat (nsec): min=847, max=96881M, avg=105838101.53, stdev=2434773317.49
     lat (usec): min=207, max=96882k, avg=109250.78, stdev=2473674.63
    clat percentiles (usec):
     |  1.00th=[     652],  5.00th=[     906], 10.00th=[    1074],
     | 20.00th=[    1369], 30.00th=[    1926], 40.00th=[    5080],
     | 50.00th=[    7504], 60.00th=[    9765], 70.00th=[   11863],
     | 80.00th=[   14222], 90.00th=[   16581], 95.00th=[   17957],
     | 99.00th=[   19530], 99.50th=[   20841], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=  128, max=133992, per=100.00%, avg=14035.04, stdev=13322.78, samples=1493
   iops        : min=   32, max=33498, avg=3508.73, stdev=3330.69, samples=1493
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 250=0.01%, 500=0.33%, 750=1.66%, 1000=5.60%
  lat (msec)   : 2=22.93%, 4=4.41%, 10=26.48%, 20=37.89%, 50=0.53%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.05%, sys=1.03%, ctx=1704633, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3709MiB/s (3889MB/s), 3709MiB/s-3709MiB/s (3889MB/s-3889MB/s), io=10.0GiB (10.7GB), run=2761-2761msec

Run status group 1 (all jobs):
  WRITE: bw=20.0MiB/s (21.0MB/s), 20.0MiB/s-20.0MiB/s (21.0MB/s-21.0MB/s), io=10.0GiB (10.7GB), run=488680-488680msec

Run status group 2 (all jobs):
   READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=10.0GiB (10.7GB), run=57670-57670msec

Run status group 3 (all jobs):
  WRITE: bw=17.1MiB/s (17.9MB/s), 17.1MiB/s-17.1MiB/s (17.9MB/s-17.9MB/s), io=10.0GiB (10.7GB), run=599765-599765msec

Run status group 4 (all jobs):
   READ: bw=95.9MiB/s (101MB/s), 95.9MiB/s-95.9MiB/s (101MB/s-101MB/s), io=10.0GiB (10.7GB), run=106823-106823msec

Run status group 5 (all jobs):
  WRITE: bw=1172KiB/s (1200kB/s), 1172KiB/s-1172KiB/s (1200kB/s-1200kB/s), io=10.0GiB (10.7GB), run=8949918-8949918msec

svh79 · Aug 22, 2020

Ich habe den Fehler gefunden: die externen Festplatten wurden zu heiß. Ich habe sie umgestellt und für ausreichend Kühlung gesorgt. Nun habe ich Schreibraten von knapp 300 MB/s (inkl. Verschlüsselung), was für das Setup und meine Ansprüche absolut genügt.

Vielen Dank für eure Unterstützung auf dem Weg zur Lösung! Manchmal kann es so simpel sein...

Search

Search

[SOLVED] Langsame writes/reads mit ZFS raidz1

svh79

Active Member

H4R0

Well-Known Member

svh79

Active Member

UdoB

Distinguished Member

H4R0

Well-Known Member

svh79

Active Member

svh79

Active Member

H4R0

Well-Known Member

svh79

Active Member

H4R0

Well-Known Member

svh79

Active Member

svh79

Active Member

svh79

Active Member

svh79

Active Member