how to best benchmark SSDs?

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=124MiB/s][w=31 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=650: Mon Aug  9 02:13:27 2021
  write: IOPS=32, BW=131MiB/s (137MB/s)(10.0GiB/78205msec); 0 zone resets
    clat (usec): min=18905, max=52034, avg=28911.67, stdev=5175.16
     lat (usec): min=18909, max=52037, avg=28915.30, stdev=5175.30
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[20579], 10.00th=[21103], 20.00th=[22414],
     | 30.00th=[26346], 40.00th=[29230], 50.00th=[30278], 60.00th=[31065],
     | 70.00th=[32113], 80.00th=[33162], 90.00th=[34866], 95.00th=[35914],
     | 99.00th=[38536], 99.50th=[41157], 99.90th=[50070], 99.95th=[50594],
     | 99.99th=[52167]
   bw (  KiB/s): min=114688, max=180224, per=99.99%, avg=134060.50, stdev=19052.65, samples=156
   iops        : min=   28, max=   44, avg=32.71, stdev= 4.66, samples=156
  lat (msec)   : 20=1.25%, 50=98.67%, 100=0.08%
  cpu          : usr=5.54%, sys=1.43%, ctx=5847, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=10.0GiB (10.7GB), run=78205-78205msec

Disk stats (read/write):
  sda: ios=0/15374, merge=0/5306, ticks=0/112221, in_queue=115512, util=96.85%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 10281 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=464MiB/s][r=116 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=858: Mon Aug  9 02:28:50 2021
  read: IOPS=116, BW=464MiB/s (487MB/s)(10.0GiB/22065msec)
    clat (usec): min=7151, max=14648, avg=8607.74, stdev=974.10
     lat (usec): min=7153, max=14650, avg=8609.57, stdev=974.13
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 7767], 10.00th=[ 7832], 20.00th=[ 7963],
     | 30.00th=[ 8094], 40.00th=[ 8225], 50.00th=[ 8356], 60.00th=[ 8455],
     | 70.00th=[ 8586], 80.00th=[ 8979], 90.00th=[ 9765], 95.00th=[10552],
     | 99.00th=[12780], 99.50th=[13566], 99.90th=[14353], 99.95th=[14615],
     | 99.99th=[14615]
   bw (  KiB/s): min=442368, max=499712, per=100.00%, avg=475322.18, stdev=10223.97, samples=44
   iops        : min=  108, max=  122, avg=116.05, stdev= 2.50, samples=44
  lat (msec)   : 10=91.64%, 20=8.36%
  cpu          : usr=0.19%, sys=2.92%, ctx=2563, majf=0, minf=523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=464MiB/s (487MB/s), 464MiB/s-464MiB/s (487MB/s-487MB/s), io=10.0GiB (10.7GB), run=22065-22065msec

Disk stats (read/write):
  sda: ios=10160/4, merge=0/6, ticks=74911/2, in_queue=73256, util=99.15%
0 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10259 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10259 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Script ran 1903 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...30400 MiB / 32192 MiB / 30491 MiB / 32 MiB / 17 MiB ( 30400 MiB / 32192 MiB / 30491 MiB / 32 MiB / 17 MiB )
2 min passed...42976 MiB / 45536 MiB / 42967 MiB / 32 MiB / 19 MiB ( 12576 MiB / 13344 MiB / 12476 MiB / 0 MiB / 2 MiB )
3 min passed...42976 MiB / 45568 MiB / 42980 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
4 min passed...43008 MiB / 45824 MiB / 42992 MiB / 32 MiB / 19 MiB ( 32 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...43008 MiB / 46048 MiB / 43005 MiB / 32 MiB / 19 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 0 MiB )
6 min passed...43008 MiB / 46048 MiB / 43016 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
7 min passed...43040 MiB / 46048 MiB / 43030 MiB / 32 MiB / 19 MiB ( 32 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
8 min passed...43040 MiB / 46080 MiB / 43039 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
9 min passed...43072 MiB / 46432 MiB / 43050 MiB / 32 MiB / 19 MiB ( 32 MiB / 352 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...43072 MiB / 46688 MiB / 43063 MiB / 32 MiB / 19 MiB ( 0 MiB / 256 MiB / 13 MiB / 0 MiB / 0 MiB )
11 min passed...43072 MiB / 46688 MiB / 43073 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
12 min passed...43072 MiB / 46720 MiB / 43084 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
13 min passed...43104 MiB / 46912 MiB / 43093 MiB / 32 MiB / 19 MiB ( 32 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...43104 MiB / 47040 MiB / 43108 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 15 MiB / 0 MiB / 0 MiB )
15 min passed...43136 MiB / 47296 MiB / 43118 MiB / 32 MiB / 19 MiB ( 32 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
16 min passed...43136 MiB / 47328 MiB / 43129 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
17 min passed...43136 MiB / 47392 MiB / 43138 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 64 MiB / 9 MiB / 12480 MiB / 12467 MiB )
18 min passed...43136 MiB / 47456 MiB / 43149 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
19 min passed...43168 MiB / 47488 MiB / 43157 MiB / 12512 MiB / 12486 MiB ( 32 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
20 min passed...43168 MiB / 47808 MiB / 43166 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 320 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...43168 MiB / 47808 MiB / 43176 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...43200 MiB / 48032 MiB / 43188 MiB / 12512 MiB / 12486 MiB ( 32 MiB / 224 MiB / 12 MiB / 0 MiB / 0 MiB )
23 min passed...43200 MiB / 48128 MiB / 43196 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 96 MiB / 8 MiB / 0 MiB / 0 MiB )
24 min passed...43200 MiB / 48128 MiB / 43208 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
25 min passed...43232 MiB / 48384 MiB / 43217 MiB / 12512 MiB / 12486 MiB ( 32 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...43232 MiB / 48544 MiB / 43227 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...43232 MiB / 48736 MiB / 43236 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
28 min passed...43264 MiB / 48768 MiB / 43244 MiB / 12512 MiB / 12486 MiB ( 32 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
29 min passed...43264 MiB / 48768 MiB / 43254 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...43264 MiB / 48992 MiB / 43265 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
31 min passed...43264 MiB / 49216 MiB / 43275 MiB / 12512 MiB / 12486 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=111MiB/s][w=28.5k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=649: Mon Aug  9 03:01:48 2021
  write: IOPS=31.9k, BW=125MiB/s (131MB/s)(1024MiB/8205msec); 0 zone resets
    slat (usec): min=6, max=4624, avg=22.36, stdev=20.42
    clat (usec): min=31, max=8612, avg=1973.61, stdev=380.16
     lat (usec): min=40, max=8674, avg=1997.28, stdev=383.88
    clat percentiles (usec):
     |  1.00th=[ 1598],  5.00th=[ 1663], 10.00th=[ 1696], 20.00th=[ 1745],
     | 30.00th=[ 1778], 40.00th=[ 1811], 50.00th=[ 1844], 60.00th=[ 1893],
     | 70.00th=[ 1975], 80.00th=[ 2147], 90.00th=[ 2474], 95.00th=[ 2704],
     | 99.00th=[ 3294], 99.50th=[ 3654], 99.90th=[ 5145], 99.95th=[ 6194],
     | 99.99th=[ 8356]
   bw (  KiB/s): min=100328, max=144592, per=100.00%, avg=128012.38, stdev=12673.66, samples=16
   iops        : min=25082, max=36148, avg=32003.06, stdev=3168.40, samples=16
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=71.55%, 4=28.21%, 10=0.23%
  cpu          : usr=18.17%, sys=56.66%, ctx=270929, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=1024MiB (1074MB), run=8205-8205msec

Disk stats (read/write):
  sda: ios=0/260240, merge=0/1238, ticks=0/14872, in_queue=16876, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 1029 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1036 MiB / 19 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1036 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=300MiB/s][r=76.9k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=889: Mon Aug  9 03:16:52 2021
  read: IOPS=75.6k, BW=295MiB/s (310MB/s)(1024MiB/3467msec)
    slat (usec): min=3, max=234, avg= 6.74, stdev= 3.46
    clat (usec): min=176, max=7158, avg=834.96, stdev=105.29
     lat (usec): min=187, max=7169, avg=842.83, stdev=105.60
    clat percentiles (usec):
     |  1.00th=[  742],  5.00th=[  766], 10.00th=[  775], 20.00th=[  791],
     | 30.00th=[  807], 40.00th=[  816], 50.00th=[  824], 60.00th=[  840],
     | 70.00th=[  848], 80.00th=[  865], 90.00th=[  881], 95.00th=[  906],
     | 99.00th=[ 1057], 99.50th=[ 1205], 99.90th=[ 2008], 99.95th=[ 2835],
     | 99.99th=[ 5080]
   bw (  KiB/s): min=298976, max=310552, per=100.00%, avg=303241.33, stdev=4436.16, samples=6
   iops        : min=74744, max=77638, avg=75810.33, stdev=1109.04, samples=6
  lat (usec)   : 250=0.01%, 500=0.01%, 750=1.95%, 1000=96.86%
  lat (msec)   : 2=1.08%, 4=0.08%, 10=0.02%
  cpu          : usr=43.88%, sys=55.86%, ctx=60, majf=0, minf=72
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=295MiB/s (310MB/s), 295MiB/s-295MiB/s (310MB/s-310MB/s), io=1024MiB (1074MB), run=3467-3467msec

Disk stats (read/write):
  sda: ios=256996/3, merge=0/0, ticks=52474/0, in_queue=52932, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1814 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3648 MiB / 4032 MiB / 3646 MiB / 0 MiB / 21 MiB ( 3648 MiB / 4032 MiB / 3646 MiB / 0 MiB / 21 MiB )
2 min passed...3648 MiB / 4256 MiB / 3659 MiB / 0 MiB / 21 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 0 MiB )
3 min passed...3680 MiB / 4256 MiB / 3669 MiB / 0 MiB / 21 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
4 min passed...3680 MiB / 4416 MiB / 3681 MiB / 0 MiB / 21 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...3680 MiB / 4512 MiB / 3692 MiB / 0 MiB / 21 MiB ( 0 MiB / 96 MiB / 11 MiB / 0 MiB / 0 MiB )
6 min passed...3680 MiB / 4672 MiB / 3704 MiB / 0 MiB / 21 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
7 min passed...3712 MiB / 4896 MiB / 3714 MiB / 0 MiB / 21 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
8 min passed...3712 MiB / 4928 MiB / 3726 MiB / 0 MiB / 21 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
9 min passed...3744 MiB / 5056 MiB / 3737 MiB / 0 MiB / 21 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...3744 MiB / 5088 MiB / 3749 MiB / 0 MiB / 21 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
11 min passed...3744 MiB / 5184 MiB / 3759 MiB / 0 MiB / 21 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
12 min passed...3776 MiB / 5440 MiB / 3769 MiB / 0 MiB / 21 MiB ( 32 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...3776 MiB / 5536 MiB / 3782 MiB / 0 MiB / 21 MiB ( 0 MiB / 96 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...3776 MiB / 5568 MiB / 3795 MiB / 0 MiB / 21 MiB ( 0 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
15 min passed...3808 MiB / 5568 MiB / 3806 MiB / 0 MiB / 21 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...3808 MiB / 5632 MiB / 3818 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 64 MiB / 12 MiB / 1024 MiB / 1024 MiB )
17 min passed...3808 MiB / 5920 MiB / 3829 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
18 min passed...3840 MiB / 6112 MiB / 3839 MiB / 1024 MiB / 1045 MiB ( 32 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...3840 MiB / 6144 MiB / 3850 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
20 min passed...3840 MiB / 6144 MiB / 3860 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...3872 MiB / 6144 MiB / 3871 MiB / 1024 MiB / 1045 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
22 min passed...3872 MiB / 6592 MiB / 3884 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 448 MiB / 13 MiB / 0 MiB / 0 MiB )
23 min passed...3872 MiB / 6784 MiB / 3891 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 192 MiB / 7 MiB / 0 MiB / 0 MiB )
24 min passed...3904 MiB / 6784 MiB / 3904 MiB / 1024 MiB / 1045 MiB ( 32 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...3904 MiB / 6816 MiB / 3915 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
26 min passed...3936 MiB / 6848 MiB / 3925 MiB / 1024 MiB / 1045 MiB ( 32 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...3936 MiB / 7200 MiB / 3934 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 352 MiB / 9 MiB / 0 MiB / 0 MiB )
28 min passed...3936 MiB / 7264 MiB / 3946 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 64 MiB / 12 MiB / 0 MiB / 0 MiB )
29 min passed...3936 MiB / 7296 MiB / 3957 MiB / 1024 MiB / 1045 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...3968 MiB / 7296 MiB / 3967 MiB / 1024 MiB / 1045 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=634: Mon Aug  9 03:37:34 2021
  write: IOPS=28.8k, BW=450MiB/s (472MB/s)(1024MiB/2275msec); 0 zone resets
    slat (usec): min=7, max=2375, avg=20.35, stdev=15.26
    clat (usec): min=88, max=7836, avg=2186.71, stdev=428.00
     lat (usec): min=97, max=7938, avg=2208.46, stdev=432.03
    clat percentiles (usec):
     |  1.00th=[ 1729],  5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1926],
     | 30.00th=[ 1975], 40.00th=[ 2040], 50.00th=[ 2114], 60.00th=[ 2180],
     | 70.00th=[ 2245], 80.00th=[ 2343], 90.00th=[ 2474], 95.00th=[ 2868],
     | 99.00th=[ 4146], 99.50th=[ 4424], 99.90th=[ 6390], 99.95th=[ 6783],
     | 99.99th=[ 7570]
   bw (  KiB/s): min=430400, max=509376, per=100.00%, avg=465600.00, stdev=35644.88, samples=4
   iops        : min=26900, max=31836, avg=29100.00, stdev=2227.80, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=34.54%, 4=63.99%, 10=1.42%
  cpu          : usr=26.69%, sys=51.54%, ctx=65476, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=450MiB/s (472MB/s), 450MiB/s-450MiB/s (472MB/s-472MB/s), io=1024MiB (1074MB), run=2275-2275msec

Disk stats (read/write):
  sda: ios=0/61662, merge=0/180, ticks=0/4747, in_queue=6372, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1025 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=847: Mon Aug  9 03:52:37 2021
  read: IOPS=27.5k, BW=429MiB/s (450MB/s)(1024MiB/2387msec)
    slat (usec): min=4, max=682, avg=20.40, stdev=33.00
    clat (usec): min=488, max=9383, avg=2298.54, stdev=669.21
     lat (usec): min=496, max=9391, avg=2321.32, stdev=676.19
    clat percentiles (usec):
     |  1.00th=[  848],  5.00th=[  979], 10.00th=[ 1221], 20.00th=[ 1516],
     | 30.00th=[ 1942], 40.00th=[ 2507], 50.00th=[ 2606], 60.00th=[ 2671],
     | 70.00th=[ 2737], 80.00th=[ 2802], 90.00th=[ 2900], 95.00th=[ 2999],
     | 99.00th=[ 3326], 99.50th=[ 3458], 99.90th=[ 3687], 99.95th=[ 3818],
     | 99.99th=[ 3982]
   bw (  KiB/s): min=378688, max=603744, per=100.00%, avg=447168.00, stdev=105400.28, samples=4
   iops        : min=23668, max=37734, avg=27948.00, stdev=6587.52, samples=4
  lat (usec)   : 500=0.01%, 750=0.30%, 1000=5.25%
  lat (msec)   : 2=25.00%, 4=69.44%, 10=0.01%
  cpu          : usr=37.09%, sys=59.56%, ctx=867, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=429MiB/s (450MB/s), 429MiB/s-429MiB/s (450MB/s-450MB/s), io=1024MiB (1074MB), run=2387-2387msec

Disk stats (read/write):
  sda: ios=59356/0, merge=0/0, ticks=34527/0, in_queue=34516, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1043 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1031 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1807 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4160 MiB / 4512 MiB / 4155 MiB / 32 MiB / 39 MiB ( 4160 MiB / 4512 MiB / 4155 MiB / 32 MiB / 39 MiB )
2 min passed...4160 MiB / 4736 MiB / 4166 MiB / 32 MiB / 39 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
3 min passed...4160 MiB / 4736 MiB / 4176 MiB / 32 MiB / 39 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
4 min passed...4192 MiB / 4896 MiB / 4187 MiB / 32 MiB / 39 MiB ( 32 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...4192 MiB / 5056 MiB / 4197 MiB / 32 MiB / 39 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...4192 MiB / 5184 MiB / 4207 MiB / 32 MiB / 39 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
7 min passed...4224 MiB / 5376 MiB / 4217 MiB / 32 MiB / 39 MiB ( 32 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
8 min passed...4224 MiB / 5408 MiB / 4228 MiB / 32 MiB / 39 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
9 min passed...4224 MiB / 5696 MiB / 4237 MiB / 32 MiB / 39 MiB ( 0 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
10 min passed...4256 MiB / 5728 MiB / 4247 MiB / 32 MiB / 39 MiB ( 32 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...4256 MiB / 5888 MiB / 4258 MiB / 32 MiB / 39 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
12 min passed...4256 MiB / 6112 MiB / 4268 MiB / 32 MiB / 39 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...4256 MiB / 6176 MiB / 4277 MiB / 32 MiB / 39 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...4288 MiB / 6176 MiB / 4287 MiB / 32 MiB / 39 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
15 min passed...4288 MiB / 6208 MiB / 4298 MiB / 32 MiB / 39 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...4320 MiB / 6368 MiB / 4310 MiB / 1056 MiB / 1042 MiB ( 32 MiB / 160 MiB / 12 MiB / 1024 MiB / 1003 MiB )
17 min passed...4320 MiB / 6560 MiB / 4319 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
18 min passed...4320 MiB / 6848 MiB / 4329 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...4320 MiB / 6848 MiB / 4339 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
20 min passed...4352 MiB / 6848 MiB / 4348 MiB / 1056 MiB / 1042 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...4352 MiB / 6976 MiB / 4356 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 128 MiB / 8 MiB / 0 MiB / 0 MiB )
22 min passed...4352 MiB / 7328 MiB / 4367 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 352 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...4384 MiB / 7456 MiB / 4377 MiB / 1056 MiB / 1042 MiB ( 32 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...4384 MiB / 7456 MiB / 4385 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
25 min passed...4384 MiB / 7456 MiB / 4396 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
26 min passed...4384 MiB / 7616 MiB / 4405 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
27 min passed...4416 MiB / 8064 MiB / 4414 MiB / 1056 MiB / 1042 MiB ( 32 MiB / 448 MiB / 9 MiB / 0 MiB / 0 MiB )
28 min passed...4416 MiB / 8096 MiB / 4425 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
29 min passed...4416 MiB / 8096 MiB / 4434 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
30 min passed...4448 MiB / 8128 MiB / 4444 MiB / 1056 MiB / 1042 MiB ( 32 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 6: async 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][91.1%][eta 00m:04s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=660: Mon Aug  9 04:16:43 2021
  write: IOPS=61, BW=247MiB/s (259MB/s)(10.0GiB/41380msec); 0 zone resets
    slat (usec): min=91, max=101879, avg=269.01, stdev=2079.79
    clat (msec): min=28, max=6908, avg=1031.66, stdev=1929.35
     lat (msec): min=29, max=6908, avg=1031.94, stdev=1929.43
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   74], 10.00th=[   81], 20.00th=[   94],
     | 30.00th=[  109], 40.00th=[  127], 50.00th=[  153], 60.00th=[  186],
     | 70.00th=[  230], 80.00th=[  575], 90.00th=[ 5336], 95.00th=[ 5604],
     | 99.00th=[ 6678], 99.50th=[ 6812], 99.90th=[ 6879], 99.95th=[ 6879],
     | 99.99th=[ 6879]
   bw (  KiB/s): min= 8192, max=2056192, per=100.00%, avg=842438.17, stdev=649101.10, samples=23
   iops        : min=    2, max=  502, avg=205.52, stdev=158.53, samples=23
  lat (msec)   : 50=0.20%, 100=24.57%, 250=46.80%, 500=7.66%, 750=1.21%
  lat (msec)   : 1000=2.07%
  cpu          : usr=7.86%, sys=1.50%, ctx=867, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=10.0GiB (10.7GB), run=41380-41380msec

Disk stats (read/write):
  sda: ios=0/10084, merge=0/179, ticks=0/8230129, in_queue=1393920, util=20.98%
0 min passed...Guest Writes/Reads (Changes): 10240 MiB / 19 MiB ( 10240 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=764MiB/s][r=191 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=910: Mon Aug  9 04:31:55 2021
  read: IOPS=233, BW=935MiB/s (980MB/s)(10.0GiB/10953msec)
    slat (usec): min=83, max=14293, avg=239.68, stdev=637.79
    clat (msec): min=31, max=655, avg=272.73, stdev=123.81
     lat (msec): min=31, max=655, avg=272.97, stdev=123.78
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   84], 10.00th=[  101], 20.00th=[  131],
     | 30.00th=[  165], 40.00th=[  245], 50.00th=[  305], 60.00th=[  351],
     | 70.00th=[  376], 80.00th=[  393], 90.00th=[  409], 95.00th=[  418],
     | 99.00th=[  460], 99.50th=[  468], 99.90th=[  642], 99.95th=[  651],
     | 99.99th=[  659]
   bw (  KiB/s): min=696320, max=1622016, per=98.99%, avg=947715.57, stdev=252166.53, samples=21
   iops        : min=  170, max=  396, avg=231.24, stdev=61.54, samples=21
  lat (msec)   : 50=0.98%, 100=8.87%, 250=30.70%, 500=58.98%, 750=0.47%
  cpu          : usr=0.45%, sys=5.84%, ctx=1518, majf=0, minf=648
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=935MiB/s (980MB/s), 935MiB/s-935MiB/s (980MB/s-980MB/s), io=10.0GiB (10.7GB), run=10953-10953msec

Disk stats (read/write):
  sda: ios=10188/0, merge=0/0, ticks=2024091/0, in_queue=2066860, util=99.75%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10242 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Script ran 1854 seconds or 30 minutes.

Host:
CPU utilization increased from 1 to 37%. IO delay increased from 0 to 35%.

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...29664 MiB / 31424 MiB / 29640 MiB / 672 MiB / 684 MiB ( 29664 MiB / 31424 MiB / 29640 MiB / 672 MiB / 684 MiB )
2 min passed...29664 MiB / 31648 MiB / 29648 MiB / 672 MiB / 684 MiB ( 0 MiB / 224 MiB / 8 MiB / 0 MiB / 0 MiB )
3 min passed...29664 MiB / 32064 MiB / 29648 MiB / 672 MiB / 684 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...29664 MiB / 32064 MiB / 29648 MiB / 672 MiB / 684 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...29664 MiB / 32064 MiB / 29651 MiB / 672 MiB / 684 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
6 min passed...29664 MiB / 32192 MiB / 29651 MiB / 672 MiB / 684 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...29664 MiB / 32416 MiB / 29651 MiB / 672 MiB / 684 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...29664 MiB / 32672 MiB / 29651 MiB / 672 MiB / 684 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...29664 MiB / 32672 MiB / 29651 MiB / 672 MiB / 684 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...29664 MiB / 32672 MiB / 29658 MiB / 672 MiB / 684 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 0 MiB )
11 min passed...29664 MiB / 32768 MiB / 29660 MiB / 672 MiB / 684 MiB ( 0 MiB / 96 MiB / 2 MiB / 0 MiB / 0 MiB )
12 min passed...29664 MiB / 32928 MiB / 29664 MiB / 672 MiB / 684 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
13 min passed...29664 MiB / 33120 MiB / 29666 MiB / 672 MiB / 684 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
14 min passed...29664 MiB / 33120 MiB / 29668 MiB / 672 MiB / 684 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...29664 MiB / 33248 MiB / 29668 MiB / 672 MiB / 684 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...29664 MiB / 33344 MiB / 29668 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 96 MiB / 0 MiB / 9280 MiB / 9280 MiB )
17 min passed...29664 MiB / 33504 MiB / 29668 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...29664 MiB / 33664 MiB / 29668 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...29664 MiB / 33664 MiB / 29668 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...29696 MiB / 33792 MiB / 29675 MiB / 9952 MiB / 9964 MiB ( 32 MiB / 128 MiB / 7 MiB / 0 MiB / 0 MiB )
21 min passed...29696 MiB / 33792 MiB / 29675 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...29696 MiB / 34080 MiB / 29686 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...29696 MiB / 34176 MiB / 29698 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 96 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...29728 MiB / 34336 MiB / 29709 MiB / 9952 MiB / 9964 MiB ( 32 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
25 min passed...29728 MiB / 34432 MiB / 29723 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 96 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...29728 MiB / 34432 MiB / 29732 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
27 min passed...29760 MiB / 34688 MiB / 29746 MiB / 9952 MiB / 9964 MiB ( 32 MiB / 256 MiB / 14 MiB / 0 MiB / 0 MiB )
28 min passed...29760 MiB / 34784 MiB / 29756 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
29 min passed...29792 MiB / 34912 MiB / 29769 MiB / 9952 MiB / 9964 MiB ( 32 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
30 min passed...29792 MiB / 34912 MiB / 29785 MiB / 9952 MiB / 9964 MiB ( 0 MiB / 0 MiB / 16 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=776: Mon Aug  9 15:53:18 2021
  write: IOPS=26.9k, BW=420MiB/s (441MB/s)(1024MiB/2437msec); 0 zone resets
    slat (usec): min=7, max=2542, avg=24.16, stdev=20.74
    clat (usec): min=145, max=7841, avg=2342.91, stdev=572.67
     lat (usec): min=174, max=7857, avg=2368.54, stdev=578.06
    clat percentiles (usec):
     |  1.00th=[ 1713],  5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1893],
     | 30.00th=[ 1942], 40.00th=[ 1991], 50.00th=[ 2073], 60.00th=[ 2212],
     | 70.00th=[ 2704], 80.00th=[ 2868], 90.00th=[ 3064], 95.00th=[ 3261],
     | 99.00th=[ 4228], 99.50th=[ 4424], 99.90th=[ 5800], 99.95th=[ 6128],
     | 99.99th=[ 7373]
   bw (  KiB/s): min=380672, max=508864, per=100.00%, avg=444936.00, stdev=52742.20, samples=4
   iops        : min=23792, max=31804, avg=27808.50, stdev=3296.39, samples=4
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=40.33%, 4=58.28%, 10=1.35%
  cpu          : usr=20.98%, sys=54.89%, ctx=63964, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=420MiB/s (441MB/s), 420MiB/s-420MiB/s (441MB/s-441MB/s), io=1024MiB (1074MB), run=2437-2437msec

Disk stats (read/write):
  sda: ios=0/64450, merge=0/187, ticks=0/6210, in_queue=6760, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 17 MiB ( 1024 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1028 MiB / 17 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1028 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1028 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1028 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 17 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 18 MiB ( 0 MiB / 1 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1030 MiB / 18 MiB ( 1 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=1021: Mon Aug  9 16:08:20 2021
  read: IOPS=61.5k, BW=962MiB/s (1008MB/s)(1024MiB/1065msec)
    slat (usec): min=4, max=648, avg= 8.82, stdev= 5.36
    clat (usec): min=509, max=6004, avg=1024.61, stdev=203.62
     lat (usec): min=518, max=6013, avg=1034.76, stdev=204.07
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  848], 10.00th=[  889], 20.00th=[  930],
     | 30.00th=[  955], 40.00th=[  979], 50.00th=[ 1004], 60.00th=[ 1029],
     | 70.00th=[ 1057], 80.00th=[ 1090], 90.00th=[ 1156], 95.00th=[ 1237],
     | 99.00th=[ 1631], 99.50th=[ 1958], 99.90th=[ 5080], 99.95th=[ 5407],
     | 99.99th=[ 5604]
   bw (  KiB/s): min=964224, max=1004064, per=99.96%, avg=984144.00, stdev=28171.13, samples=2
   iops        : min=60264, max=62754, avg=61509.00, stdev=1760.70, samples=2
  lat (usec)   : 750=0.99%, 1000=48.34%
  lat (msec)   : 2=50.27%, 4=0.31%, 10=0.10%
  cpu          : usr=39.00%, sys=60.34%, ctx=60, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=962MiB/s (1008MB/s), 962MiB/s-962MiB/s (1008MB/s-1008MB/s), io=1024MiB (1074MB), run=1065-1065msec

Disk stats (read/write):
  sda: ios=55113/0, merge=0/0, ticks=23050/0, in_queue=23324, util=99.80%
0 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1032 MiB / 1042 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1032 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1032 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1042 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1033 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 1806 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1760 MiB / 1952 MiB / 1762 MiB / 32 MiB / 24 MiB ( 1760 MiB / 1952 MiB / 1762 MiB / 32 MiB / 24 MiB )
2 min passed...1792 MiB / 2144 MiB / 1782 MiB / 32 MiB / 24 MiB ( 32 MiB / 192 MiB / 20 MiB / 0 MiB / 0 MiB )
3 min passed...1824 MiB / 2304 MiB / 1802 MiB / 32 MiB / 24 MiB ( 32 MiB / 160 MiB / 20 MiB / 0 MiB / 0 MiB )
4 min passed...1824 MiB / 2336 MiB / 1823 MiB / 32 MiB / 24 MiB ( 0 MiB / 32 MiB / 21 MiB / 0 MiB / 0 MiB )
5 min passed...1856 MiB / 2336 MiB / 1843 MiB / 32 MiB / 24 MiB ( 32 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )
6 min passed...1856 MiB / 2528 MiB / 1862 MiB / 32 MiB / 24 MiB ( 0 MiB / 192 MiB / 19 MiB / 0 MiB / 0 MiB )
7 min passed...1888 MiB / 2848 MiB / 1881 MiB / 32 MiB / 24 MiB ( 32 MiB / 320 MiB / 19 MiB / 0 MiB / 0 MiB )
8 min passed...1920 MiB / 3008 MiB / 1901 MiB / 32 MiB / 24 MiB ( 32 MiB / 160 MiB / 20 MiB / 0 MiB / 0 MiB )
9 min passed...1920 MiB / 3040 MiB / 1921 MiB / 32 MiB / 24 MiB ( 0 MiB / 32 MiB / 20 MiB / 0 MiB / 0 MiB )
10 min passed...1952 MiB / 3072 MiB / 1938 MiB / 32 MiB / 24 MiB ( 32 MiB / 32 MiB / 17 MiB / 0 MiB / 0 MiB )
11 min passed...1952 MiB / 3104 MiB / 1955 MiB / 32 MiB / 24 MiB ( 0 MiB / 32 MiB / 17 MiB / 0 MiB / 0 MiB )
12 min passed...1984 MiB / 3296 MiB / 1975 MiB / 32 MiB / 24 MiB ( 32 MiB / 192 MiB / 20 MiB / 0 MiB / 0 MiB )
13 min passed...2016 MiB / 3296 MiB / 1993 MiB / 32 MiB / 24 MiB ( 32 MiB / 0 MiB / 18 MiB / 0 MiB / 0 MiB )
14 min passed...2016 MiB / 3296 MiB / 2013 MiB / 32 MiB / 24 MiB ( 0 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )
15 min passed...2048 MiB / 3360 MiB / 2030 MiB / 32 MiB / 24 MiB ( 32 MiB / 64 MiB / 17 MiB / 0 MiB / 0 MiB )
16 min passed...2048 MiB / 3360 MiB / 2049 MiB / 576 MiB / 555 MiB ( 0 MiB / 0 MiB / 19 MiB / 544 MiB / 531 MiB )
17 min passed...2080 MiB / 3424 MiB / 2066 MiB / 576 MiB / 555 MiB ( 32 MiB / 64 MiB / 17 MiB / 0 MiB / 0 MiB )
18 min passed...2080 MiB / 3456 MiB / 2082 MiB / 576 MiB / 555 MiB ( 0 MiB / 32 MiB / 16 MiB / 0 MiB / 0 MiB )
19 min passed...2112 MiB / 3456 MiB / 2101 MiB / 576 MiB / 555 MiB ( 32 MiB / 0 MiB / 19 MiB / 0 MiB / 0 MiB )
20 min passed...2144 MiB / 3488 MiB / 2120 MiB / 576 MiB / 555 MiB ( 32 MiB / 32 MiB / 19 MiB / 0 MiB / 0 MiB )
21 min passed...2144 MiB / 3680 MiB / 2137 MiB / 576 MiB / 555 MiB ( 0 MiB / 192 MiB / 17 MiB / 0 MiB / 0 MiB )
22 min passed...2144 MiB / 3744 MiB / 2153 MiB / 576 MiB / 555 MiB ( 0 MiB / 64 MiB / 16 MiB / 0 MiB / 0 MiB )
23 min passed...2176 MiB / 3744 MiB / 2172 MiB / 576 MiB / 555 MiB ( 32 MiB / 0 MiB / 19 MiB / 0 MiB / 0 MiB )
24 min passed...2208 MiB / 3808 MiB / 2191 MiB / 576 MiB / 555 MiB ( 32 MiB / 64 MiB / 19 MiB / 0 MiB / 0 MiB )
25 min passed...2208 MiB / 3840 MiB / 2211 MiB / 576 MiB / 555 MiB ( 0 MiB / 32 MiB / 20 MiB / 0 MiB / 0 MiB )
26 min passed...2240 MiB / 4000 MiB / 2228 MiB / 576 MiB / 555 MiB ( 32 MiB / 160 MiB / 17 MiB / 0 MiB / 0 MiB )
27 min passed...2240 MiB / 4032 MiB / 2245 MiB / 576 MiB / 555 MiB ( 0 MiB / 32 MiB / 17 MiB / 0 MiB / 0 MiB )
28 min passed...2272 MiB / 4032 MiB / 2261 MiB / 576 MiB / 555 MiB ( 32 MiB / 0 MiB / 16 MiB / 0 MiB / 0 MiB )
29 min passed...2272 MiB / 4064 MiB / 2278 MiB / 576 MiB / 555 MiB ( 0 MiB / 32 MiB / 17 MiB / 0 MiB / 0 MiB )
30 min passed...2304 MiB / 4128 MiB / 2295 MiB / 576 MiB / 555 MiB ( 32 MiB / 64 MiB / 17 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=12.5MiB/s][w=399 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=615: Mon Aug  9 16:42:28 2021
  write: IOPS=404, BW=12.6MiB/s (13.2MB/s)(1024MiB/81102msec); 0 zone resets
    clat (usec): min=1381, max=34626, avg=2448.26, stdev=795.24
     lat (usec): min=1383, max=34627, avg=2450.57, stdev=795.25
    clat percentiles (usec):
     |  1.00th=[ 1713],  5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2073],
     | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2343], 60.00th=[ 2442],
     | 70.00th=[ 2507], 80.00th=[ 2606], 90.00th=[ 2769], 95.00th=[ 2933],
     | 99.00th=[ 6063], 99.50th=[ 7046], 99.90th=[10159], 99.95th=[11469],
     | 99.99th=[21103]
   bw (  KiB/s): min= 6784, max=17152, per=99.97%, avg=12925.49, stdev=1947.00, samples=162
   iops        : min=  212, max=  536, avg=403.92, stdev=60.84, samples=162
  lat (msec)   : 2=13.72%, 4=83.23%, 10=2.94%, 20=0.09%, 50=0.02%
  cpu          : usr=1.36%, sys=5.57%, ctx=78279, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=12.6MiB/s (13.2MB/s), 12.6MiB/s-12.6MiB/s (13.2MB/s-13.2MB/s), io=1024MiB (1074MB), run=81102-81102msec

Disk stats (read/write):
  sda: ios=0/98173, merge=0/65931, ticks=0/72635, in_queue=80860, util=99.87%
0 min passed...Guest Writes/Reads (Changes): 1539 MiB / 19 MiB ( 1539 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1539 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1542 MiB / 19 MiB ( 1 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=56.1MiB/s][r=1794 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=820: Mon Aug  9 16:57:47 2021
  read: IOPS=1769, BW=55.3MiB/s (57.0MB/s)(1024MiB/18520msec)
    clat (usec): min=377, max=7522, avg=555.69, stdev=308.50
     lat (usec): min=379, max=7524, avg=557.52, stdev=308.54
    clat percentiles (usec):
     |  1.00th=[  429],  5.00th=[  449], 10.00th=[  457], 20.00th=[  474],
     | 30.00th=[  482], 40.00th=[  494], 50.00th=[  502], 60.00th=[  515],
     | 70.00th=[  529], 80.00th=[  545], 90.00th=[  578], 95.00th=[  635],
     | 99.00th=[ 2704], 99.50th=[ 2966], 99.90th=[ 3294], 99.95th=[ 3392],
     | 99.99th=[ 3949]
   bw (  KiB/s): min=49856, max=58688, per=100.00%, avg=56640.00, stdev=1618.95, samples=37
   iops        : min= 1558, max= 1834, avg=1770.00, stdev=50.59, samples=37
  lat (usec)   : 500=46.72%, 750=49.90%, 1000=0.71%
  lat (msec)   : 2=1.08%, 4=1.57%, 10=0.01%
  cpu          : usr=2.79%, sys=7.65%, ctx=32869, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=55.3MiB/s (57.0MB/s), 55.3MiB/s-55.3MiB/s (57.0MB/s-57.0MB/s), io=1024MiB (1074MB), run=18520-18520msec

Disk stats (read/write):
  sda: ios=32563/7, merge=0/6, ticks=17052/7, in_queue=18468, util=99.84%
0 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1043 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1544 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1902 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...8736 MiB / 9248 MiB / 8737 MiB / 32 MiB / 17 MiB ( 8736 MiB / 9248 MiB / 8737 MiB / 32 MiB / 17 MiB )
2 min passed...12416 MiB / 13344 MiB / 12388 MiB / 32 MiB / 19 MiB ( 3680 MiB / 4096 MiB / 3651 MiB / 0 MiB / 2 MiB )
3 min passed...12416 MiB / 13408 MiB / 12399 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
4 min passed...12416 MiB / 13408 MiB / 12412 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...12448 MiB / 13440 MiB / 12426 MiB / 32 MiB / 19 MiB ( 32 MiB / 32 MiB / 14 MiB / 0 MiB / 0 MiB )
6 min passed...12448 MiB / 13472 MiB / 12438 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
7 min passed...12480 MiB / 13792 MiB / 12451 MiB / 32 MiB / 19 MiB ( 32 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
8 min passed...12480 MiB / 13984 MiB / 12461 MiB / 32 MiB / 19 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
9 min passed...12480 MiB / 13984 MiB / 12474 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
10 min passed...12512 MiB / 13984 MiB / 12486 MiB / 32 MiB / 19 MiB ( 32 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
11 min passed...12512 MiB / 14016 MiB / 12498 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
12 min passed...12544 MiB / 14208 MiB / 12508 MiB / 32 MiB / 19 MiB ( 32 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...12544 MiB / 14272 MiB / 12521 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...12544 MiB / 14272 MiB / 12532 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
15 min passed...12576 MiB / 14304 MiB / 12546 MiB / 32 MiB / 19 MiB ( 32 MiB / 32 MiB / 14 MiB / 0 MiB / 0 MiB )
16 min passed...12576 MiB / 14432 MiB / 12557 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
17 min passed...12576 MiB / 14816 MiB / 12567 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 384 MiB / 10 MiB / 1024 MiB / 1024 MiB )
18 min passed...12608 MiB / 14816 MiB / 12579 MiB / 1056 MiB / 1043 MiB ( 32 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
19 min passed...12608 MiB / 14848 MiB / 12588 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
20 min passed...12608 MiB / 14848 MiB / 12599 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...12640 MiB / 15072 MiB / 12609 MiB / 1056 MiB / 1043 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...12640 MiB / 15456 MiB / 12620 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 384 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...12640 MiB / 15456 MiB / 12631 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
24 min passed...12672 MiB / 15456 MiB / 12642 MiB / 1056 MiB / 1043 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
25 min passed...12672 MiB / 15456 MiB / 12654 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
26 min passed...12672 MiB / 15616 MiB / 12665 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
27 min passed...12704 MiB / 15616 MiB / 12676 MiB / 1056 MiB / 1043 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
28 min passed...12704 MiB / 15616 MiB / 12685 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
29 min passed...12704 MiB / 15648 MiB / 12696 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...12736 MiB / 15648 MiB / 12705 MiB / 1056 MiB / 1043 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
31 min passed...12736 MiB / 15904 MiB / 12717 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=620: Mon Aug  9 17:17:40 2021
  write: IOPS=23.8k, BW=745MiB/s (781MB/s)(1024MiB/1374msec); 0 zone resets
    slat (usec): min=8, max=8332, avg=21.13, stdev=60.95
    clat (usec): min=121, max=28978, avg=2640.15, stdev=1302.70
     lat (usec): min=142, max=28991, avg=2662.67, stdev=1309.59
    clat percentiles (usec):
     |  1.00th=[ 1991],  5.00th=[ 2057], 10.00th=[ 2089], 20.00th=[ 2147],
     | 30.00th=[ 2180], 40.00th=[ 2212], 50.00th=[ 2245], 60.00th=[ 2311],
     | 70.00th=[ 2606], 80.00th=[ 2868], 90.00th=[ 3359], 95.00th=[ 4228],
     | 99.00th=[ 7177], 99.50th=[ 8717], 99.90th=[21103], 99.95th=[23200],
     | 99.99th=[24249]
   bw (  KiB/s): min=798208, max=882011, per=100.00%, avg=840109.50, stdev=59257.67, samples=2
   iops        : min=24944, max=27562, avg=26253.00, stdev=1851.21, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.00%, 4=92.79%, 10=5.76%, 20=0.20%, 50=0.20%
  cpu          : usr=36.93%, sys=40.79%, ctx=30364, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=745MiB/s (781MB/s), 745MiB/s-745MiB/s (781MB/s-781MB/s), io=1024MiB (1074MB), run=1374-1374msec

Disk stats (read/write):
  sda: ios=0/28582, merge=0/100, ticks=0/4612, in_queue=5276, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1025 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=835: Mon Aug  9 17:32:42 2021
  read: IOPS=22.2k, BW=694MiB/s (727MB/s)(1024MiB/1476msec)
    slat (usec): min=5, max=769, avg=15.83, stdev=23.49
    clat (usec): min=595, max=12546, avg=2858.40, stdev=510.79
     lat (usec): min=605, max=12562, avg=2875.64, stdev=510.10
    clat percentiles (usec):
     |  1.00th=[ 1188],  5.00th=[ 2089], 10.00th=[ 2343], 20.00th=[ 2573],
     | 30.00th=[ 2704], 40.00th=[ 2802], 50.00th=[ 2868], 60.00th=[ 2966],
     | 70.00th=[ 3064], 80.00th=[ 3163], 90.00th=[ 3359], 95.00th=[ 3490],
     | 99.00th=[ 4047], 99.50th=[ 4817], 99.90th=[ 6849], 99.95th=[ 7046],
     | 99.99th=[ 7570]
   bw (  KiB/s): min=698688, max=734336, per=100.00%, avg=716512.00, stdev=25206.94, samples=2
   iops        : min=21834, max=22948, avg=22391.00, stdev=787.72, samples=2
  lat (usec)   : 750=0.09%, 1000=0.47%
  lat (msec)   : 2=3.53%, 4=94.84%, 10=1.07%, 20=0.01%
  cpu          : usr=21.15%, sys=42.37%, ctx=6468, majf=0, minf=519
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=694MiB/s (727MB/s), 694MiB/s-694MiB/s (727MB/s-727MB/s), io=1024MiB (1074MB), run=1476-1476msec

Disk stats (read/write):
  sda: ios=31116/0, merge=0/0, ticks=85345/0, in_queue=85808, util=99.87%
0 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1027 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2784 MiB / 2944 MiB / 2791 MiB / 32 MiB / 43 MiB ( 2784 MiB / 2944 MiB / 2791 MiB / 32 MiB / 43 MiB )
2 min passed...2816 MiB / 2976 MiB / 2804 MiB / 32 MiB / 43 MiB ( 32 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
3 min passed...2816 MiB / 3008 MiB / 2816 MiB / 32 MiB / 43 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
4 min passed...2816 MiB / 3200 MiB / 2827 MiB / 32 MiB / 43 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...2848 MiB / 3232 MiB / 2837 MiB / 32 MiB / 43 MiB ( 32 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...2848 MiB / 3232 MiB / 2846 MiB / 32 MiB / 43 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
7 min passed...2848 MiB / 3232 MiB / 2858 MiB / 32 MiB / 43 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
8 min passed...2880 MiB / 3392 MiB / 2867 MiB / 32 MiB / 43 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
9 min passed...2880 MiB / 3456 MiB / 2869 MiB / 32 MiB / 43 MiB ( 0 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...2880 MiB / 3456 MiB / 2869 MiB / 32 MiB / 43 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2880 MiB / 3456 MiB / 2869 MiB / 32 MiB / 43 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2880 MiB / 3456 MiB / 2869 MiB / 32 MiB / 43 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2880 MiB / 3648 MiB / 2871 MiB / 32 MiB / 43 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
14 min passed...2880 MiB / 3904 MiB / 2874 MiB / 32 MiB / 43 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
15 min passed...2880 MiB / 3968 MiB / 2877 MiB / 32 MiB / 43 MiB ( 0 MiB / 64 MiB / 3 MiB / 0 MiB / 0 MiB )
16 min passed...2880 MiB / 3968 MiB / 2877 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 1216 MiB / 1207 MiB )
17 min passed...2880 MiB / 4000 MiB / 2877 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2880 MiB / 4192 MiB / 2880 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 192 MiB / 3 MiB / 0 MiB / 0 MiB )
19 min passed...2880 MiB / 4256 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
20 min passed...2880 MiB / 4256 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2880 MiB / 4256 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2880 MiB / 4416 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2880 MiB / 4480 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2880 MiB / 4480 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2880 MiB / 4480 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2880 MiB / 4480 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2880 MiB / 4640 MiB / 2882 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2880 MiB / 4800 MiB / 2886 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
29 min passed...2880 MiB / 4992 MiB / 2888 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
30 min passed...2880 MiB / 4992 MiB / 2888 MiB / 1248 MiB / 1250 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Here are the diagrams for run 3 (2 disk mirror):

Total Write Amplification:

	Total Write Amplifcation	Read Overhead	Total Read Amplification
R3T1: sync 4K rand	56,84	0,00	1,00
R3T2: sync 16K rand	17,09	0,03	1,00
R3T3: sync 4M seq	4,62	0,00	1,22
R3T4: async 4K rand	5,44	0,00	1,00
R3T5: async 16K rand	6,06	0,03	1,00
R3T6: async 4M seq	3,25	0,07	0,91
R3T7: async 16K rand 50% cmp	3,28	0,03	0,53
R3T8: sync 32K rand	14,09	0,03	1,00
R3T9: async 32K rand	3,88	0,03	1,19

Performance:

	Write Throughput (MiB/s)	Read Throughput (MiB/s)
R3T1: sync 4K rand	1,87	9,25
R3T2: sync 16K rand	7,19	32,90
R3T3: sync 4M seq	131,00	464,00
R3T4: async 4K rand	125,00	295,00
R3T5: async 16K rand	450,00	429,00
R3T6: async 4M seq	247,00	935,00
R3T7: async 16K rand 50% cmp	420,00	962,00
R3T8: sync 32K rand	12,60	55,30
R3T9: async 32K rand	745,00	694,00

Detailed Write Amplification:

	Write Amplification Fio -> Guest	Write Amplification Guest -> Host	Write Amplification Host -> NAND
R3T1: sync 4K rand	5,11	10,02	1,11
R3T2: sync 16K rand	2,01	7,16	1,19
R3T3: sync 4M seq	1,00	4,19	1,10
R3T4: async 4K rand	1,01	3,67	1,46
R3T5: async 16K rand	1,00	4,18	1,45
R3T6: async 4M seq	1,00	2,90	1,12
R3T7: async 16K rand 50% cmp	1,01	1,97	1,64
R3T8: sync 32K rand	1,51	8,14	1,15
R3T9: async 32K rand	1,00	2,80	1,38

Dunuin · Aug 9, 2021

And here some diagrams comparing results of run 1 to 3...

What I don't understand is why the write amplification is always that bad no matter what kind of drive setup or volblocksize I'm using.
Its also strange that the 2 disk mirror beat the 5 disk raidz1 and the 8 disk striped mirror everywhere concerning performance except for sequential 4M writes.

Can someone explain why a 4K volblocksize 2 disk mirror can write 32K sync writes faster than a 16K striped mirror of 8 disks?
The 32K block should be devided into 8x 4K blocks so that the 8 disk stripe mirror can write 4 4K in parallel so that should result in 4 times the performance in theory. Here it looks like 6 of 8 disks are doing nothing at all for async/sync random writes. Same with the 16K sync random writes.

Dunuin · Aug 9, 2021

Looks like XFS is better for small sync writes then ext4.

Did a test (fio writing 1GiB of sync 4K random writes inside a debian 10 VM on a ashift 12 zfs mirror using 4K volblocksize) comparing xfs with ext4 because I read in a paper that xfs should got less write amplification on small writes because the journaling is more efficient:

	xfs	ext4
Guest writes (MiB):	2932 (W.A. fio -> guest: 2,86x)	5226 (W.A. fio -> guest: 5,1x)
NAND writes (MiB):	40544 (W.A. total: 39,6x)	57312 (W.A. total: 56x)
Performance (MiB/s):	2,42	1,93

So using XFS instead of ext4 is the first thing I found out to really make a difference. ext2 would also be an option that should be even more efficient but that only because it lacks the journaling and I don't think its a good idea to sacrifice journaling just to get a lower write amplification.

Someone knows what I could test to lower the write amplification of ZFS or virtio?

Dunuin · Aug 10, 2021

Another thing that is causing alot of write amplification is the native encryption...

Did a test (fio writing 1GiB of sync 16K random writes that are 50% compressible inside a debian 10 VM using xfs on a ashift 12 zfs 4 disk striped mirror using 8K volblocksize) comparing encrypted vs unencrypted VM:

	encrypted	unencrypted
Guest writes (MiB):	1.519 (W.A. fio -> guest: 1,48x)	1.519 (W.A. fio -> guest: 1,48x)
NAND writes (MiB):	12.384 (W.A. total: 12,09x)	6.240 (W.A. total: 6,09x)
Performance (MiB/s):	8,00	10,1

So using encryption basically doubled the writes ZFS is doing.

chrcoluk · Aug 10, 2021

Very good thread, my first thoughts are (without reading any of the replies).

I am wasnt aware that the virtio layer has a configurable block size, 4k should very likely perform better than 512bytes or on par at worse, less i/o and alignment with modern disks.

Ashift is akin to minimum block size, so again never set this below 4k.

I have absolutely no experience with raidz, my reason been I prefer the simplicity of mirror and the fact mirror vdevs can be setup with lower hardware capital (just 2 disks), I feel a raidz2 only starts to be worth it with minimum 6 disks and I have never had a use case that justifies that expenditure. However I have done some reading up on it, and as I understand it read performance on mirrors is multiple of number of disks, whilst write on raidz is multiple of number of disks, based on slowest disk.

With zvol I found the block size is best to match the block size on the guest filesystem, and should be a multiple of the host storage block size for correct alignment. With spindle backed storage and using large files I found increasing the block size extremely effective, so e.g. 64k blocks on guest, 64k zvol blocksize with of course 4k ashift.

In addition something I very recently discovered from some help on reddit is getting the ARC to help data and prefetch performance properly, on a proxmox system I had horrific data caching levels, the worst I ever seen from a ZFS server, increasing ARC size barely had any impact, and the prefetch hit rate was < 0.1%. It was suggested to me to make one adjustment, which was to increase 'zfetch_max_distance', I increased it to 67108864. The prefetch hit rate was still low but no longer abysmal, it increased to around 32.7%. The impact on i/o delay was immediate on the graph, its now almost flatlined at a very low level. Enabling prefetch on default tunables had no visible affect, so this was a wow moment.

Now of course you are using SSDs like many others, they are a luxury I cannot afford without justification, so the metrics may be very different. However I provided this information as it just may have some use to you.

I am an odd one, most people use raidz and tons of disks, I use mirrors with two disks or four disks.

Also in regards to amplification I have something else to report.

Two nights ago my first proxmox server started an automated srub on all pools, the server collapsed almost, extreme lag, high i/o delay you name it. I got things to a usable state by cancelling all scrubs except on rpool and also crippling scrub to one i/o thread. When it got down to the final gig, it was running at 28meg/sec and should have been done in 40 seconds at that rate but took 32 minutes, there seemed to be very severe read amplification for a unknown reason. The estimated time remaining moved down by 1 second for approx every 30 seconds of actual time passed, was very painful to watch. I had to disable the srcub cron on that server for now pending further investigation.

Dunuin · Aug 10, 2021

chrcoluk said:
With zvol I found the block size is best to match the block size on the guest filesystem, and should be a multiple of the host storage block size for correct alignment. With spindle backed storage and using large files I found increasing the block size extremely effective, so e.g. 64k blocks on guest, 64k zvol blocksize with of course 4k ashift.

Yes but thats a problem. My physical disks are all 4K and my guests OSs all use 4K blocksize too. Some stuff is writing bigger blocks of data (like MySQL with 16K blocks) but stuff like the elasticsearch DB uses java and java is using the defaults of the OS its installed on, so the elastic search DB should write with 4K blocks. So no matter what volblocksize I choose, there is always some stuff that will write with a blocksize that is lower than my volblocksize.

chrcoluk said:
In addition something I very recently discovered from some help on reddit is getting the ARC to help data and prefetch performance properly, on a proxmox system I had horrific data caching levels, the worst I ever seen from a ZFS server, increasing ARC size barely had any impact, and the prefetch hit rate was < 0.1%. It was suggested to me to make one adjustment, which was to increase 'zfetch_max_distance', I increased it to 67108864. The prefetch hit rate was still low but no longer abysmal, it increased to around 32.7%. The impact on i/o delay was immediate on the graph, its now almost flatlined at a very low level. Enabling prefetch on default tunables had no visible affect, so this was a wow moment.

Thanks. Didn't know about that. But I think my prefetch value also was very low.

chrcoluk said:
Now of course you are using SSDs like many others, they are a luxury I cannot afford without justification, so the metrics may be very different. However I provided this information as it just may have some use to you.

I am an odd one, most people use raidz and tons of disks, I use mirrors with two disks or four disks.

They aren't that expensive if you buy them second hand. The 10 SSDs in my proxmox server were 250€ all together. Thats basically 127 TBW per Euro so a really good deal. A Samsung QVO would be only 4 TBW per Euro.

Dunuin · Aug 10, 2021

There is still the question why I always get the same sync/async random write performance doing the same fio test with 5 drives raidz1, 8 drives striped mirror and 2 drives mirror. For 4K random writes I would unterstand it because 4K is the smallest possible blocksize with ashift12 so it can't be split and written to multiple striped drives. But its the same with 16K and 32K random writes.

I really would like to finish this so I can start my services again. This is my production server so I can't run any VMs while testing pools. But right now I don't have any idea how I should setup my pool.

Right know it looks like l would get the best porformance by just creating 3 individual mirrors and spread VMs across them. According to the fio benchmarks that would be 3-4 times faster than a striped mirror of 8 disks...

Or for lowest write amplification I could use a 5 disk raidz1 for my normal VMs + 1 disk LVM for write heavy DB VMs so I don't get all the ZFS/CoW overhead.

So there really must be something wrong...

Any ideas what pool configuration should give the lowest write amplification while still keeping the data safe?

guletz · Aug 11, 2021

Dunuin said:
So if fio wrote 1 GiB but the guest reported 5 GiB were written to the virtual disks thats a write amplification of factor 5.

Hi again @Dunuin

This was also intrigued me.... But you also give the possible explanation. .. The problem is ext 4/3 jurnal. After some google the fact is that for each 4k block the jurnal is writing 32 k data + 4k metadata. Thx. a lot for this very important info, I never think until I read your post about the ext 3/4 jurnal.

Now if you want and have some time maybe you can repeat at least 2 of your tests(sync and async) but with a ext 3/4 jurnal disabled.

Source: https://ilinuxkernel.com/?p=1477

Good luck / Bafta !

Dunuin · Aug 11, 2021

Will sync 4K + async 4K on a 2 disk mirror with 4k volblocksize be fine?

Do you know if a journal inside the VM is needed at all if a zvol is used as storage? On an power outage or host kernel crash the data of the zvol still should be safe because ZFS is journaling too. Or is a journal inside the guest still useful if just the VM is crashing or needs a hard-reset becasue its frozen?

guletz · Aug 11, 2021

Dunuin said:
Will sync 4K + async 4K on a 2 disk mirror with 4k volblocksize be fine?

Do you know if a journal inside the VM is needed at all if a zvol is used as storage? On an power outage or host kernel crash the data of the zvol still should be safe because ZFS is journaling too.

yes. I have do some tests in the past with some ancient fox-pro dbs, and and I do not see any data corruption at db point of view.

Dunuin said:
Or is a journal inside the guest still useful if just the VM is crashing or needs a hard-reset

Yes. But tomorrow I can try to test this scenario to see what is happend with a VM (journal disable).

Good luck / Bafta !

Dunuin · Aug 11, 2021

Special Test Nr. 14: Like Round 2 Run 3 Test 1 (sync 4K on 2 disk mirror with 4k volblocksize) but ext4 without journaling

Guest:

Code:

root@DebianTest2:~# dmesg | grep EXT4
[   33.452710] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[   33.890136] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[   34.315889] EXT4-fs (sdb1): mounted filesystem without journal. Opts: (null)
root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=5029KiB/s][w=1257 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=663: Wed Aug 11 19:24:44 2021
  write: IOPS=1089, BW=4359KiB/s (4464kB/s)(1024MiB/240540msec); 0 zone resets
    clat (usec): min=462, max=23561, avg=905.31, stdev=284.73
     lat (usec): min=463, max=23563, avg=907.18, stdev=284.78
    clat percentiles (usec):
     |  1.00th=[  644],  5.00th=[  701], 10.00th=[  742], 20.00th=[  783],
     | 30.00th=[  816], 40.00th=[  848], 50.00th=[  881], 60.00th=[  914],
     | 70.00th=[  947], 80.00th=[  996], 90.00th=[ 1057], 95.00th=[ 1106],
     | 99.00th=[ 1303], 99.50th=[ 2540], 99.90th=[ 4080], 99.95th=[ 5407],
     | 99.99th=[ 9765]
   bw (  KiB/s): min= 3080, max= 5760, per=100.00%, avg=4359.02, stdev=408.27, samples=481
   iops        : min=  770, max= 1440, avg=1089.75, stdev=102.07, samples=481
  lat (usec)   : 500=0.01%, 750=12.58%, 1000=68.87%
  lat (msec)   : 2=17.86%, 4=0.59%, 10=0.10%, 20=0.01%, 50=0.01%
  cpu          : usr=2.19%, sys=13.47%, ctx=1109778, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4359KiB/s (4464kB/s), 4359KiB/s-4359KiB/s (4464kB/s-4464kB/s), io=1024MiB (1074MB), run=240540-240540msec

Disk stats (read/write):
  sdb: ios=0/847574, merge=0/38, ticks=0/214279, in_queue=234236, util=97.41%
0 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 2289 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2289 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2290 MiB / 19 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2290 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2290 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2290 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2290 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][99.1%][r=8892KiB/s][r=2223 IOPS][eta 00m:01s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=868: Wed Aug 11 19:41:30 2021
  read: IOPS=2476, BW=9906KiB/s (10.1MB/s)(1024MiB/105854msec)
    clat (usec): min=192, max=5931, avg=393.68, stdev=118.39
     lat (usec): min=194, max=5933, avg=395.52, stdev=118.49
    clat percentiles (usec):
     |  1.00th=[  260],  5.00th=[  277], 10.00th=[  289], 20.00th=[  306],
     | 30.00th=[  322], 40.00th=[  347], 50.00th=[  367], 60.00th=[  400],
     | 70.00th=[  424], 80.00th=[  478], 90.00th=[  545], 95.00th=[  578],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[ 1516], 99.95th=[ 2376],
     | 99.99th=[ 3097]
   bw (  KiB/s): min= 7256, max=13920, per=99.88%, avg=9893.19, stdev=1732.77, samples=211
   iops        : min= 1814, max= 3480, avg=2473.29, stdev=433.20, samples=211
  lat (usec)   : 250=0.32%, 500=83.35%, 750=16.15%, 1000=0.04%
  lat (msec)   : 2=0.07%, 4=0.07%, 10=0.01%
  cpu          : usr=3.89%, sys=11.83%, ctx=262218, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=9906KiB/s (10.1MB/s), 9906KiB/s-9906KiB/s (10.1MB/s-10.1MB/s), io=1024MiB (1074MB), run=105854-105854msec

Disk stats (read/write):
  sdb: ios=261402/0, merge=0/0, ticks=94317/0, in_queue=105608, util=99.95%
0 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2290 MiB / 1043 MiB ( 0 MiB / 0 MiB )
rm: cannot remove '/mnt/test/lost+found': Is a directory
Script ran 2148 seconds or 35 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5088 MiB / 5696 MiB / 5107 MiB / 32 MiB / 17 MiB ( 5088 MiB / 5696 MiB / 5107 MiB / 32 MiB / 17 MiB )
2 min passed...10592 MiB / 11488 MiB / 10596 MiB / 32 MiB / 17 MiB ( 5504 MiB / 5792 MiB / 5489 MiB / 0 MiB / 0 MiB )
3 min passed...16192 MiB / 17440 MiB / 16197 MiB / 32 MiB / 17 MiB ( 5600 MiB / 5952 MiB / 5601 MiB / 0 MiB / 0 MiB )
4 min passed...22048 MiB / 23744 MiB / 22048 MiB / 32 MiB / 17 MiB ( 5856 MiB / 6304 MiB / 5851 MiB / 0 MiB / 0 MiB )
5 min passed...22336 MiB / 24064 MiB / 22340 MiB / 32 MiB / 19 MiB ( 288 MiB / 320 MiB / 292 MiB / 0 MiB / 2 MiB )
6 min passed...22336 MiB / 24192 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 3 MiB / 0 MiB / 0 MiB )
7 min passed...22336 MiB / 24192 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...22336 MiB / 24288 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...22336 MiB / 24448 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...22336 MiB / 24576 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...22336 MiB / 24672 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...22336 MiB / 24672 MiB / 22343 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...22336 MiB / 24928 MiB / 22349 MiB / 32 MiB / 19 MiB ( 0 MiB / 256 MiB / 6 MiB / 0 MiB / 0 MiB )
14 min passed...22336 MiB / 25024 MiB / 22351 MiB / 32 MiB / 19 MiB ( 0 MiB / 96 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...22368 MiB / 25280 MiB / 22358 MiB / 32 MiB / 19 MiB ( 32 MiB / 256 MiB / 7 MiB / 0 MiB / 0 MiB )
16 min passed...22368 MiB / 25312 MiB / 22358 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...22368 MiB / 25408 MiB / 22358 MiB / 32 MiB / 19 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...22368 MiB / 25664 MiB / 22358 MiB / 32 MiB / 19 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...22368 MiB / 25792 MiB / 22358 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...22368 MiB / 25920 MiB / 22358 MiB / 608 MiB / 585 MiB ( 0 MiB / 128 MiB / 0 MiB / 576 MiB / 566 MiB )
21 min passed...22368 MiB / 25920 MiB / 22358 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 448 MiB / 458 MiB )
22 min passed...22368 MiB / 26048 MiB / 22358 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...22368 MiB / 26048 MiB / 22358 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...22368 MiB / 26272 MiB / 22358 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...22368 MiB / 26304 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 5 MiB / 0 MiB / 0 MiB )
26 min passed...22368 MiB / 26400 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...22368 MiB / 26592 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...22368 MiB / 26688 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...22368 MiB / 26688 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...22368 MiB / 26688 MiB / 22363 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...22368 MiB / 26688 MiB / 22365 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
32 min passed...22368 MiB / 26688 MiB / 22366 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
33 min passed...22368 MiB / 26848 MiB / 22366 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...22368 MiB / 26848 MiB / 22366 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...22368 MiB / 26944 MiB / 22372 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 96 MiB / 6 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 11, 2021

Special Test Nr. 15: Like Round 2 Run 3 Test 4 (async 4K on 2 disk mirror with 4k volblocksize) but ext4 without journaling

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=148MiB/s][w=37.8k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=690: Wed Aug 11 20:02:32 2021
  write: IOPS=34.0k, BW=133MiB/s (139MB/s)(1024MiB/7703msec); 0 zone resets
    slat (usec): min=6, max=5264, avg=21.21, stdev=17.48
    clat (usec): min=40, max=16992, avg=1852.94, stdev=399.52
     lat (usec): min=97, max=17005, avg=1875.45, stdev=401.69
    clat percentiles (usec):
     |  1.00th=[ 1549],  5.00th=[ 1614], 10.00th=[ 1631], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1745], 50.00th=[ 1762], 60.00th=[ 1795],
     | 70.00th=[ 1827], 80.00th=[ 1893], 90.00th=[ 2212], 95.00th=[ 2507],
     | 99.00th=[ 2900], 99.50th=[ 3195], 99.90th=[ 4948], 99.95th=[ 7373],
     | 99.99th=[16581]
   bw (  KiB/s): min=107240, max=154448, per=100.00%, avg=137139.93, stdev=14063.07, samples=15
   iops        : min=26810, max=38612, avg=34284.93, stdev=3515.87, samples=15
  lat (usec)   : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=85.35%, 4=14.46%, 10=0.16%, 20=0.02%
  cpu          : usr=17.41%, sys=57.47%, ctx=278752, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=1024MiB (1074MB), run=7703-7703msec

Disk stats (read/write):
  sdb: ios=0/260921, merge=0/651, ticks=0/11186, in_queue=12472, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 1027 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=288MiB/s][r=73.8k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=920: Wed Aug 11 20:17:36 2021
  read: IOPS=72.3k, BW=282MiB/s (296MB/s)(1024MiB/3627msec)
    slat (usec): min=4, max=188, avg= 7.21, stdev= 3.55
    clat (usec): min=199, max=2628, avg=873.38, stdev=76.89
     lat (usec): min=212, max=2633, avg=881.72, stdev=77.53
    clat percentiles (usec):
     |  1.00th=[  783],  5.00th=[  799], 10.00th=[  816], 20.00th=[  832],
     | 30.00th=[  840], 40.00th=[  857], 50.00th=[  865], 60.00th=[  873],
     | 70.00th=[  889], 80.00th=[  906], 90.00th=[  930], 95.00th=[  955],
     | 99.00th=[ 1139], 99.50th=[ 1188], 99.90th=[ 1745], 99.95th=[ 2114],
     | 99.99th=[ 2442]
   bw (  KiB/s): min=278320, max=295264, per=99.97%, avg=289021.71, stdev=7692.78, samples=7
   iops        : min=69580, max=73816, avg=72255.43, stdev=1923.20, samples=7
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.07%, 1000=96.51%
  lat (msec)   : 2=3.34%, 4=0.06%
  cpu          : usr=43.24%, sys=56.73%, ctx=21, majf=0, minf=73
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=1024MiB (1074MB), run=3627-3627msec

Disk stats (read/write):
  sdb: ios=245405/0, merge=0/0, ticks=49116/0, in_queue=49356, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
rm: cannot remove '/mnt/test/lost+found': Is a directory
Script ran 1813 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2208 MiB / 2336 MiB / 2204 MiB / 32 MiB / 20 MiB ( 2208 MiB / 2336 MiB / 2204 MiB / 32 MiB / 20 MiB )
2 min passed...2208 MiB / 2336 MiB / 2204 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2208 MiB / 2336 MiB / 2206 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...2208 MiB / 2400 MiB / 2210 MiB / 32 MiB / 20 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
5 min passed...2208 MiB / 2560 MiB / 2212 MiB / 32 MiB / 20 MiB ( 0 MiB / 160 MiB / 2 MiB / 0 MiB / 0 MiB )
6 min passed...2208 MiB / 2560 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
7 min passed...2208 MiB / 2688 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2208 MiB / 2720 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2208 MiB / 2912 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2208 MiB / 2944 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2208 MiB / 2944 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2208 MiB / 3040 MiB / 2214 MiB / 32 MiB / 20 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2208 MiB / 3072 MiB / 2221 MiB / 32 MiB / 20 MiB ( 0 MiB / 32 MiB / 7 MiB / 0 MiB / 0 MiB )
14 min passed...2208 MiB / 3232 MiB / 2226 MiB / 32 MiB / 20 MiB ( 0 MiB / 160 MiB / 5 MiB / 0 MiB / 0 MiB )
15 min passed...2208 MiB / 3232 MiB / 2228 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
16 min passed...2240 MiB / 3296 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 64 MiB / 6 MiB / 1024 MiB / 1024 MiB )
17 min passed...2240 MiB / 3296 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2240 MiB / 3456 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2240 MiB / 3520 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2240 MiB / 3520 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2240 MiB / 3520 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2240 MiB / 3552 MiB / 2234 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2240 MiB / 3808 MiB / 2236 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
24 min passed...2240 MiB / 3808 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
25 min passed...2240 MiB / 3808 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2240 MiB / 3808 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2240 MiB / 3968 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2240 MiB / 4096 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2240 MiB / 4096 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2240 MiB / 4160 MiB / 2240 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 11, 2021

Special Test Nr. 16: Like Round 2 Run 3 Test 1 (sync 4K on 2 disk mirror with 4k volblocksize) but with xfs with journaling instead of ext4 with journaling

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=2676KiB/s][w=669 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=649: Wed Aug 11 20:51:07 2021
  write: IOPS=615, BW=2462KiB/s (2521kB/s)(1024MiB/425934msec); 0 zone resets
    clat (usec): min=677, max=37736, avg=1610.80, stdev=535.40
     lat (usec): min=678, max=37738, avg=1612.89, stdev=535.43
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1123], 10.00th=[ 1221], 20.00th=[ 1336],
     | 30.00th=[ 1434], 40.00th=[ 1516], 50.00th=[ 1598], 60.00th=[ 1663],
     | 70.00th=[ 1729], 80.00th=[ 1811], 90.00th=[ 1926], 95.00th=[ 2008],
     | 99.00th=[ 2409], 99.50th=[ 4621], 99.90th=[ 7701], 99.95th=[10552],
     | 99.99th=[17957]
   bw (  KiB/s): min= 1640, max= 3504, per=100.00%, avg=2461.20, stdev=266.89, samples=851
   iops        : min=  410, max=  876, avg=615.29, stdev=66.72, samples=851
  lat (usec)   : 750=0.01%, 1000=1.02%
  lat (msec)   : 2=93.49%, 4=4.86%, 10=0.56%, 20=0.05%, 50=0.01%
  cpu          : usr=1.18%, sys=6.37%, ctx=788426, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2462KiB/s (2521kB/s), 2462KiB/s-2462KiB/s (2521kB/s-2521kB/s), io=1024MiB (1074MB), run=425934-425934msec

Disk stats (read/write):
  sdb: ios=1/542361, merge=0/5505, ticks=0/399396, in_queue=426960, util=99.91%
0 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 2929 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2929 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][99.1%][r=9293KiB/s][r=2323 IOPS][eta 00m:01s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=860: Wed Aug 11 21:07:53 2021
  read: IOPS=2476, BW=9906KiB/s (10.1MB/s)(1024MiB/105848msec)
    clat (usec): min=176, max=6639, avg=393.78, stdev=108.30
     lat (usec): min=177, max=6641, avg=395.64, stdev=108.42
    clat percentiles (usec):
     |  1.00th=[  262],  5.00th=[  281], 10.00th=[  293], 20.00th=[  310],
     | 30.00th=[  322], 40.00th=[  347], 50.00th=[  371], 60.00th=[  400],
     | 70.00th=[  429], 80.00th=[  482], 90.00th=[  545], 95.00th=[  578],
     | 99.00th=[  627], 99.50th=[  652], 99.90th=[  824], 99.95th=[ 1729],
     | 99.99th=[ 2933]
   bw (  KiB/s): min= 7296, max=14200, per=99.94%, avg=9899.75, stdev=1627.59, samples=211
   iops        : min= 1824, max= 3550, avg=2474.92, stdev=406.90, samples=211
  lat (usec)   : 250=0.27%, 500=83.37%, 750=16.25%, 1000=0.03%
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.01%
  cpu          : usr=4.16%, sys=11.88%, ctx=262222, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=9906KiB/s (10.1MB/s), 9906KiB/s-9906KiB/s (10.1MB/s-10.1MB/s), io=1024MiB (1074MB), run=105848-105848msec

Disk stats (read/write):
  sdb: ios=261555/0, merge=0/0, ticks=93995/0, in_queue=105748, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2929 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2930 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 2334 seconds or 38 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5312 MiB / 5696 MiB / 5327 MiB / 32 MiB / 18 MiB ( 5312 MiB / 5696 MiB / 5327 MiB / 32 MiB / 18 MiB )
2 min passed...11008 MiB / 11712 MiB / 11025 MiB / 32 MiB / 18 MiB ( 5696 MiB / 6016 MiB / 5698 MiB / 0 MiB / 0 MiB )
3 min passed...16128 MiB / 17280 MiB / 16151 MiB / 32 MiB / 18 MiB ( 5120 MiB / 5568 MiB / 5126 MiB / 0 MiB / 0 MiB )
4 min passed...20032 MiB / 21504 MiB / 20049 MiB / 32 MiB / 18 MiB ( 3904 MiB / 4224 MiB / 3898 MiB / 0 MiB / 0 MiB )
5 min passed...23936 MiB / 25600 MiB / 23932 MiB / 32 MiB / 18 MiB ( 3904 MiB / 4096 MiB / 3883 MiB / 0 MiB / 0 MiB )
6 min passed...29632 MiB / 31616 MiB / 29652 MiB / 32 MiB / 18 MiB ( 5696 MiB / 6016 MiB / 5720 MiB / 0 MiB / 0 MiB )
7 min passed...35456 MiB / 37824 MiB / 35474 MiB / 32 MiB / 18 MiB ( 5824 MiB / 6208 MiB / 5822 MiB / 0 MiB / 0 MiB )
8 min passed...36128 MiB / 38688 MiB / 36119 MiB / 32 MiB / 19 MiB ( 672 MiB / 864 MiB / 645 MiB / 0 MiB / 1 MiB )
9 min passed...36128 MiB / 38688 MiB / 36122 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
10 min passed...36128 MiB / 38688 MiB / 36122 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...36128 MiB / 38688 MiB / 36122 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...36128 MiB / 38752 MiB / 36126 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
13 min passed...36128 MiB / 38912 MiB / 36126 MiB / 32 MiB / 19 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...36128 MiB / 38912 MiB / 36126 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...36128 MiB / 38912 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
16 min passed...36128 MiB / 38912 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...36128 MiB / 39136 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...36128 MiB / 39168 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...36128 MiB / 39168 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...36128 MiB / 39168 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...36128 MiB / 39232 MiB / 36130 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...36128 MiB / 39296 MiB / 36134 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
23 min passed...36128 MiB / 39296 MiB / 36134 MiB / 576 MiB / 564 MiB ( 0 MiB / 0 MiB / 0 MiB / 544 MiB / 545 MiB )
24 min passed...36128 MiB / 39392 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 96 MiB / 0 MiB / 480 MiB / 479 MiB )
25 min passed...36128 MiB / 39392 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...36128 MiB / 39584 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...36128 MiB / 39584 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...36128 MiB / 39616 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...36128 MiB / 39840 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...36128 MiB / 39936 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...36128 MiB / 40064 MiB / 36134 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...36128 MiB / 40064 MiB / 36138 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
33 min passed...36128 MiB / 40256 MiB / 36138 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...36128 MiB / 40288 MiB / 36145 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 7 MiB / 0 MiB / 0 MiB )
35 min passed...36128 MiB / 40480 MiB / 36145 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...36128 MiB / 40512 MiB / 36145 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...36128 MiB / 40544 MiB / 36145 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...36128 MiB / 40704 MiB / 36145 MiB / 1056 MiB / 1043 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member

Distinguished Member

Famous Member

Distinguished Member

Famous Member

Distinguished Member

Distinguished Member

Distinguished Member