how to best benchmark SSDs?

Round 2 - Run 4 - Test X6: 4x async sequential read/write 32K

Guest Part 1:
Code: 
root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=3): [W(3),f(1)][100.0%][w=390MiB/s][w=12.5k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=685: Thu Aug 12 17:35:24 2021
  write: IOPS=4056, BW=127MiB/s (133MB/s)(3072MiB/24234msec); 0 zone resets
    slat (usec): min=10, max=12206, avg=53.52, stdev=94.77
    clat (usec): min=2, max=42253, avg=164.54, stdev=457.77
     lat (usec): min=64, max=42281, avg=221.16, stdev=467.91
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   69], 20.00th=[   90],
     | 30.00th=[  104], 40.00th=[  119], 50.00th=[  135], 60.00th=[  151],
     | 70.00th=[  174], 80.00th=[  202], 90.00th=[  251], 95.00th=[  306],
     | 99.00th=[  510], 99.50th=[  750], 99.90th=[ 5211], 99.95th=[ 9765],
     | 99.99th=[21890]
   bw (  KiB/s): min=75113, max=176000, per=25.01%, avg=129641.17, stdev=23802.88, samples=48
   iops        : min= 2347, max= 5500, avg=4051.19, stdev=743.86, samples=48
  lat (usec)   : 4=6.28%, 10=0.57%, 20=0.21%, 50=1.14%, 100=18.81%
  lat (usec)   : 250=62.84%, 500=9.09%, 750=0.56%, 1000=0.16%
  lat (msec)   : 2=0.14%, 4=0.07%, 10=0.08%, 20=0.03%, 50=0.01%
  cpu          : usr=7.67%, sys=14.50%, ctx=138861, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=686: Thu Aug 12 17:35:24 2021
  write: IOPS=4080, BW=128MiB/s (134MB/s)(3072MiB/24089msec); 0 zone resets
    slat (usec): min=11, max=16585, avg=53.95, stdev=121.14
    clat (usec): min=2, max=42104, avg=162.96, stdev=448.18
     lat (usec): min=69, max=42125, avg=219.88, stdev=468.05
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   68], 20.00th=[   89],
     | 30.00th=[  103], 40.00th=[  118], 50.00th=[  133], 60.00th=[  151],
     | 70.00th=[  172], 80.00th=[  202], 90.00th=[  251], 95.00th=[  306],
     | 99.00th=[  510], 99.50th=[  725], 99.90th=[ 4883], 99.95th=[ 9503],
     | 99.99th=[21890]
   bw (  KiB/s): min=77888, max=181312, per=25.21%, avg=130678.15, stdev=24656.71, samples=47
   iops        : min= 2434, max= 5666, avg=4083.66, stdev=770.51, samples=47
  lat (usec)   : 4=6.42%, 10=0.53%, 20=0.22%, 50=1.16%, 100=19.55%
  lat (usec)   : 250=62.11%, 500=8.96%, 750=0.59%, 1000=0.15%
  lat (msec)   : 2=0.12%, 4=0.07%, 10=0.08%, 20=0.03%, 50=0.01%
  cpu          : usr=7.50%, sys=14.65%, ctx=139093, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=687: Thu Aug 12 17:35:24 2021
  write: IOPS=4048, BW=127MiB/s (133MB/s)(3072MiB/24279msec); 0 zone resets
    slat (usec): min=10, max=12207, avg=55.25, stdev=91.95
    clat (usec): min=2, max=42057, avg=163.03, stdev=435.05
     lat (usec): min=67, max=42079, avg=221.35, stdev=444.13
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   64], 20.00th=[   88],
     | 30.00th=[  103], 40.00th=[  118], 50.00th=[  135], 60.00th=[  151],
     | 70.00th=[  174], 80.00th=[  202], 90.00th=[  253], 95.00th=[  306],
     | 99.00th=[  519], 99.50th=[  734], 99.90th=[ 4883], 99.95th=[ 8979],
     | 99.99th=[19268]
   bw (  KiB/s): min=72640, max=175168, per=24.96%, avg=129362.00, stdev=24142.82, samples=48
   iops        : min= 2270, max= 5474, avg=4042.52, stdev=754.52, samples=48
  lat (usec)   : 4=6.80%, 10=0.63%, 20=0.20%, 50=1.21%, 100=19.42%
  lat (usec)   : 250=61.38%, 500=9.27%, 750=0.62%, 1000=0.14%
  lat (msec)   : 2=0.14%, 4=0.08%, 10=0.08%, 20=0.03%, 50=0.01%
  cpu          : usr=7.69%, sys=14.31%, ctx=140696, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=688: Thu Aug 12 17:35:24 2021
  write: IOPS=4124, BW=129MiB/s (135MB/s)(3072MiB/23835msec); 0 zone resets
    slat (usec): min=10, max=8929, avg=53.25, stdev=78.52
    clat (usec): min=2, max=41885, avg=161.06, stdev=435.80
     lat (usec): min=66, max=42095, avg=217.36, stdev=439.88
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   63], 20.00th=[   86],
     | 30.00th=[  100], 40.00th=[  115], 50.00th=[  131], 60.00th=[  149],
     | 70.00th=[  172], 80.00th=[  200], 90.00th=[  249], 95.00th=[  306],
     | 99.00th=[  515], 99.50th=[  734], 99.90th=[ 5014], 99.95th=[ 9503],
     | 99.99th=[18220]
   bw (  KiB/s): min=72704, max=177152, per=25.50%, avg=132154.19, stdev=24543.04, samples=47
   iops        : min= 2272, max= 5536, avg=4129.74, stdev=767.00, samples=47
  lat (usec)   : 4=6.75%, 10=0.61%, 20=0.21%, 50=1.18%, 100=21.06%
  lat (usec)   : 250=60.24%, 500=8.89%, 750=0.57%, 1000=0.16%
  lat (msec)   : 2=0.13%, 4=0.07%, 10=0.08%, 20=0.04%, 50=0.01%
  cpu          : usr=7.68%, sys=14.22%, ctx=138657, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=506MiB/s (531MB/s), 127MiB/s-129MiB/s (133MB/s-135MB/s), io=12.0GiB (12.9GB), run=23835-24279msec

Disk stats (read/write):
  sda: ios=1/391794, merge=0/234, ticks=1/66337, in_queue=74404, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 12289 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 1 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 20 MiB ( 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test X6: 4x async sequential read/write 32K

Guest Part 2:
Code: 
async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 3 (f=3): [R(3),_(1)][100.0%][r=235MiB/s][r=7506 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=898: Thu Aug 12 17:51:14 2021
  read: IOPS=1985, BW=62.0MiB/s (65.1MB/s)(3072MiB/49517msec)
    slat (usec): min=9, max=220, avg=30.58, stdev= 9.08
    clat (usec): min=104, max=6167, avg=464.13, stdev=289.98
     lat (usec): min=122, max=6197, avg=496.46, stdev=290.75
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  347], 10.00th=[  363], 20.00th=[  388],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  441],
     | 70.00th=[  453], 80.00th=[  474], 90.00th=[  506], 95.00th=[  545],
     | 99.00th=[ 2409], 99.50th=[ 2835], 99.90th=[ 3228], 99.95th=[ 3392],
     | 99.99th=[ 5342]
   bw (  KiB/s): min=56960, max=143360, per=25.02%, avg=63555.38, stdev=8681.28, samples=98
   iops        : min= 1780, max= 4480, avg=1986.10, stdev=271.29, samples=98
  lat (usec)   : 250=2.35%, 500=86.38%, 750=8.58%, 1000=0.69%
  lat (msec)   : 2=0.69%, 4=1.29%, 10=0.02%
  cpu          : usr=3.59%, sys=8.83%, ctx=98416, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=899: Thu Aug 12 17:51:14 2021
  read: IOPS=1984, BW=62.0MiB/s (65.0MB/s)(3072MiB/49537msec)
    slat (usec): min=7, max=451, avg=30.58, stdev= 9.37
    clat (usec): min=99, max=8346, avg=464.30, stdev=292.76
     lat (usec): min=113, max=8387, avg=496.63, stdev=293.52
    clat percentiles (usec):
     |  1.00th=[  163],  5.00th=[  343], 10.00th=[  363], 20.00th=[  383],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  441],
     | 70.00th=[  453], 80.00th=[  474], 90.00th=[  506], 95.00th=[  553],
     | 99.00th=[ 2474], 99.50th=[ 2868], 99.90th=[ 3163], 99.95th=[ 3326],
     | 99.99th=[ 5407]
   bw (  KiB/s): min=56960, max=124608, per=25.00%, avg=63506.66, stdev=7699.44, samples=98
   iops        : min= 1780, max= 3894, avg=1984.56, stdev=240.60, samples=98
  lat (usec)   : 100=0.01%, 250=2.48%, 500=86.24%, 750=8.57%, 1000=0.67%
  lat (msec)   : 2=0.72%, 4=1.31%, 10=0.02%
  cpu          : usr=3.37%, sys=9.08%, ctx=98448, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=900: Thu Aug 12 17:51:14 2021
  read: IOPS=2008, BW=62.8MiB/s (65.8MB/s)(3072MiB/48933msec)
    slat (usec): min=9, max=251, avg=30.13, stdev= 9.02
    clat (usec): min=71, max=7355, avg=458.61, stdev=290.40
     lat (usec): min=119, max=7384, avg=490.50, stdev=291.18
    clat percentiles (usec):
     |  1.00th=[  153],  5.00th=[  338], 10.00th=[  359], 20.00th=[  379],
     | 30.00th=[  396], 40.00th=[  408], 50.00th=[  420], 60.00th=[  433],
     | 70.00th=[  449], 80.00th=[  469], 90.00th=[  502], 95.00th=[  545],
     | 99.00th=[ 2442], 99.50th=[ 2835], 99.90th=[ 3163], 99.95th=[ 3261],
     | 99.99th=[ 5145]
   bw (  KiB/s): min=56640, max=139648, per=25.29%, avg=64247.41, stdev=8815.41, samples=97
   iops        : min= 1770, max= 4364, avg=2007.72, stdev=275.48, samples=97
  lat (usec)   : 100=0.01%, 250=2.80%, 500=86.85%, 750=7.63%, 1000=0.67%
  lat (msec)   : 2=0.72%, 4=1.31%, 10=0.01%
  cpu          : usr=3.83%, sys=8.63%, ctx=98412, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=901: Thu Aug 12 17:51:14 2021
  read: IOPS=2024, BW=63.3MiB/s (66.3MB/s)(3072MiB/48561msec)
    slat (usec): min=9, max=246, avg=30.49, stdev= 9.10
    clat (usec): min=5, max=6048, avg=454.52, stdev=284.03
     lat (usec): min=113, max=6081, avg=486.75, stdev=284.84
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  302], 10.00th=[  355], 20.00th=[  379],
     | 30.00th=[  396], 40.00th=[  408], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  453], 80.00th=[  469], 90.00th=[  502], 95.00th=[  545],
     | 99.00th=[ 2311], 99.50th=[ 2802], 99.90th=[ 3195], 99.95th=[ 3392],
     | 99.99th=[ 5735]
   bw (  KiB/s): min=57536, max=152832, per=25.50%, avg=64772.74, stdev=11179.78, samples=97
   iops        : min= 1798, max= 4776, avg=2024.12, stdev=349.34, samples=97
  lat (usec)   : 10=0.01%, 100=0.01%, 250=4.30%, 500=85.42%, 750=7.79%
  lat (usec)   : 1000=0.60%
  lat (msec)   : 2=0.68%, 4=1.19%, 10=0.03%
  cpu          : usr=3.86%, sys=8.78%, ctx=98470, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=248MiB/s (260MB/s), 62.0MiB/s-63.3MiB/s (65.0MB/s-66.3MB/s), io=12.0GiB (12.9GB), run=48561-49537msec

Disk stats (read/write):
  sda: ios=392727/0, merge=0/0, ticks=179884/0, in_queue=189996, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 12289 MiB / 12308 MiB ( 0 MiB / 0 MiB )
Script ran 1876 seconds or 31 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...27968 MiB / 29792 MiB / 27998 MiB / 160 MiB / 161 MiB ( 27968 MiB / 29792 MiB / 27998 MiB / 160 MiB / 161 MiB )
2 min passed...27968 MiB / 29888 MiB / 27998 MiB / 160 MiB / 161 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...27968 MiB / 30304 MiB / 28001 MiB / 160 MiB / 161 MiB ( 0 MiB / 416 MiB / 3 MiB / 0 MiB / 0 MiB )
4 min passed...27968 MiB / 30304 MiB / 28003 MiB / 160 MiB / 161 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
5 min passed...27968 MiB / 30336 MiB / 28003 MiB / 160 MiB / 161 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...27968 MiB / 30528 MiB / 28003 MiB / 160 MiB / 161 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...27968 MiB / 31008 MiB / 28005 MiB / 160 MiB / 161 MiB ( 0 MiB / 480 MiB / 2 MiB / 0 MiB / 0 MiB )
8 min passed...27968 MiB / 31072 MiB / 28007 MiB / 160 MiB / 161 MiB ( 0 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...27968 MiB / 31072 MiB / 28007 MiB / 160 MiB / 161 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...27968 MiB / 31200 MiB / 28007 MiB / 160 MiB / 161 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...27968 MiB / 31360 MiB / 28016 MiB / 160 MiB / 161 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...27968 MiB / 32032 MiB / 28016 MiB / 160 MiB / 161 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...27968 MiB / 32064 MiB / 28016 MiB / 160 MiB / 161 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...27968 MiB / 32096 MiB / 28023 MiB / 160 MiB / 161 MiB ( 0 MiB / 32 MiB / 7 MiB / 0 MiB / 0 MiB )
15 min passed...27968 MiB / 32128 MiB / 28023 MiB / 160 MiB / 161 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...27968 MiB / 32768 MiB / 28023 MiB / 9504 MiB / 9542 MiB ( 0 MiB / 640 MiB / 0 MiB / 9344 MiB / 9381 MiB )
17 min passed...27968 MiB / 32864 MiB / 28023 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 96 MiB / 0 MiB / 2528 MiB / 2494 MiB )
18 min passed...27968 MiB / 32864 MiB / 28023 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...27968 MiB / 32960 MiB / 28023 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...27968 MiB / 33120 MiB / 28023 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...27968 MiB / 33792 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 672 MiB / 7 MiB / 0 MiB / 0 MiB )
22 min passed...27968 MiB / 33824 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...27968 MiB / 33920 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...27968 MiB / 33984 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...27968 MiB / 34560 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...27968 MiB / 34848 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...27968 MiB / 34848 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...27968 MiB / 35040 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...27968 MiB / 35232 MiB / 28030 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...27968 MiB / 35616 MiB / 28031 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 384 MiB / 1 MiB / 0 MiB / 0 MiB )
31 min passed...27968 MiB / 35616 MiB / 28037 MiB / 12032 MiB / 12036 MiB ( 0 MiB / 0 MiB / 6 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test X7: 4x sync sequential read/write 32K

Guest Part 1:
Code: 
root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=18.2MiB/s][w=582 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=664: Thu Aug 12 18:33:22 2021
  write: IOPS=145, BW=4666KiB/s (4778kB/s)(512MiB/112360msec); 0 zone resets
    clat (usec): min=4019, max=52753, avg=6831.26, stdev=2120.81
     lat (usec): min=4021, max=52755, avg=6833.63, stdev=2120.79
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5669], 10.00th=[ 5866], 20.00th=[ 6063],
     | 30.00th=[ 6194], 40.00th=[ 6325], 50.00th=[ 6456], 60.00th=[ 6521],
     | 70.00th=[ 6718], 80.00th=[ 6980], 90.00th=[ 8291], 95.00th=[ 8848],
     | 99.00th=[17171], 99.50th=[21890], 99.90th=[33162], 99.95th=[40633],
     | 99.99th=[51643]
   bw (  KiB/s): min= 3136, max= 5504, per=25.00%, avg=4665.75, stdev=422.79, samples=224
   iops        : min=   98, max=  172, avg=145.79, stdev=13.22, samples=224
  lat (msec)   : 10=97.78%, 20=1.54%, 50=0.66%, 100=0.01%
  cpu          : usr=0.56%, sys=1.82%, ctx=50127, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=665: Thu Aug 12 18:33:22 2021
  write: IOPS=145, BW=4666KiB/s (4778kB/s)(512MiB/112357msec); 0 zone resets
    clat (usec): min=4007, max=61222, avg=6831.39, stdev=2109.40
     lat (usec): min=4009, max=61224, avg=6833.74, stdev=2109.50
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5669], 10.00th=[ 5866], 20.00th=[ 6063],
     | 30.00th=[ 6194], 40.00th=[ 6325], 50.00th=[ 6390], 60.00th=[ 6521],
     | 70.00th=[ 6718], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 8848],
     | 99.00th=[17695], 99.50th=[21627], 99.90th=[30016], 99.95th=[40109],
     | 99.99th=[55837]
   bw (  KiB/s): min= 3136, max= 5504, per=25.00%, avg=4665.76, stdev=420.15, samples=224
   iops        : min=   98, max=  172, avg=145.79, stdev=13.13, samples=224
  lat (msec)   : 10=98.27%, 20=1.04%, 50=0.67%, 100=0.02%
  cpu          : usr=0.51%, sys=1.86%, ctx=51205, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=666: Thu Aug 12 18:33:22 2021
  write: IOPS=145, BW=4666KiB/s (4778kB/s)(512MiB/112356msec); 0 zone resets
    clat (usec): min=3996, max=61182, avg=6831.25, stdev=2102.17
     lat (usec): min=3998, max=61184, avg=6833.58, stdev=2102.14
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5669], 10.00th=[ 5866], 20.00th=[ 6063],
     | 30.00th=[ 6194], 40.00th=[ 6325], 50.00th=[ 6390], 60.00th=[ 6521],
     | 70.00th=[ 6718], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 8848],
     | 99.00th=[17695], 99.50th=[21627], 99.90th=[30016], 99.95th=[38011],
     | 99.99th=[55837]
   bw (  KiB/s): min= 3136, max= 5504, per=25.00%, avg=4665.75, stdev=420.08, samples=224
   iops        : min=   98, max=  172, avg=145.79, stdev=13.13, samples=224
  lat (msec)   : 4=0.01%, 10=98.30%, 20=1.01%, 50=0.65%, 100=0.03%
  cpu          : usr=0.49%, sys=1.89%, ctx=51169, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=667: Thu Aug 12 18:33:22 2021
  write: IOPS=145, BW=4666KiB/s (4778kB/s)(512MiB/112360msec); 0 zone resets
    clat (usec): min=4015, max=52853, avg=6831.82, stdev=2121.50
     lat (usec): min=4016, max=52855, avg=6834.16, stdev=2121.47
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5669], 10.00th=[ 5866], 20.00th=[ 6063],
     | 30.00th=[ 6194], 40.00th=[ 6325], 50.00th=[ 6456], 60.00th=[ 6521],
     | 70.00th=[ 6718], 80.00th=[ 6980], 90.00th=[ 8291], 95.00th=[ 8848],
     | 99.00th=[17171], 99.50th=[21890], 99.90th=[33162], 99.95th=[40633],
     | 99.99th=[51643]
   bw (  KiB/s): min= 3136, max= 5504, per=25.00%, avg=4665.71, stdev=422.80, samples=224
   iops        : min=   98, max=  172, avg=145.78, stdev=13.22, samples=224
  lat (msec)   : 10=97.78%, 20=1.53%, 50=0.67%, 100=0.01%
  cpu          : usr=0.49%, sys=1.81%, ctx=49249, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=18.2MiB/s (19.1MB/s), 4666KiB/s-4666KiB/s (4778kB/s-4778kB/s), io=2048MiB (2147MB), run=112356-112360msec

Disk stats (read/write):
  sda: ios=0/130887, merge=0/114509, ticks=0/123523, in_queue=133500, util=99.82%
0 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 2753 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2753 MiB / 19 MiB ( 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test X7: 4x sync sequential read/write 32K

Guest Part 2:
Code: 
sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=269MiB/s][r=8619 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=872: Thu Aug 12 18:48:30 2021
  read: IOPS=2179, BW=68.1MiB/s (71.4MB/s)(512MiB/7518msec)
    clat (usec): min=294, max=1909, avg=450.30, stdev=51.53
     lat (usec): min=295, max=1911, avg=451.96, stdev=51.61
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  379], 10.00th=[  392], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  445], 60.00th=[  457],
     | 70.00th=[  469], 80.00th=[  486], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  734], 99.95th=[  865],
     | 99.99th=[ 1500]
   bw (  KiB/s): min=67904, max=72768, per=25.14%, avg=69751.47, stdev=1497.00, samples=15
   iops        : min= 2122, max= 2274, avg=2179.73, stdev=46.88, samples=15
  lat (usec)   : 500=85.88%, 750=14.03%, 1000=0.05%
  lat (msec)   : 2=0.04%
  cpu          : usr=2.94%, sys=8.97%, ctx=16393, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=873: Thu Aug 12 18:48:30 2021
  read: IOPS=2174, BW=67.0MiB/s (71.3MB/s)(512MiB/7533msec)
    clat (usec): min=313, max=2801, avg=451.19, stdev=54.70
     lat (usec): min=314, max=2803, avg=452.83, stdev=54.77
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  379], 10.00th=[  392], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  461],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  515], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  742], 99.95th=[  889],
     | 99.99th=[ 1500]
   bw (  KiB/s): min=67456, max=73664, per=25.09%, avg=69610.67, stdev=1745.63, samples=15
   iops        : min= 2108, max= 2302, avg=2175.33, stdev=54.55, samples=15
  lat (usec)   : 500=85.40%, 750=14.51%, 1000=0.05%
  lat (msec)   : 2=0.02%, 4=0.01%
  cpu          : usr=3.05%, sys=8.87%, ctx=16388, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=874: Thu Aug 12 18:48:30 2021
  read: IOPS=2170, BW=67.8MiB/s (71.1MB/s)(512MiB/7547msec)
    clat (usec): min=300, max=2893, avg=452.15, stdev=60.27
     lat (usec): min=302, max=2895, avg=453.79, stdev=60.35
    clat percentiles (usec):
     |  1.00th=[  359],  5.00th=[  379], 10.00th=[  392], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  461],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  515], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  734], 99.95th=[ 1020],
     | 99.99th=[ 2573]
   bw (  KiB/s): min=67776, max=72320, per=25.04%, avg=69482.67, stdev=1405.43, samples=15
   iops        : min= 2118, max= 2260, avg=2171.33, stdev=43.92, samples=15
  lat (usec)   : 500=85.03%, 750=14.87%, 1000=0.04%
  lat (msec)   : 2=0.03%, 4=0.02%
  cpu          : usr=3.38%, sys=8.68%, ctx=16388, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=875: Thu Aug 12 18:48:30 2021
  read: IOPS=2167, BW=67.7MiB/s (71.0MB/s)(512MiB/7559msec)
    clat (usec): min=303, max=2743, avg=452.78, stdev=57.02
     lat (usec): min=304, max=2745, avg=454.45, stdev=57.11
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  379], 10.00th=[  392], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  461],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  515], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  742], 99.95th=[  881],
     | 99.99th=[ 2606]
   bw (  KiB/s): min=67328, max=72896, per=25.00%, avg=69367.47, stdev=1754.02, samples=15
   iops        : min= 2104, max= 2278, avg=2167.73, stdev=54.81, samples=15
  lat (usec)   : 500=84.32%, 750=15.58%, 1000=0.07%
  lat (msec)   : 2=0.02%, 4=0.01%
  cpu          : usr=3.45%, sys=8.83%, ctx=16389, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=271MiB/s (284MB/s), 67.7MiB/s-68.1MiB/s (71.0MB/s-71.4MB/s), io=2048MiB (2147MB), run=7518-7559msec

Disk stats (read/write):
  sda: ios=64323/0, merge=0/0, ticks=26966/0, in_queue=28600, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2753 MiB / 2067 MiB ( 0 MiB / 0 MiB )
Script ran 1921 seconds or 32 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...10944 MiB / 11936 MiB / 10982 MiB / 160 MiB / 150 MiB ( 10944 MiB / 11936 MiB / 10982 MiB / 160 MiB / 150 MiB )
2 min passed...21312 MiB / 23040 MiB / 21349 MiB / 256 MiB / 273 MiB ( 10368 MiB / 11104 MiB / 10367 MiB / 96 MiB / 123 MiB )
3 min passed...21312 MiB / 23168 MiB / 21351 MiB / 256 MiB / 273 MiB ( 0 MiB / 128 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...21312 MiB / 23296 MiB / 21358 MiB / 256 MiB / 273 MiB ( 0 MiB / 128 MiB / 7 MiB / 0 MiB / 0 MiB )
5 min passed...21312 MiB / 23872 MiB / 21364 MiB / 256 MiB / 273 MiB ( 0 MiB / 576 MiB / 6 MiB / 0 MiB / 0 MiB )
6 min passed...21312 MiB / 23904 MiB / 21364 MiB / 256 MiB / 273 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...21312 MiB / 23968 MiB / 21364 MiB / 256 MiB / 273 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...21312 MiB / 24000 MiB / 21366 MiB / 256 MiB / 273 MiB ( 0 MiB / 32 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...21312 MiB / 24544 MiB / 21368 MiB / 256 MiB / 273 MiB ( 0 MiB / 544 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...21312 MiB / 24864 MiB / 21368 MiB / 256 MiB / 273 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...21312 MiB / 24864 MiB / 21368 MiB / 256 MiB / 273 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...21312 MiB / 25024 MiB / 21368 MiB / 256 MiB / 273 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...21312 MiB / 25152 MiB / 21368 MiB / 256 MiB / 273 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...21312 MiB / 25760 MiB / 21370 MiB / 256 MiB / 273 MiB ( 0 MiB / 608 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...21376 MiB / 25792 MiB / 21378 MiB / 256 MiB / 273 MiB ( 64 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
16 min passed...21376 MiB / 25856 MiB / 21382 MiB / 256 MiB / 273 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
17 min passed...21376 MiB / 25984 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 128 MiB / 4 MiB / 2080 MiB / 2048 MiB )
18 min passed...21376 MiB / 26336 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...21376 MiB / 26368 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...21376 MiB / 26368 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...21376 MiB / 26464 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...21376 MiB / 26624 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...21376 MiB / 26976 MiB / 21386 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...21376 MiB / 26976 MiB / 21389 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
25 min passed...21408 MiB / 27040 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 32 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
26 min passed...21408 MiB / 27136 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...21408 MiB / 27712 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...21408 MiB / 28000 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...21408 MiB / 28000 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...21408 MiB / 28128 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...21408 MiB / 28384 MiB / 21393 MiB / 2336 MiB / 2321 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...21440 MiB / 28800 MiB / 21432 MiB / 2336 MiB / 2321 MiB ( 32 MiB / 416 MiB / 39 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 0: just idleing for 1 hour

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_0.sh
0 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
16 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
17 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
18 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
19 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
20 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
21 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
22 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
23 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
24 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
25 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
26 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 1 MiB / 0 MiB )
27 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
28 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
29 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
30 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
31 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
32 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
33 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
34 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
35 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
36 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
37 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
38 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
39 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
40 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
41 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
42 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
43 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
44 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
45 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
46 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
47 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
48 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
49 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
50 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
51 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
52 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
53 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
54 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
55 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
56 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
57 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
58 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
59 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
60 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
Script ran 3601 seconds or 60 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
2 min passed...0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...0 MiB / 864 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...0 MiB / 864 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...0 MiB / 960 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...0 MiB / 992 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 4 MiB / 0 MiB / 0 MiB )
7 min passed...0 MiB / 1504 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...0 MiB / 1504 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
9 min passed...0 MiB / 1504 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...0 MiB / 1600 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...0 MiB / 1856 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...0 MiB / 1856 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...0 MiB / 1856 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...0 MiB / 1984 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...0 MiB / 2048 MiB / 8 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...0 MiB / 2624 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 576 MiB / 5 MiB / 0 MiB / 0 MiB )
17 min passed...0 MiB / 2624 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...0 MiB / 2656 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...0 MiB / 2752 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...0 MiB / 3168 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...0 MiB / 3424 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...0 MiB / 3424 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...0 MiB / 3584 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...0 MiB / 3680 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...0 MiB / 4192 MiB / 13 MiB / 0 MiB / 0 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...0 MiB / 4256 MiB / 17 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
27 min passed...0 MiB / 4288 MiB / 17 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...0 MiB / 4416 MiB / 17 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...0 MiB / 4896 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 480 MiB / 3 MiB / 0 MiB / 0 MiB )
30 min passed...0 MiB / 5120 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...0 MiB / 5120 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...0 MiB / 5312 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...0 MiB / 5440 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...0 MiB / 6016 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...0 MiB / 6048 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...0 MiB / 6112 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 4 MiB / 0 MiB / 0 MiB )
37 min passed...0 MiB / 6272 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...0 MiB / 6656 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...0 MiB / 6656 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...0 MiB / 6656 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...0 MiB / 6720 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
42 min passed...0 MiB / 6912 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...0 MiB / 7296 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...0 MiB / 7296 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...0 MiB / 7360 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...0 MiB / 7648 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 288 MiB / 4 MiB / 0 MiB / 0 MiB )
47 min passed...0 MiB / 8160 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
48 min passed...0 MiB / 8384 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
49 min passed...0 MiB / 8384 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
50 min passed...0 MiB / 8576 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
51 min passed...0 MiB / 8768 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
52 min passed...0 MiB / 9024 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
53 min passed...0 MiB / 9024 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
54 min passed...0 MiB / 9120 MiB / 28 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
55 min passed...0 MiB / 9344 MiB / 31 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 3 MiB / 0 MiB / 0 MiB )
56 min passed...0 MiB / 9696 MiB / 33 MiB / 0 MiB / 0 MiB ( 0 MiB / 352 MiB / 2 MiB / 0 MiB / 0 MiB )
57 min passed...0 MiB / 9696 MiB / 33 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
58 min passed...0 MiB / 9696 MiB / 33 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
59 min passed...0 MiB / 9728 MiB / 33 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
60 min passed...0 MiB / 9888 MiB / 33 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Last edited:
Round 2 - Run 3 - Test X6: 4x async sequential read/write 32K

Guest Part 1:
Code: 
root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=4): [W(4)][97.5%][w=452MiB/s][w=14.5k IOPS][eta 00m:01s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=645: Thu Aug 12 20:34:24 2021
  write: IOPS=2484, BW=77.6MiB/s (81.4MB/s)(3072MiB/39564msec); 0 zone resets
    slat (usec): min=11, max=4062, avg=55.78, stdev=83.73
    clat (usec): min=2, max=3438.9k, avg=318.35, stdev=15937.01
     lat (usec): min=88, max=3438.9k, avg=377.50, stdev=15936.97
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   92], 20.00th=[  115],
     | 30.00th=[  131], 40.00th=[  147], 50.00th=[  165], 60.00th=[  190],
     | 70.00th=[  225], 80.00th=[  273], 90.00th=[  371], 95.00th=[  486],
     | 99.00th=[  947], 99.50th=[ 1532], 99.90th=[ 4948], 99.95th=[ 6849],
     | 99.99th=[17695]
   bw (  KiB/s): min= 1152, max=159872, per=30.35%, avg=96515.02, stdev=31840.81, samples=65
   iops        : min=   36, max= 4996, avg=3016.05, stdev=995.00, samples=65
  lat (usec)   : 4=5.02%, 10=0.28%, 20=0.13%, 50=1.04%, 100=6.51%
  lat (usec)   : 250=62.77%, 500=19.57%, 750=3.00%, 1000=0.78%
  lat (msec)   : 2=0.53%, 4=0.23%, 10=0.11%, 20=0.01%, 50=0.01%
  lat (msec)   : 500=0.01%
  cpu          : usr=4.73%, sys=9.57%, ctx=139554, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=646: Thu Aug 12 20:34:24 2021
  write: IOPS=2491, BW=77.8MiB/s (81.6MB/s)(3072MiB/39462msec); 0 zone resets
    slat (usec): min=11, max=6179, avg=55.17, stdev=90.43
    clat (usec): min=2, max=3438.7k, avg=318.38, stdev=15937.17
     lat (usec): min=89, max=3438.8k, avg=376.54, stdev=15937.19
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    6], 10.00th=[   94], 20.00th=[  116],
     | 30.00th=[  133], 40.00th=[  147], 50.00th=[  165], 60.00th=[  190],
     | 70.00th=[  225], 80.00th=[  273], 90.00th=[  371], 95.00th=[  490],
     | 99.00th=[  947], 99.50th=[ 1500], 99.90th=[ 4817], 99.95th=[ 6652],
     | 99.99th=[17695]
   bw (  KiB/s): min= 1216, max=162432, per=30.36%, avg=96545.88, stdev=32446.90, samples=64
   iops        : min=   38, max= 5076, avg=3017.03, stdev=1013.96, samples=64
  lat (usec)   : 4=4.74%, 10=0.30%, 20=0.12%, 50=1.00%, 100=6.29%
  lat (usec)   : 250=63.28%, 500=19.54%, 750=3.07%, 1000=0.76%
  lat (msec)   : 2=0.54%, 4=0.23%, 10=0.11%, 20=0.01%, 50=0.01%
  lat (msec)   : 500=0.01%
  cpu          : usr=4.55%, sys=9.84%, ctx=137709, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=647: Thu Aug 12 20:34:24 2021
  write: IOPS=2508, BW=78.4MiB/s (82.2MB/s)(3072MiB/39186msec); 0 zone resets
    slat (usec): min=11, max=6164, avg=55.42, stdev=89.15
    clat (usec): min=2, max=3437.9k, avg=315.61, stdev=15935.18
     lat (usec): min=86, max=3437.0k, avg=374.11, stdev=15935.14
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   91], 20.00th=[  114],
     | 30.00th=[  130], 40.00th=[  147], 50.00th=[  165], 60.00th=[  190],
     | 70.00th=[  223], 80.00th=[  269], 90.00th=[  367], 95.00th=[  482],
     | 99.00th=[  938], 99.50th=[ 1500], 99.90th=[ 4883], 99.95th=[ 6849],
     | 99.99th=[17695]
   bw (  KiB/s): min= 1216, max=159168, per=30.65%, avg=97478.50, stdev=32343.75, samples=64
   iops        : min=   38, max= 4974, avg=3046.20, stdev=1010.74, samples=64
  lat (usec)   : 4=5.05%, 10=0.23%, 20=0.13%, 50=0.96%, 100=7.04%
  lat (usec)   : 250=62.98%, 500=19.10%, 750=2.90%, 1000=0.72%
  lat (msec)   : 2=0.53%, 4=0.22%, 10=0.11%, 20=0.01%, 50=0.01%
  lat (msec)   : 500=0.01%
  cpu          : usr=5.10%, sys=9.21%, ctx=136748, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=648: Thu Aug 12 20:34:24 2021
  write: IOPS=2487, BW=77.7MiB/s (81.5MB/s)(3072MiB/39527msec); 0 zone resets
    slat (usec): min=11, max=7434, avg=55.69, stdev=96.87
    clat (usec): min=2, max=3437.9k, avg=318.57, stdev=15934.97
     lat (usec): min=89, max=3437.9k, avg=377.27, stdev=15934.97
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    9], 10.00th=[   94], 20.00th=[  116],
     | 30.00th=[  133], 40.00th=[  149], 50.00th=[  167], 60.00th=[  190],
     | 70.00th=[  225], 80.00th=[  273], 90.00th=[  371], 95.00th=[  482],
     | 99.00th=[  938], 99.50th=[ 1483], 99.90th=[ 4948], 99.95th=[ 7177],
     | 99.99th=[16057]
   bw (  KiB/s): min= 1280, max=164928, per=30.40%, avg=96670.40, stdev=32787.95, samples=65
   iops        : min=   40, max= 5154, avg=3020.88, stdev=1024.61, samples=65
  lat (usec)   : 4=4.74%, 10=0.27%, 20=0.12%, 50=0.94%, 100=6.41%
  lat (usec)   : 250=63.27%, 500=19.67%, 750=2.93%, 1000=0.76%
  lat (msec)   : 2=0.53%, 4=0.23%, 10=0.11%, 20=0.02%, 50=0.01%
  lat (msec)   : 500=0.01%
  cpu          : usr=4.83%, sys=9.51%, ctx=137367, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=311MiB/s (326MB/s), 77.6MiB/s-78.4MiB/s (81.4MB/s-82.2MB/s), io=12.0GiB (12.9GB), run=39186-39564msec

Disk stats (read/write):
  sda: ios=0/391414, merge=0/215, ticks=0/128459, in_queue=110300, util=77.81%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 12289 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 19 MiB ( 0 MiB / 0 MiB )
 
Round 2 - Run 3 - Test X6: 4x async sequential read/write 32K

Guest Part 2:
Code: 
async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=253MiB/s][r=8095 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=857: Thu Aug 12 20:50:15 2021
  read: IOPS=1952, BW=61.0MiB/s (63.0MB/s)(3072MiB/50359msec)
    slat (usec): min=9, max=251, avg=26.85, stdev= 8.37
    clat (usec): min=187, max=12484, avg=477.47, stdev=185.92
     lat (usec): min=213, max=12504, avg=505.93, stdev=186.21
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  383], 10.00th=[  396], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  469],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  537], 95.00th=[  570],
     | 99.00th=[  848], 99.50th=[ 2057], 99.90th=[ 2966], 99.95th=[ 3064],
     | 99.99th=[ 3523]
   bw (  KiB/s): min=57280, max=65600, per=24.99%, avg=62445.01, stdev=2562.30, samples=100
   iops        : min= 1790, max= 2050, avg=1951.38, stdev=80.07, samples=100
  lat (usec)   : 250=0.03%, 500=77.29%, 750=21.50%, 1000=0.37%
  lat (msec)   : 2=0.30%, 4=0.50%, 10=0.01%, 20=0.01%
  cpu          : usr=3.29%, sys=7.53%, ctx=98329, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=858: Thu Aug 12 20:50:15 2021
  read: IOPS=1957, BW=61.2MiB/s (64.1MB/s)(3072MiB/50227msec)
    slat (usec): min=9, max=252, avg=26.82, stdev= 8.36
    clat (usec): min=155, max=12493, avg=476.14, stdev=184.38
     lat (usec): min=173, max=12516, avg=504.57, stdev=184.63
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  379], 10.00th=[  396], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  469],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  537], 95.00th=[  570],
     | 99.00th=[  848], 99.50th=[ 2057], 99.90th=[ 2900], 99.95th=[ 3032],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=57280, max=73728, per=25.06%, avg=62610.19, stdev=2885.03, samples=100
   iops        : min= 1790, max= 2304, avg=1956.58, stdev=90.14, samples=100
  lat (usec)   : 250=0.49%, 500=76.75%, 750=21.60%, 1000=0.34%
  lat (msec)   : 2=0.31%, 4=0.51%, 10=0.01%, 20=0.01%
  cpu          : usr=2.98%, sys=7.91%, ctx=98361, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=859: Thu Aug 12 20:50:15 2021
  read: IOPS=1956, BW=61.2MiB/s (64.1MB/s)(3072MiB/50236msec)
    slat (usec): min=9, max=383, avg=26.90, stdev= 8.72
    clat (usec): min=4, max=12444, avg=476.17, stdev=184.48
     lat (usec): min=185, max=12468, avg=504.69, stdev=184.84
    clat percentiles (usec):
     |  1.00th=[  347],  5.00th=[  379], 10.00th=[  396], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  474],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  537], 95.00th=[  570],
     | 99.00th=[  816], 99.50th=[ 1942], 99.90th=[ 2900], 99.95th=[ 3032],
     | 99.99th=[ 3490]
   bw (  KiB/s): min=58176, max=70400, per=25.06%, avg=62603.66, stdev=2688.18, samples=100
   iops        : min= 1818, max= 2200, avg=1956.34, stdev=84.00, samples=100
  lat (usec)   : 10=0.01%, 250=0.40%, 500=76.69%, 750=21.76%, 1000=0.35%
  lat (msec)   : 2=0.30%, 4=0.48%, 10=0.01%, 20=0.01%
  cpu          : usr=3.21%, sys=7.72%, ctx=98356, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=860: Thu Aug 12 20:50:15 2021
  read: IOPS=1952, BW=61.0MiB/s (63.0MB/s)(3072MiB/50341msec)
    slat (usec): min=9, max=252, avg=27.03, stdev= 8.63
    clat (usec): min=189, max=12479, avg=476.96, stdev=185.52
     lat (usec): min=213, max=12504, avg=505.70, stdev=185.88
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  383], 10.00th=[  396], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  469],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  537], 95.00th=[  570],
     | 99.00th=[  857], 99.50th=[ 2073], 99.90th=[ 2933], 99.95th=[ 3032],
     | 99.99th=[ 3556]
   bw (  KiB/s): min=56000, max=65920, per=25.00%, avg=62465.42, stdev=2592.21, samples=100
   iops        : min= 1750, max= 2060, avg=1952.02, stdev=81.00, samples=100
  lat (usec)   : 250=0.02%, 500=77.52%, 750=21.28%, 1000=0.35%
  lat (msec)   : 2=0.30%, 4=0.51%, 10=0.01%, 20=0.01%
  cpu          : usr=2.90%, sys=7.98%, ctx=98419, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=244MiB/s (256MB/s), 61.0MiB/s-61.2MiB/s (63.0MB/s-64.1MB/s), io=12.0GiB (12.9GB), run=50227-50359msec

Disk stats (read/write):
  sda: ios=392038/2, merge=0/1, ticks=186274/2, in_queue=195892, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 12289 MiB / 12307 MiB ( 0 MiB / 0 MiB )
Script ran 1892 seconds or 31 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...26272 MiB / 27968 MiB / 26277 MiB / 352 MiB / 356 MiB ( 26272 MiB / 27968 MiB / 26277 MiB / 352 MiB / 356 MiB )
2 min passed...26272 MiB / 28096 MiB / 26285 MiB / 352 MiB / 356 MiB ( 0 MiB / 128 MiB / 8 MiB / 0 MiB / 0 MiB )
3 min passed...26272 MiB / 28096 MiB / 26287 MiB / 352 MiB / 356 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...26272 MiB / 28128 MiB / 26291 MiB / 352 MiB / 356 MiB ( 0 MiB / 32 MiB / 4 MiB / 0 MiB / 0 MiB )
5 min passed...26272 MiB / 28384 MiB / 26291 MiB / 352 MiB / 356 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...26272 MiB / 28480 MiB / 26291 MiB / 352 MiB / 356 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...26272 MiB / 28704 MiB / 26291 MiB / 352 MiB / 356 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...26272 MiB / 28704 MiB / 26293 MiB / 352 MiB / 356 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...26272 MiB / 28768 MiB / 26295 MiB / 352 MiB / 356 MiB ( 0 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...26272 MiB / 29056 MiB / 26295 MiB / 352 MiB / 356 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...26272 MiB / 29120 MiB / 26295 MiB / 352 MiB / 356 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...26304 MiB / 29248 MiB / 26301 MiB / 352 MiB / 356 MiB ( 32 MiB / 128 MiB / 6 MiB / 0 MiB / 0 MiB )
13 min passed...26304 MiB / 29248 MiB / 26301 MiB / 352 MiB / 356 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...26304 MiB / 29376 MiB / 26301 MiB / 352 MiB / 356 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...26304 MiB / 29568 MiB / 26308 MiB / 352 MiB / 356 MiB ( 0 MiB / 192 MiB / 7 MiB / 0 MiB / 0 MiB )
16 min passed...26304 MiB / 29664 MiB / 26310 MiB / 5216 MiB / 5227 MiB ( 0 MiB / 96 MiB / 2 MiB / 4864 MiB / 4871 MiB )
17 min passed...26304 MiB / 29728 MiB / 26311 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 64 MiB / 1 MiB / 7424 MiB / 7402 MiB )
18 min passed...26304 MiB / 29728 MiB / 26311 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...26304 MiB / 29888 MiB / 26311 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...26304 MiB / 30080 MiB / 26311 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...26304 MiB / 30144 MiB / 26311 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...26304 MiB / 30240 MiB / 26317 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 96 MiB / 6 MiB / 0 MiB / 0 MiB )
23 min passed...26304 MiB / 30240 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
24 min passed...26304 MiB / 30272 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...26304 MiB / 30272 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...26304 MiB / 30368 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...26304 MiB / 30592 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...26304 MiB / 30592 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...26304 MiB / 30816 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...26304 MiB / 30816 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...26304 MiB / 30880 MiB / 26318 MiB / 12640 MiB / 12629 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 3 - Test X7: 4x sync sequential read/write 32K

Guest Part 1:
Code: 
root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=20.5MiB/s][w=656 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=624: Thu Aug 12 21:36:06 2021
  write: IOPS=184, BW=5911KiB/s (6053kB/s)(512MiB/88698msec); 0 zone resets
    clat (usec): min=3068, max=38861, avg=5387.80, stdev=1745.18
     lat (usec): min=3069, max=38862, avg=5390.08, stdev=1745.08
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4817], 40.00th=[ 4948], 50.00th=[ 5014], 60.00th=[ 5145],
     | 70.00th=[ 5211], 80.00th=[ 5342], 90.00th=[ 5538], 95.00th=[ 9372],
     | 99.00th=[13435], 99.50th=[15008], 99.90th=[19530], 99.95th=[23200],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2560, max= 7680, per=24.98%, avg=5906.80, stdev=1044.81, samples=177
   iops        : min=   80, max=  240, avg=184.51, stdev=32.65, samples=177
  lat (msec)   : 4=0.71%, 10=94.94%, 20=4.25%, 50=0.10%
  cpu          : usr=0.68%, sys=2.39%, ctx=54603, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=625: Thu Aug 12 21:36:06 2021
  write: IOPS=184, BW=5911KiB/s (6053kB/s)(512MiB/88693msec); 0 zone resets
    clat (usec): min=3135, max=37461, avg=5387.21, stdev=1748.92
     lat (usec): min=3137, max=37464, avg=5389.55, stdev=1748.90
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4817], 40.00th=[ 4948], 50.00th=[ 5080], 60.00th=[ 5145],
     | 70.00th=[ 5211], 80.00th=[ 5342], 90.00th=[ 5538], 95.00th=[ 9372],
     | 99.00th=[13435], 99.50th=[15139], 99.90th=[20317], 99.95th=[22414],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2560, max= 7680, per=24.99%, avg=5908.18, stdev=1046.80, samples=177
   iops        : min=   80, max=  240, avg=184.61, stdev=32.71, samples=177
  lat (msec)   : 4=0.69%, 10=95.05%, 20=4.15%, 50=0.11%
  cpu          : usr=0.75%, sys=2.51%, ctx=50943, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=626: Thu Aug 12 21:36:06 2021
  write: IOPS=184, BW=5911KiB/s (6053kB/s)(512MiB/88698msec); 0 zone resets
    clat (usec): min=2884, max=34499, avg=5387.95, stdev=1735.79
     lat (usec): min=2886, max=34501, avg=5390.22, stdev=1735.74
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4817], 40.00th=[ 4948], 50.00th=[ 5080], 60.00th=[ 5145],
     | 70.00th=[ 5211], 80.00th=[ 5342], 90.00th=[ 5538], 95.00th=[ 9372],
     | 99.00th=[13435], 99.50th=[15008], 99.90th=[20055], 99.95th=[22414],
     | 99.99th=[32375]
   bw (  KiB/s): min= 2560, max= 7680, per=24.99%, avg=5908.06, stdev=1046.79, samples=177
   iops        : min=   80, max=  240, avg=184.62, stdev=32.72, samples=177
  lat (msec)   : 4=0.75%, 10=94.96%, 20=4.19%, 50=0.10%
  cpu          : usr=0.67%, sys=2.40%, ctx=55675, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=627: Thu Aug 12 21:36:06 2021
  write: IOPS=184, BW=5911KiB/s (6053kB/s)(512MiB/88695msec); 0 zone resets
    clat (usec): min=3085, max=39233, avg=5387.13, stdev=1742.90
     lat (usec): min=3086, max=39235, avg=5389.43, stdev=1742.88
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4817], 40.00th=[ 4948], 50.00th=[ 5014], 60.00th=[ 5145],
     | 70.00th=[ 5211], 80.00th=[ 5342], 90.00th=[ 5538], 95.00th=[ 9372],
     | 99.00th=[13304], 99.50th=[14877], 99.90th=[19530], 99.95th=[22938],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2560, max= 7616, per=24.98%, avg=5906.54, stdev=1045.20, samples=177
   iops        : min=   80, max=  238, avg=184.49, stdev=32.68, samples=177
  lat (msec)   : 4=0.72%, 10=94.91%, 20=4.28%, 50=0.09%
  cpu          : usr=0.66%, sys=2.55%, ctx=49265, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=23.1MiB/s (24.2MB/s), 5911KiB/s-5911KiB/s (6053kB/s-6053kB/s), io=2048MiB (2147MB), run=88693-88698msec

Disk stats (read/write):
  sda: ios=1/131063, merge=0/112491, ticks=1/102522, in_queue=108656, util=99.80%
0 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 2744 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2744 MiB / 19 MiB ( 0 MiB / 0 MiB )
 
Round 2 - Run 3 - Test X7: 4x sync sequential read/write 32K

Guest Part 2:
Code: 
sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=255MiB/s][r=8172 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=833: Thu Aug 12 21:51:15 2021
  read: IOPS=2023, BW=63.2MiB/s (66.3MB/s)(512MiB/8098msec)
    clat (usec): min=334, max=3103, avg=486.62, stdev=62.65
     lat (usec): min=335, max=3105, avg=488.13, stdev=62.70
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  408], 10.00th=[  424], 20.00th=[  445],
     | 30.00th=[  457], 40.00th=[  469], 50.00th=[  482], 60.00th=[  494],
     | 70.00th=[  510], 80.00th=[  529], 90.00th=[  553], 95.00th=[  578],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[  766], 99.95th=[  824],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=64000, max=65536, per=24.98%, avg=64692.00, stdev=439.21, samples=16
   iops        : min= 2000, max= 2048, avg=2021.62, stdev=13.73, samples=16
  lat (usec)   : 500=64.28%, 750=35.60%, 1000=0.09%
  lat (msec)   : 2=0.01%, 4=0.02%
  cpu          : usr=2.68%, sys=7.14%, ctx=16388, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=834: Thu Aug 12 21:51:15 2021
  read: IOPS=2027, BW=63.4MiB/s (66.4MB/s)(512MiB/8081msec)
    clat (usec): min=343, max=3181, avg=485.66, stdev=64.47
     lat (usec): min=344, max=3182, avg=487.14, stdev=64.53
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  408], 10.00th=[  424], 20.00th=[  441],
     | 30.00th=[  457], 40.00th=[  469], 50.00th=[  482], 60.00th=[  494],
     | 70.00th=[  506], 80.00th=[  529], 90.00th=[  553], 95.00th=[  578],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[  775], 99.95th=[  873],
     | 99.99th=[ 3064]
   bw (  KiB/s): min=64192, max=65536, per=25.04%, avg=64852.00, stdev=365.58, samples=16
   iops        : min= 2006, max= 2048, avg=2026.62, stdev=11.42, samples=16
  lat (usec)   : 500=64.94%, 750=34.92%, 1000=0.09%
  lat (msec)   : 2=0.03%, 4=0.02%
  cpu          : usr=2.34%, sys=7.65%, ctx=16388, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=835: Thu Aug 12 21:51:15 2021
  read: IOPS=2031, BW=63.5MiB/s (66.6MB/s)(512MiB/8065msec)
    clat (usec): min=338, max=3083, avg=484.66, stdev=64.14
     lat (usec): min=340, max=3085, avg=486.15, stdev=64.19
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  408], 10.00th=[  420], 20.00th=[  441],
     | 30.00th=[  457], 40.00th=[  469], 50.00th=[  482], 60.00th=[  494],
     | 70.00th=[  506], 80.00th=[  529], 90.00th=[  553], 95.00th=[  578],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[  775], 99.95th=[  898],
     | 99.99th=[ 3064]
   bw (  KiB/s): min=64192, max=65984, per=25.10%, avg=65012.00, stdev=471.01, samples=16
   iops        : min= 2006, max= 2062, avg=2031.62, stdev=14.72, samples=16
  lat (usec)   : 500=65.81%, 750=34.06%, 1000=0.09%
  lat (msec)   : 2=0.02%, 4=0.02%
  cpu          : usr=2.34%, sys=7.53%, ctx=16396, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=836: Thu Aug 12 21:51:15 2021
  read: IOPS=2031, BW=63.5MiB/s (66.6MB/s)(512MiB/8066msec)
    clat (usec): min=325, max=3100, avg=484.70, stdev=65.21
     lat (usec): min=326, max=3101, avg=486.19, stdev=65.26
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  408], 10.00th=[  424], 20.00th=[  441],
     | 30.00th=[  457], 40.00th=[  469], 50.00th=[  482], 60.00th=[  494],
     | 70.00th=[  506], 80.00th=[  523], 90.00th=[  553], 95.00th=[  578],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[  758], 99.95th=[  865],
     | 99.99th=[ 2933]
   bw (  KiB/s): min=64192, max=65920, per=25.09%, avg=64988.00, stdev=525.15, samples=16
   iops        : min= 2006, max= 2060, avg=2030.88, stdev=16.41, samples=16
  lat (usec)   : 500=65.47%, 750=34.41%, 1000=0.08%
  lat (msec)   : 2=0.02%, 4=0.02%
  cpu          : usr=3.01%, sys=6.83%, ctx=16389, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=253MiB/s (265MB/s), 63.2MiB/s-63.5MiB/s (66.3MB/s-66.6MB/s), io=2048MiB (2147MB), run=8065-8098msec

Disk stats (read/write):
  sda: ios=64019/0, merge=0/0, ticks=29291/0, in_queue=31296, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2744 MiB / 2067 MiB ( 0 MiB / 0 MiB )
Script ran 1899 seconds or 31 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...14208 MiB / 15168 MiB / 14214 MiB / 0 MiB / 17 MiB ( 14208 MiB / 15168 MiB / 14214 MiB / 0 MiB / 17 MiB )
2 min passed...22112 MiB / 23584 MiB / 22111 MiB / 0 MiB / 19 MiB ( 7904 MiB / 8416 MiB / 7897 MiB / 0 MiB / 2 MiB )
3 min passed...22112 MiB / 23584 MiB / 22111 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...22112 MiB / 23584 MiB / 22114 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
5 min passed...22112 MiB / 23648 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 64 MiB / 1 MiB / 0 MiB / 0 MiB )
6 min passed...22112 MiB / 23840 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...22112 MiB / 23840 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...22112 MiB / 23840 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...22112 MiB / 23840 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...22112 MiB / 23968 MiB / 22115 MiB / 0 MiB / 19 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...22112 MiB / 24096 MiB / 22120 MiB / 0 MiB / 19 MiB ( 0 MiB / 128 MiB / 5 MiB / 0 MiB / 0 MiB )
12 min passed...22112 MiB / 24096 MiB / 22124 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
13 min passed...22112 MiB / 24096 MiB / 22124 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...22112 MiB / 24096 MiB / 22124 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...22144 MiB / 24256 MiB / 22131 MiB / 0 MiB / 19 MiB ( 32 MiB / 160 MiB / 7 MiB / 0 MiB / 0 MiB )
16 min passed...22144 MiB / 24288 MiB / 22131 MiB / 0 MiB / 19 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...22144 MiB / 24288 MiB / 22131 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 2048 MiB / 2055 MiB )
18 min passed...22144 MiB / 24288 MiB / 22131 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...22144 MiB / 24288 MiB / 22131 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...22144 MiB / 24352 MiB / 22131 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...22144 MiB / 24352 MiB / 22136 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
22 min passed...22144 MiB / 24352 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
23 min passed...22144 MiB / 24352 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...22144 MiB / 24352 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...22144 MiB / 24416 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...22144 MiB / 24416 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...22144 MiB / 24416 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...22144 MiB / 24416 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...22144 MiB / 24480 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...22144 MiB / 24512 MiB / 22138 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...22144 MiB / 24512 MiB / 22144 MiB / 2048 MiB / 2074 MiB ( 0 MiB / 0 MiB / 6 MiB / 0 MiB / 0 MiB )
 
Last edited:
Special Test Nr. 19: Like Round 2 - Run 4 - Test 8 but 32K volblocksize + ext4 created with 32K clustersize (mkfs.ext4 -b 4096 -O extent -O bigalloc -O has_journal -C 32k) + mounted with nodelalloc (additional to noatime,nodiratime).

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=11.1MiB/s][w=356 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=706: Fri Aug 13 00:22:57 2021
  write: IOPS=367, BW=11.5MiB/s (12.0MB/s)(1024MiB/89282msec); 0 zone resets
    clat (usec): min=1204, max=27323, avg=2697.69, stdev=915.19
     lat (usec): min=1205, max=27325, avg=2700.02, stdev=915.21
    clat percentiles (usec):
     |  1.00th=[ 1729],  5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2147],
     | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2507], 60.00th=[ 2671],
     | 70.00th=[ 2900], 80.00th=[ 3097], 90.00th=[ 3326], 95.00th=[ 3851],
     | 99.00th=[ 5669], 99.50th=[ 6915], 99.90th=[11863], 99.95th=[15008],
     | 99.99th=[25822]
   bw (  KiB/s): min= 8704, max=14848, per=99.98%, avg=11742.09, stdev=1075.24, samples=178
   iops        : min=  272, max=  464, avg=366.93, stdev=33.61, samples=178
  lat (msec)   : 2=9.13%, 4=86.26%, 10=4.36%, 20=0.21%, 50=0.02%
  cpu          : usr=1.27%, sys=5.25%, ctx=74449, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=11.5MiB/s (12.0MB/s), 11.5MiB/s-11.5MiB/s (12.0MB/s-12.0MB/s), io=1024MiB (1074MB), run=89282-89282msec

Disk stats (read/write):
  sdb: ios=0/98966, merge=0/65729, ticks=0/81911, in_queue=89156, util=99.89%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 1541 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 18 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=42.8MiB/s][r=1368 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=911: Fri Aug 13 00:38:20 2021
  read: IOPS=1439, BW=44.0MiB/s (47.2MB/s)(1024MiB/22764msec)
    clat (usec): min=420, max=3401, avg=682.75, stdev=85.37
     lat (usec): min=422, max=3403, avg=684.79, stdev=85.45
    clat percentiles (usec):
     |  1.00th=[  486],  5.00th=[  537], 10.00th=[  562], 20.00th=[  611],
     | 30.00th=[  644], 40.00th=[  668], 50.00th=[  693], 60.00th=[  717],
     | 70.00th=[  742], 80.00th=[  758], 90.00th=[  783], 95.00th=[  799],
     | 99.00th=[  824], 99.50th=[  840], 99.90th=[  922], 99.95th=[  988],
     | 99.99th=[ 1254]
   bw (  KiB/s): min=41472, max=56320, per=100.00%, avg=46110.91, stdev=3065.21, samples=45
   iops        : min= 1296, max= 1760, avg=1440.93, stdev=95.78, samples=45
  lat (usec)   : 500=1.82%, 750=73.68%, 1000=24.46%
  lat (msec)   : 2=0.03%, 4=0.01%
  cpu          : usr=2.77%, sys=9.19%, ctx=32775, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=44.0MiB/s (47.2MB/s), 44.0MiB/s-44.0MiB/s (47.2MB/s-47.2MB/s), io=1024MiB (1074MB), run=22764-22764msec

Disk stats (read/write):
  sdb: ios=32620/0, merge=0/0, ticks=20784/0, in_queue=22700, util=99.87%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1541 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 1914 seconds or 31 minutes.

Host:
Code: 
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...7264 MiB / 8288 MiB / 7415 MiB / 128 MiB / 147 MiB ( 7264 MiB / 8288 MiB / 7415 MiB / 128 MiB / 147 MiB )
2 min passed...11168 MiB / 12576 MiB / 11138 MiB / 224 MiB / 225 MiB ( 3904 MiB / 4288 MiB / 3723 MiB / 96 MiB / 78 MiB )
3 min passed...11200 MiB / 12608 MiB / 11140 MiB / 224 MiB / 225 MiB ( 32 MiB / 32 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...11200 MiB / 12608 MiB / 11144 MiB / 224 MiB / 225 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
5 min passed...11200 MiB / 12800 MiB / 11148 MiB / 224 MiB / 225 MiB ( 0 MiB / 192 MiB / 4 MiB / 0 MiB / 0 MiB )
6 min passed...11200 MiB / 13280 MiB / 11148 MiB / 224 MiB / 225 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11200 MiB / 13312 MiB / 11148 MiB / 224 MiB / 225 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...11200 MiB / 13312 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
9 min passed...11200 MiB / 13312 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...11200 MiB / 13728 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11200 MiB / 14080 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...11200 MiB / 14080 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...11200 MiB / 14112 MiB / 11152 MiB / 224 MiB / 225 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...11200 MiB / 14176 MiB / 11154 MiB / 224 MiB / 225 MiB ( 0 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...11200 MiB / 14816 MiB / 11162 MiB / 224 MiB / 225 MiB ( 0 MiB / 640 MiB / 8 MiB / 0 MiB / 0 MiB )
16 min passed...11200 MiB / 14944 MiB / 11164 MiB / 224 MiB / 225 MiB ( 0 MiB / 128 MiB / 2 MiB / 0 MiB / 0 MiB )
17 min passed...11200 MiB / 14944 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 2 MiB / 1024 MiB / 1024 MiB )
18 min passed...11200 MiB / 14944 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...11200 MiB / 15232 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11200 MiB / 15840 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...11200 MiB / 15936 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...11200 MiB / 15936 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...11200 MiB / 15936 MiB / 11166 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11200 MiB / 16320 MiB / 11170 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 384 MiB / 4 MiB / 0 MiB / 0 MiB )
25 min passed...11200 MiB / 16576 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
26 min passed...11200 MiB / 16576 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...11200 MiB / 16576 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11200 MiB / 16576 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...11200 MiB / 17024 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...11200 MiB / 17152 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...11200 MiB / 17152 MiB / 11172 MiB / 1248 MiB / 1249 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Special Test Nr. 20: Like Round 2 - Run 4 - Test 9 but 32K volblocksize + ext4 created with 32K clustersize (mkfs.ext4 -b 4096 -O extent -O bigalloc -O has_journal -C 32k) + mounted with nodelalloc (additional to noatime,nodiratime).

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=649: Fri Aug 13 01:01:53 2021
  write: IOPS=23.9k, BW=748MiB/s (784MB/s)(1024MiB/1369msec); 0 zone resets
    slat (usec): min=8, max=1492, avg=21.93, stdev=17.86
    clat (usec): min=72, max=13585, avg=2629.32, stdev=391.56
     lat (usec): min=113, max=13602, avg=2652.55, stdev=394.97
    clat percentiles (usec):
     |  1.00th=[ 2245],  5.00th=[ 2311], 10.00th=[ 2343], 20.00th=[ 2409],
     | 30.00th=[ 2442], 40.00th=[ 2474], 50.00th=[ 2507], 60.00th=[ 2573],
     | 70.00th=[ 2671], 80.00th=[ 2802], 90.00th=[ 3064], 95.00th=[ 3326],
     | 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5473], 99.95th=[ 6325],
     | 99.99th=[ 7242]
   bw (  KiB/s): min=745088, max=833216, per=100.00%, avg=789152.00, stdev=62315.91, samples=2
   iops        : min=23284, max=26038, avg=24661.00, stdev=1947.37, samples=2
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.24%, 4=98.14%, 10=1.53%, 20=0.01%
  cpu          : usr=27.78%, sys=54.09%, ctx=30267, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=748MiB/s (784MB/s), 748MiB/s-748MiB/s (784MB/s-784MB/s), io=1024MiB (1074MB), run=1369-1369msec

Disk stats (read/write):
  sdb: ios=0/27819, merge=0/0, ticks=0/2866, in_queue=3532, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process

async_rand_32K: (groupid=0, jobs=1): err= 0: pid=862: Fri Aug 13 01:16:54 2021
  read: IOPS=39.0k, BW=1249MiB/s (1309MB/s)(1024MiB/820msec)
    slat (usec): min=5, max=262, avg=14.52, stdev=11.13
    clat (usec): min=760, max=5709, avg=1576.09, stdev=257.83
     lat (usec): min=778, max=5727, avg=1592.33, stdev=259.40
    clat percentiles (usec):
     |  1.00th=[ 1045],  5.00th=[ 1172], 10.00th=[ 1254], 20.00th=[ 1369],
     | 30.00th=[ 1467], 40.00th=[ 1532], 50.00th=[ 1598], 60.00th=[ 1647],
     | 70.00th=[ 1696], 80.00th=[ 1745], 90.00th=[ 1827], 95.00th=[ 1893],
     | 99.00th=[ 2409], 99.50th=[ 2704], 99.90th=[ 3589], 99.95th=[ 3851],
     | 99.99th=[ 4359]
   bw (  MiB/s): min= 1289, max= 1289, per=100.00%, avg=1289.69, stdev= 0.00, samples=1
   iops        : min=41270, max=41270, avg=41270.00, stdev= 0.00, samples=1
  lat (usec)   : 1000=0.47%
  lat (msec)   : 2=97.18%, 4=2.33%, 10=0.02%
  cpu          : usr=32.48%, sys=65.57%, ctx=493, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1249MiB/s (1309MB/s), 1249MiB/s-1249MiB/s (1309MB/s-1309MB/s), io=1024MiB (1074MB), run=820-820msec

Disk stats (read/write):
  sdb: ios=26279/0, merge=0/0, ticks=22792/0, in_queue=23072, util=99.87%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1804 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2560 MiB / 2656 MiB / 2507 MiB / 0 MiB / 15 MiB ( 2560 MiB / 2656 MiB / 2507 MiB / 0 MiB / 15 MiB )
2 min passed...2560 MiB / 2944 MiB / 2507 MiB / 0 MiB / 15 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2560 MiB / 3456 MiB / 2509 MiB / 0 MiB / 15 MiB ( 0 MiB / 512 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...2560 MiB / 3488 MiB / 2517 MiB / 0 MiB / 16 MiB ( 0 MiB / 32 MiB / 8 MiB / 0 MiB / 1 MiB )
5 min passed...2560 MiB / 3488 MiB / 2517 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2560 MiB / 3520 MiB / 2517 MiB / 0 MiB / 16 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2560 MiB / 3872 MiB / 2518 MiB / 0 MiB / 16 MiB ( 0 MiB / 352 MiB / 1 MiB / 0 MiB / 0 MiB )
8 min passed...2560 MiB / 4000 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 128 MiB / 3 MiB / 0 MiB / 0 MiB )
9 min passed...2560 MiB / 4000 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2560 MiB / 4000 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2560 MiB / 4128 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2560 MiB / 4192 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2560 MiB / 4192 MiB / 2521 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2560 MiB / 4256 MiB / 2529 MiB / 0 MiB / 16 MiB ( 0 MiB / 64 MiB / 8 MiB / 0 MiB / 0 MiB )
15 min passed...2560 MiB / 4256 MiB / 2533 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 4 MiB / 1024 MiB / 1024 MiB )
16 min passed...2560 MiB / 4480 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 224 MiB / 7 MiB / 0 MiB / 0 MiB )
17 min passed...2560 MiB / 4896 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2560 MiB / 4928 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2560 MiB / 4928 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2560 MiB / 4960 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2560 MiB / 5344 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2560 MiB / 5568 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2560 MiB / 5568 MiB / 2540 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2560 MiB / 5568 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 0 MiB )
25 min passed...2560 MiB / 5664 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2560 MiB / 6112 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2560 MiB / 6144 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2560 MiB / 6144 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2560 MiB / 6144 MiB / 2547 MiB / 1024 MiB / 1040 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2592 MiB / 6400 MiB / 2561 MiB / 1024 MiB / 1041 MiB ( 32 MiB / 256 MiB / 14 MiB / 0 MiB / 1 MiB )
 
Round 2 Run 5 is the same pool used in run 4 but this time the test scripts are directly run on the host and not inside a Debian VM.
For this I created a zvol, created a partition on it, formated it with ext4 and mounted that partition directly on the host. My host is a debian 10 too and all zvol/ext4 options should be the same. So it is basically like round 2 run 4 just without the virtualization.

I created the zvol like this:
Code: 
zfs create -o volblocksize=8k -V 32G VMpool11/VLT/VM/testvol
fdisk /dev/VMpool11/VLT/VM/testvol
g
n
enter
enter
enter
w
mkfs.ext4 -b 4096 /dev/zd32p1
mkdir /mnt/test
mount -t ext4 -o noatime,nodiratime /dev/zd32p1 /mnt/test

Host is a striped mirror of 4x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=8K, primarycache=metadata.
There is no guest.

For info how the pool was created, "zpool status" and "zfs get all VMpool10/VLT/VM" see round 2 run 4.
 
Last edited:
Round 2 - Run 5 - Test 1: sync 4K random read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1593KiB/s][w=398 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=14512: Sat Aug 14 19:38:15 2021
  write: IOPS=376, BW=1506KiB/s (1542kB/s)(1024MiB/696182msec); 0 zone resets
    clat (usec): min=599, max=52837, avg=2648.11, stdev=1110.37
     lat (usec): min=599, max=52837, avg=2648.68, stdev=1110.38
    clat percentiles (usec):
     |  1.00th=[ 1045],  5.00th=[ 1827], 10.00th=[ 1975], 20.00th=[ 2114],
     | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2638],
     | 70.00th=[ 2769], 80.00th=[ 2933], 90.00th=[ 3261], 95.00th=[ 4424],
     | 99.00th=[ 5604], 99.50th=[ 6063], 99.90th=[16909], 99.95th=[19792],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1133, max= 4000, per=100.00%, avg=1506.06, stdev=258.33, samples=1392
   iops        : min=  283, max= 1000, avg=376.51, stdev=64.58, samples=1392
  lat (usec)   : 750=0.01%, 1000=0.76%
  lat (msec)   : 2=10.78%, 4=81.96%, 10=6.21%, 20=0.23%, 50=0.05%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.68%, sys=3.56%, ctx=524470, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1506KiB/s (1542kB/s), 1506KiB/s-1506KiB/s (1542kB/s-1542kB/s), io=1024MiB (1074MB), run=696182-696182msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=16.9MiB/s][r=4339 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=13814: Sat Aug 14 19:54:23 2021
  read: IOPS=3918, BW=15.3MiB/s (16.0MB/s)(1024MiB/66907msec)
    clat (usec): min=69, max=4074, avg=253.02, stdev=61.58
     lat (usec): min=69, max=4074, avg=253.22, stdev=61.67
    clat percentiles (usec):
     |  1.00th=[  194],  5.00th=[  198], 10.00th=[  200], 20.00th=[  204],
     | 30.00th=[  215], 40.00th=[  219], 50.00th=[  227], 60.00th=[  239],
     | 70.00th=[  265], 80.00th=[  302], 90.00th=[  351], 95.00th=[  392],
     | 99.00th=[  429], 99.50th=[  433], 99.90th=[  445], 99.95th=[  457],
     | 99.99th=[  490]
   bw (  KiB/s): min=10688, max=19776, per=99.93%, avg=15661.14, stdev=2041.22, samples=133
   iops        : min= 2672, max= 4944, avg=3915.28, stdev=510.30, samples=133
  lat (usec)   : 100=0.01%, 250=64.85%, 500=35.14%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.77%, sys=7.55%, ctx=262148, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=15.3MiB/s (16.0MB/s), 15.3MiB/s-15.3MiB/s (16.0MB/s-16.0MB/s), io=1024MiB (1074MB), run=66907-66907msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Script ran 2568 seconds or 42 minutes.

iostat logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4864 MiB / 5120 MiB / 4915 MiB / 288 MiB / 351 MiB ( 4864 MiB / 5120 MiB / 4915 MiB / 288 MiB / 351 MiB )
2 min passed...9344 MiB / 10144 MiB / 9322 MiB / 800 MiB / 829 MiB ( 4480 MiB / 5024 MiB / 4407 MiB / 512 MiB / 478 MiB )
3 min passed...13696 MiB / 15200 MiB / 13674 MiB / 1312 MiB / 1339 MiB ( 4352 MiB / 5056 MiB / 4352 MiB / 512 MiB / 510 MiB )
4 min passed...17920 MiB / 19808 MiB / 17932 MiB / 1824 MiB / 1861 MiB ( 4224 MiB / 4608 MiB / 4258 MiB / 512 MiB / 522 MiB )
5 min passed...22144 MiB / 24288 MiB / 22196 MiB / 2432 MiB / 2412 MiB ( 4224 MiB / 4480 MiB / 4264 MiB / 608 MiB / 551 MiB )
6 min passed...26368 MiB / 28768 MiB / 26390 MiB / 2944 MiB / 2986 MiB ( 4224 MiB / 4480 MiB / 4194 MiB / 512 MiB / 574 MiB )
7 min passed...30592 MiB / 33248 MiB / 30615 MiB / 3552 MiB / 3567 MiB ( 4224 MiB / 4480 MiB / 4225 MiB / 608 MiB / 581 MiB )
8 min passed...34752 MiB / 38080 MiB / 34757 MiB / 4128 MiB / 4147 MiB ( 4160 MiB / 4832 MiB / 4142 MiB / 576 MiB / 580 MiB )
9 min passed...38912 MiB / 43072 MiB / 38904 MiB / 4672 MiB / 4739 MiB ( 4160 MiB / 4992 MiB / 4147 MiB / 544 MiB / 592 MiB )
10 min passed...42944 MiB / 47392 MiB / 43019 MiB / 5312 MiB / 5330 MiB ( 4032 MiB / 4320 MiB / 4115 MiB / 640 MiB / 591 MiB )
11 min passed...47040 MiB / 51776 MiB / 47073 MiB / 5888 MiB / 5923 MiB ( 4096 MiB / 4384 MiB / 4054 MiB / 576 MiB / 593 MiB )
12 min passed...49472 MiB / 54336 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 2432 MiB / 2560 MiB / 2379 MiB / 384 MiB / 328 MiB )
13 min passed...49472 MiB / 54496 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...49472 MiB / 55104 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...49472 MiB / 55616 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...49472 MiB / 55648 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...49472 MiB / 55648 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...49472 MiB / 55648 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...49472 MiB / 55968 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...49472 MiB / 56640 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...49472 MiB / 56960 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...49472 MiB / 56960 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...49472 MiB / 56960 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...49472 MiB / 56992 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...49472 MiB / 57536 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...49472 MiB / 58080 MiB / 49452 MiB / 6272 MiB / 6251 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...49472 MiB / 58080 MiB / 49452 MiB / 7104 MiB / 7141 MiB ( 0 MiB / 0 MiB / 0 MiB / 832 MiB / 890 MiB )
28 min passed...49472 MiB / 58080 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 1152 MiB / 1158 MiB )
29 min passed...49472 MiB / 58080 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...49472 MiB / 58304 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...49472 MiB / 58464 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...49472 MiB / 59008 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...49472 MiB / 59008 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...49472 MiB / 59008 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...49472 MiB / 59008 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...49472 MiB / 59232 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...49472 MiB / 59648 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...49472 MiB / 59712 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...49472 MiB / 59712 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...49472 MiB / 59712 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...49472 MiB / 59744 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
42 min passed...49472 MiB / 60320 MiB / 49452 MiB / 8256 MiB / 8299 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 5 - Test 2: sync 16K random read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=6918KiB/s][w=432 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=31204: Sat Aug 14 20:22:04 2021
  write: IOPS=414, BW=6628KiB/s (6787kB/s)(1024MiB/158200msec); 0 zone resets
    clat (usec): min=1192, max=53423, avg=2401.15, stdev=987.59
     lat (usec): min=1193, max=53424, avg=2401.73, stdev=987.60
    clat percentiles (usec):
     |  1.00th=[ 1729],  5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2024],
     | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2245], 60.00th=[ 2311],
     | 70.00th=[ 2376], 80.00th=[ 2474], 90.00th=[ 2671], 95.00th=[ 3949],
     | 99.00th=[ 5080], 99.50th=[ 6194], 99.90th=[14877], 99.95th=[17957],
     | 99.99th=[27657]
   bw (  KiB/s): min= 5184, max= 7744, per=99.99%, avg=6627.23, stdev=485.23, samples=316
   iops        : min=  324, max=  484, avg=414.20, stdev=30.33, samples=316
  lat (msec)   : 2=17.10%, 4=78.04%, 10=4.55%, 20=0.27%, 50=0.03%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.99%, sys=3.44%, ctx=131164, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=6628KiB/s (6787kB/s), 6628KiB/s-6628KiB/s (6787kB/s-6787kB/s), io=1024MiB (1074MB), run=158200-158200msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][96.2%][r=42.1MiB/s][r=2693 IOPS][eta 00m:01s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=30802: Sat Aug 14 20:37:31 2021
  read: IOPS=2532, BW=39.6MiB/s (41.5MB/s)(1024MiB/25883msec)
    clat (usec): min=197, max=7021, avg=391.60, stdev=259.08
     lat (usec): min=197, max=7022, avg=391.93, stdev=259.11
    clat percentiles (usec):
     |  1.00th=[  245],  5.00th=[  269], 10.00th=[  285], 20.00th=[  306],
     | 30.00th=[  326], 40.00th=[  343], 50.00th=[  359], 60.00th=[  375],
     | 70.00th=[  392], 80.00th=[  416], 90.00th=[  445], 95.00th=[  465],
     | 99.00th=[ 1975], 99.50th=[ 2638], 99.90th=[ 3097], 99.95th=[ 3228],
     | 99.99th=[ 4621]
   bw (  KiB/s): min=34400, max=49952, per=99.93%, avg=40484.39, stdev=4050.13, samples=51
   iops        : min= 2150, max= 3122, avg=2530.27, stdev=253.13, samples=51
  lat (usec)   : 250=1.56%, 500=95.74%, 750=0.54%, 1000=0.45%
  lat (msec)   : 2=0.74%, 4=0.96%, 10=0.02%
  cpu          : usr=2.71%, sys=7.96%, ctx=65541, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=1024MiB (1074MB), run=25883-25883msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Script ran 1988 seconds or 33 minutes.

Iostat/SMART logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4768 MiB / 5664 MiB / 4785 MiB / 416 MiB / 376 MiB ( 4768 MiB / 5664 MiB / 4785 MiB / 416 MiB / 376 MiB )
2 min passed...9664 MiB / 11136 MiB / 9813 MiB / 768 MiB / 770 MiB ( 4896 MiB / 5472 MiB / 5028 MiB / 352 MiB / 394 MiB )
3 min passed...13120 MiB / 14816 MiB / 13140 MiB / 1056 MiB / 1027 MiB ( 3456 MiB / 3680 MiB / 3327 MiB / 288 MiB / 257 MiB )
4 min passed...13120 MiB / 14816 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 1 MiB )
5 min passed...13120 MiB / 14816 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...13120 MiB / 15264 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...13120 MiB / 15744 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...13120 MiB / 15936 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...13120 MiB / 15936 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...13120 MiB / 15936 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...13120 MiB / 16032 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...13120 MiB / 16672 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...13120 MiB / 17024 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...13120 MiB / 17056 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...13120 MiB / 17056 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...13120 MiB / 17056 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...13120 MiB / 17472 MiB / 13142 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...13120 MiB / 18080 MiB / 13142 MiB / 2048 MiB / 2042 MiB ( 0 MiB / 608 MiB / 0 MiB / 992 MiB / 1014 MiB )
19 min passed...13120 MiB / 18144 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 10 MiB )
20 min passed...13120 MiB / 18144 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...13120 MiB / 18144 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...13120 MiB / 18240 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...13120 MiB / 18784 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...13120 MiB / 19392 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...13120 MiB / 19392 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...13120 MiB / 19392 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...13120 MiB / 19392 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...13120 MiB / 19680 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...13120 MiB / 20256 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...13120 MiB / 20416 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...13120 MiB / 20416 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...13120 MiB / 20416 MiB / 13142 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...13120 MiB / 20576 MiB / 13147 MiB / 2048 MiB / 2052 MiB ( 0 MiB / 160 MiB / 5 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 5 - Test 3: sync 4M sequential read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=224MiB/s][w=56 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=29153: Sat Aug 14 21:20:35 2021
  write: IOPS=44, BW=179MiB/s (188MB/s)(10.0GiB/57084msec); 0 zone resets
    clat (usec): min=12735, max=89782, avg=20668.63, stdev=6988.79
     lat (usec): min=12735, max=89784, avg=20669.97, stdev=6988.87
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15008], 10.00th=[15401], 20.00th=[15926],
     | 30.00th=[16319], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[19006], 80.00th=[28443], 90.00th=[31589], 95.00th=[33817],
     | 99.00th=[39584], 99.50th=[43779], 99.90th=[64750], 99.95th=[65274],
     | 99.99th=[89654]
   bw (  KiB/s): min=98304, max=237568, per=99.93%, avg=183559.69, stdev=44202.70, samples=114
   iops        : min=   24, max=   58, avg=44.79, stdev=10.81, samples=114
  lat (msec)   : 20=70.90%, 50=28.95%, 100=0.16%
  cpu          : usr=7.21%, sys=1.68%, ctx=5881, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=10.0GiB (10.7GB), run=57084-57084msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=420MiB/s][r=105 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=27886: Sat Aug 14 21:36:01 2021
  read: IOPS=100, BW=404MiB/s (424MB/s)(10.0GiB/25349msec)
    clat (usec): min=8624, max=15613, avg=9897.12, stdev=698.21
     lat (usec): min=8625, max=15614, avg=9897.66, stdev=698.21
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9110], 10.00th=[ 9241], 20.00th=[ 9503],
     | 30.00th=[ 9503], 40.00th=[ 9634], 50.00th=[ 9634], 60.00th=[ 9765],
     | 70.00th=[ 9896], 80.00th=[10552], 90.00th=[10683], 95.00th=[11076],
     | 99.00th=[12649], 99.50th=[13173], 99.90th=[14091], 99.95th=[14353],
     | 99.99th=[15664]
   bw (  KiB/s): min=376832, max=434176, per=99.88%, avg=413172.42, stdev=17858.66, samples=50
   iops        : min=   92, max=  106, avg=100.84, stdev= 4.39, samples=50
  lat (msec)   : 10=71.95%, 20=28.05%
  cpu          : usr=0.08%, sys=2.92%, ctx=2562, majf=0, minf=1035
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=10.0GiB (10.7GB), run=25349-25349msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1887 seconds or 31 minutes.

Iostat/SMART logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...41760 MiB / 44288 MiB / 41986 MiB / 64 MiB / 40 MiB ( 41760 MiB / 44288 MiB / 41986 MiB / 64 MiB / 40 MiB )
2 min passed...42688 MiB / 45888 MiB / 42620 MiB / 64 MiB / 40 MiB ( 928 MiB / 1600 MiB / 634 MiB / 0 MiB / 0 MiB )
3 min passed...42688 MiB / 46240 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...42688 MiB / 46432 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...42688 MiB / 46432 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...42688 MiB / 46528 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...42688 MiB / 46784 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...42688 MiB / 47328 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...42688 MiB / 47328 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...42688 MiB / 47328 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...42688 MiB / 47424 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...42688 MiB / 47488 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...42688 MiB / 47968 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...42688 MiB / 48096 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...42688 MiB / 48096 MiB / 42620 MiB / 64 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...42688 MiB / 48160 MiB / 42620 MiB / 2752 MiB / 2779 MiB ( 0 MiB / 64 MiB / 0 MiB / 2688 MiB / 2739 MiB )
17 min passed...42688 MiB / 48224 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 64 MiB / 0 MiB / 7520 MiB / 7505 MiB )
18 min passed...42688 MiB / 48416 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...42688 MiB / 48704 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...42688 MiB / 48704 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...42688 MiB / 48704 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...42688 MiB / 48800 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...42688 MiB / 48992 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...42688 MiB / 49280 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...42688 MiB / 49408 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...42688 MiB / 49408 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...42688 MiB / 49504 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...42688 MiB / 49600 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...42688 MiB / 49888 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...42688 MiB / 50080 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...42688 MiB / 50272 MiB / 42620 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 5 - Test 4: async 4K random read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][80.0%][w=108MiB/s][w=27.6k IOPS][eta 00m:01s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=3654: Sat Aug 14 22:15:15 2021
  write: IOPS=56.2k, BW=219MiB/s (230MB/s)(1024MiB/4667msec); 0 zone resets
    slat (usec): min=5, max=34298, avg=12.49, stdev=96.09
    clat (nsec): min=1658, max=46837k, avg=1124839.39, stdev=1341215.94
     lat (usec): min=54, max=46848, avg=1137.46, stdev=1346.21
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[  693], 10.00th=[  742], 20.00th=[  807],
     | 30.00th=[  824], 40.00th=[  840], 50.00th=[  857], 60.00th=[  865],
     | 70.00th=[  898], 80.00th=[ 1057], 90.00th=[ 1237], 95.00th=[ 1991],
     | 99.00th=[ 7504], 99.50th=[ 8586], 99.90th=[14746], 99.95th=[21890],
     | 99.99th=[39060]
   bw (  KiB/s): min=74184, max=295704, per=98.87%, avg=222144.89, stdev=86900.23, samples=9
   iops        : min=18546, max=73926, avg=55536.22, stdev=21725.24, samples=9
  lat (usec)   : 2=0.01%, 100=0.01%, 250=0.03%, 500=0.55%, 750=10.15%
  lat (usec)   : 1000=67.05%
  lat (msec)   : 2=17.23%, 4=2.00%, 10=2.76%, 20=0.16%, 50=0.06%
  cpu          : usr=17.02%, sys=64.25%, ctx=4686, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=1024MiB (1074MB), run=4667-4667msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=326MiB/s][r=83.5k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=8615: Sat Aug 14 22:30:19 2021
  read: IOPS=81.8k, BW=320MiB/s (335MB/s)(1024MiB/3203msec)
    slat (usec): min=2, max=447, avg= 8.24, stdev= 9.28
    clat (usec): min=233, max=6358, avg=771.74, stdev=159.05
     lat (usec): min=237, max=6362, avg=780.30, stdev=159.84
    clat percentiles (usec):
     |  1.00th=[  433],  5.00th=[  519], 10.00th=[  619], 20.00th=[  725],
     | 30.00th=[  750], 40.00th=[  766], 50.00th=[  783], 60.00th=[  799],
     | 70.00th=[  816], 80.00th=[  832], 90.00th=[  865], 95.00th=[  889],
     | 99.00th=[  971], 99.50th=[ 1237], 99.90th=[ 3064], 99.95th=[ 3294],
     | 99.99th=[ 3589]
   bw (  KiB/s): min=314552, max=383528, per=100.00%, avg=327552.00, stdev=27466.53, samples=6
   iops        : min=78638, max=95882, avg=81888.00, stdev=6866.63, samples=6
  lat (usec)   : 250=0.01%, 500=3.88%, 750=26.77%, 1000=68.51%
  lat (msec)   : 2=0.54%, 4=0.29%, 10=0.01%
  cpu          : usr=18.55%, sys=78.42%, ctx=6190, majf=0, minf=74
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=320MiB/s (335MB/s), 320MiB/s-320MiB/s (335MB/s-335MB/s), io=1024MiB (1074MB), run=3203-3203msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1813 seconds or 30 minutes.

Iostat/SMART logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3712 MiB / 3968 MiB / 3724 MiB / 640 MiB / 620 MiB ( 3712 MiB / 3968 MiB / 3724 MiB / 640 MiB / 620 MiB )
2 min passed...3712 MiB / 4192 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...3712 MiB / 4320 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...3712 MiB / 4448 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...3712 MiB / 4640 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...3712 MiB / 4640 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...3712 MiB / 4736 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...3712 MiB / 4960 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...3712 MiB / 5152 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...3712 MiB / 5344 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...3712 MiB / 5440 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...3712 MiB / 5472 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...3712 MiB / 5760 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...3712 MiB / 5824 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...3712 MiB / 6144 MiB / 3724 MiB / 640 MiB / 620 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...3712 MiB / 6432 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 288 MiB / 0 MiB / 2048 MiB / 2044 MiB )
17 min passed...3712 MiB / 6432 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...3712 MiB / 6528 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...3712 MiB / 6528 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...3712 MiB / 6816 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...3712 MiB / 7232 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...3712 MiB / 7296 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...3712 MiB / 7328 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...3712 MiB / 7328 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...3712 MiB / 7488 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...3712 MiB / 7904 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...3712 MiB / 8096 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...3712 MiB / 8160 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...3712 MiB / 8288 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...3712 MiB / 8416 MiB / 3724 MiB / 2688 MiB / 2664 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 5 - Test 5: async 16K random read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=28726: Sat Aug 14 23:24:58 2021
  write: IOPS=58.0k, BW=907MiB/s (951MB/s)(1024MiB/1129msec); 0 zone resets
    slat (usec): min=6, max=6797, avg=11.74, stdev=49.06
    clat (usec): min=62, max=19485, avg=1084.64, stdev=601.52
     lat (usec): min=73, max=19495, avg=1096.51, stdev=605.59
    clat percentiles (usec):
     |  1.00th=[  791],  5.00th=[  873], 10.00th=[  906], 20.00th=[  930],
     | 30.00th=[  938], 40.00th=[  955], 50.00th=[  963], 60.00th=[  971],
     | 70.00th=[  988], 80.00th=[ 1057], 90.00th=[ 1319], 95.00th=[ 1614],
     | 99.00th=[ 2868], 99.50th=[ 5669], 99.90th=[ 8717], 99.95th=[ 9765],
     | 99.99th=[17695]
   bw (  KiB/s): min=860263, max=1052384, per=100.00%, avg=956323.50, stdev=135850.06, samples=2
   iops        : min=53766, max=65774, avg=59770.00, stdev=8490.94, samples=2
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.24%, 1000=72.54%
  lat (msec)   : 2=24.85%, 4=1.66%, 10=0.61%, 20=0.05%
  cpu          : usr=32.71%, sys=61.52%, ctx=684, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=907MiB/s (951MB/s), 907MiB/s-907MiB/s (951MB/s-951MB/s), io=1024MiB (1074MB), run=1129-1129msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2704: Sat Aug 14 23:40:00 2021
  read: IOPS=56.7k, BW=886MiB/s (929MB/s)(1024MiB/1156msec)
    slat (usec): min=2, max=454, avg= 9.89, stdev=23.62
    clat (usec): min=267, max=8881, avg=1115.42, stdev=370.05
     lat (usec): min=270, max=8886, avg=1125.65, stdev=371.61
    clat percentiles (usec):
     |  1.00th=[  553],  5.00th=[  709], 10.00th=[  775], 20.00th=[  857],
     | 30.00th=[  930], 40.00th=[ 1020], 50.00th=[ 1123], 60.00th=[ 1188],
     | 70.00th=[ 1237], 80.00th=[ 1270], 90.00th=[ 1369], 95.00th=[ 1500],
     | 99.00th=[ 2769], 99.50th=[ 3490], 99.90th=[ 4555], 99.95th=[ 4817],
     | 99.99th=[ 5342]
   bw (  KiB/s): min=897248, max=920288, per=100.00%, avg=908768.00, stdev=16291.74, samples=2
   iops        : min=56078, max=57518, avg=56798.00, stdev=1018.23, samples=2
  lat (usec)   : 500=0.57%, 750=7.44%, 1000=30.08%
  lat (msec)   : 2=60.11%, 4=1.55%, 10=0.26%
  cpu          : usr=14.11%, sys=65.28%, ctx=8977, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=1024MiB (1074MB), run=1156-1156msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1807 seconds or 30 minutes.


Iostat/SMART logging on host:
Code: 
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2560 MiB / 2784 MiB / 2620 MiB / 0 MiB / 1 MiB ( 2560 MiB / 2784 MiB / 2620 MiB / 0 MiB / 1 MiB )
2 min passed...2560 MiB / 2784 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2560 MiB / 3072 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2560 MiB / 3456 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2560 MiB / 3840 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2560 MiB / 3840 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2560 MiB / 3840 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2560 MiB / 3936 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2560 MiB / 4224 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2560 MiB / 4640 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2560 MiB / 4640 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2560 MiB / 4640 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2560 MiB / 4640 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2560 MiB / 5056 MiB / 2620 MiB / 0 MiB / 1 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2560 MiB / 5472 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 416 MiB / 0 MiB / 1088 MiB / 1036 MiB )
16 min passed...2560 MiB / 5696 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2560 MiB / 5696 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2560 MiB / 5696 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2560 MiB / 5952 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2560 MiB / 6400 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2560 MiB / 6880 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2560 MiB / 6880 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2560 MiB / 6880 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2560 MiB / 6912 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2560 MiB / 7392 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2560 MiB / 7744 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2560 MiB / 7744 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2560 MiB / 7744 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2560 MiB / 7744 MiB / 2620 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2560 MiB / 7904 MiB / 2624 MiB / 1088 MiB / 1037 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
 
I forgot to mention earlier, try logbias=throughput, for spindles its a very bad idea as it increases fragmentation, but for ssds it might be a good idea as it might reduce the number of writes occurring even with no slog, please try it on one of your tests since you are measuring smart data for writes.
 
  • Like
Reactions: Dunuin
Round 2 - Run 5 - Test 6: async 4M sequential read/write

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][95.0%][w=537MiB/s][w=134 IOPS][eta 00m:01s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=1795: Sun Aug 15 00:03:48 2021
  write: IOPS=133, BW=535MiB/s (561MB/s)(10.0GiB/19137msec); 0 zone resets
    slat (usec): min=79, max=52611, avg=500.71, stdev=1714.57
    clat (msec): min=5, max=4551, avg=475.96, stdev=967.36
     lat (msec): min=5, max=4551, avg=476.46, stdev=967.26
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   67], 10.00th=[   68], 20.00th=[   80],
     | 30.00th=[   90], 40.00th=[  102], 50.00th=[  122], 60.00th=[  146],
     | 70.00th=[  161], 80.00th=[  192], 90.00th=[ 1485], 95.00th=[ 3037],
     | 99.00th=[ 3842], 99.50th=[ 3876], 99.90th=[ 4530], 99.95th=[ 4530],
     | 99.99th=[ 4530]
   bw (  KiB/s): min=32062, max=2621440, per=100.00%, avg=1040469.05, stdev=749357.68, samples=19
   iops        : min=    7, max=  640, avg=253.89, stdev=183.03, samples=19
  lat (msec)   : 10=0.20%, 20=0.12%, 50=2.42%, 100=36.72%, 250=45.90%
  lat (msec)   : 1000=2.15%
  cpu          : usr=13.55%, sys=2.06%, ctx=1904, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=10.0GiB (10.7GB), run=19137-19137msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1369MiB/s][r=342 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=22032: Sun Aug 15 00:18:56 2021
  read: IOPS=351, BW=1406MiB/s (1474MB/s)(10.0GiB/7283msec)
    slat (usec): min=120, max=12376, avg=324.26, stdev=858.15
    clat (msec): min=55, max=335, avg=179.57, stdev=25.59
     lat (msec): min=58, max=335, avg=179.89, stdev=25.38
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[  146], 10.00th=[  163], 20.00th=[  171],
     | 30.00th=[  176], 40.00th=[  178], 50.00th=[  182], 60.00th=[  184],
     | 70.00th=[  188], 80.00th=[  192], 90.00th=[  199], 95.00th=[  207],
     | 99.00th=[  271], 99.50th=[  305], 99.90th=[  334], 99.95th=[  334],
     | 99.99th=[  334]
   bw (  MiB/s): min= 1304, max= 1584, per=99.65%, avg=1401.11, stdev=67.17, samples=14
   iops        : min=  326, max=  396, avg=350.21, stdev=16.79, samples=14
  lat (msec)   : 100=2.07%, 250=96.68%, 500=1.25%
  cpu          : usr=0.45%, sys=7.53%, ctx=2326, majf=0, minf=65545
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1406MiB/s (1474MB/s), 1406MiB/s-1406MiB/s (1474MB/s-1474MB/s), io=10.0GiB (10.7GB), run=7283-7283msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1831 seconds or 30 minutes.

Iostat/SMART logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...21952 MiB / 23200 MiB / 21891 MiB / 384 MiB / 435 MiB ( 21952 MiB / 23200 MiB / 21891 MiB / 384 MiB / 435 MiB )
2 min passed...21952 MiB / 23200 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...21952 MiB / 23328 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...21952 MiB / 23584 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...21952 MiB / 23584 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...21952 MiB / 23584 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...21952 MiB / 23584 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...21952 MiB / 23584 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...21952 MiB / 23616 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...21952 MiB / 23616 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...21952 MiB / 23616 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...21952 MiB / 23616 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...21952 MiB / 23616 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...21952 MiB / 23712 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...21952 MiB / 23872 MiB / 21891 MiB / 384 MiB / 435 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...21952 MiB / 23872 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 10720 MiB / 10738 MiB )
17 min passed...21952 MiB / 23872 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...21952 MiB / 23872 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...21952 MiB / 23904 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...21952 MiB / 24128 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...21952 MiB / 24128 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...21952 MiB / 24128 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...21952 MiB / 24128 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...21952 MiB / 24320 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...21952 MiB / 24672 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...21952 MiB / 24800 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...21952 MiB / 24800 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...21952 MiB / 24800 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...21952 MiB / 25088 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...21952 MiB / 25280 MiB / 21891 MiB / 11104 MiB / 11173 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 5 - Test 7: async 16K random read/write but 50% compressible

Fio test on host:
Code: 
root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=1886: Sun Aug 15 00:41:33 2021
  write: IOPS=59.3k, BW=927MiB/s (972MB/s)(1024MiB/1105msec); 0 zone resets
    slat (usec): min=6, max=3755, avg=12.22, stdev=17.70
    clat (usec): min=72, max=15478, avg=1062.00, stdev=258.78
     lat (usec): min=79, max=15492, avg=1074.39, stdev=260.85
    clat percentiles (usec):
     |  1.00th=[  840],  5.00th=[  898], 10.00th=[  922], 20.00th=[  938],
     | 30.00th=[  955], 40.00th=[  971], 50.00th=[  988], 60.00th=[ 1020],
     | 70.00th=[ 1090], 80.00th=[ 1188], 90.00th=[ 1270], 95.00th=[ 1369],
     | 99.00th=[ 1745], 99.50th=[ 1876], 99.90th=[ 5014], 99.95th=[ 5473],
     | 99.99th=[ 6587]
   bw (  KiB/s): min=837824, max=1026752, per=98.25%, avg=932288.00, stdev=133592.27, samples=2
   iops        : min=52364, max=64172, avg=58268.00, stdev=8349.52, samples=2
  lat (usec)   : 100=0.01%, 250=0.02%, 500=0.06%, 750=0.17%, 1000=54.27%
  lat (msec)   : 2=45.16%, 4=0.18%, 10=0.13%, 20=0.01%
  cpu          : usr=26.99%, sys=71.11%, ctx=296, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=927MiB/s (972MB/s), 927MiB/s-927MiB/s (972MB/s-972MB/s), io=1024MiB (1074MB), run=1105-1105msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process

async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=19397: Sun Aug 15 00:56:34 2021
  read: IOPS=82.0k, BW=1296MiB/s (1359MB/s)(1024MiB/790msec)
    slat (usec): min=2, max=236, avg= 7.98, stdev= 8.57
    clat (usec): min=228, max=6892, avg=761.00, stdev=277.57
     lat (usec): min=231, max=6897, avg=769.20, stdev=278.55
    clat percentiles (usec):
     |  1.00th=[  388],  5.00th=[  461], 10.00th=[  502], 20.00th=[  570],
     | 30.00th=[  668], 40.00th=[  775], 50.00th=[  807], 60.00th=[  824],
     | 70.00th=[  848], 80.00th=[  873], 90.00th=[  906], 95.00th=[  955],
     | 99.00th=[ 1156], 99.50th=[ 1401], 99.90th=[ 5276], 99.95th=[ 6652],
     | 99.99th=[ 6783]
   bw (  MiB/s): min= 1269, max= 1269, per=97.94%, avg=1269.47, stdev= 0.00, samples=1
   iops        : min=81246, max=81246, avg=81246.00, stdev= 0.00, samples=1
  lat (usec)   : 250=0.01%, 500=9.68%, 750=25.72%, 1000=61.23%
  lat (msec)   : 2=3.14%, 4=0.03%, 10=0.20%
  cpu          : usr=20.28%, sys=70.85%, ctx=3177, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1296MiB/s (1359MB/s), 1296MiB/s-1296MiB/s (1359MB/s-1359MB/s), io=1024MiB (1074MB), run=790-790msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1806 seconds or 30 minutes.

Iostat/SMART logging on host:
Code: 
root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2176 MiB / 2368 MiB / 2202 MiB / 0 MiB / 1 MiB ( 2176 MiB / 2368 MiB / 2202 MiB / 0 MiB / 1 MiB )
2 min passed...2176 MiB / 2656 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2176 MiB / 2784 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2176 MiB / 2784 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2176 MiB / 2784 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2176 MiB / 2784 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2176 MiB / 2816 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2176 MiB / 2816 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2176 MiB / 2816 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2176 MiB / 2816 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2176 MiB / 2816 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2176 MiB / 2848 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2176 MiB / 3008 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2176 MiB / 3008 MiB / 2202 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2176 MiB / 3008 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 512 MiB / 512 MiB )
16 min passed...2176 MiB / 3008 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2176 MiB / 3072 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2176 MiB / 3392 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2176 MiB / 3680 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2176 MiB / 3680 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2176 MiB / 3680 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2176 MiB / 3680 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2176 MiB / 4064 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2176 MiB / 4736 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2176 MiB / 4736 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2176 MiB / 4736 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2176 MiB / 4736 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2176 MiB / 5024 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2176 MiB / 5632 MiB / 2202 MiB / 512 MiB / 513 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2176 MiB / 5824 MiB / 2204 MiB / 512 MiB / 513 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back