how to best benchmark SSDs?

Round 2 - Run 10 - Test 1: sync 4K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=31.4MiB/s][w=8045 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=26030: Mon Aug 30 08:23:50 2021
  write: IOPS=7237, BW=28.3MiB/s (29.6MB/s)(1024MiB/36219msec); 0 zone resets
    clat (usec): min=99, max=2489, avg=135.52, stdev=23.64
     lat (usec): min=99, max=2489, avg=135.66, stdev=23.65
    clat percentiles (usec):
     |  1.00th=[  105],  5.00th=[  108], 10.00th=[  110], 20.00th=[  113],
     | 30.00th=[  126], 40.00th=[  135], 50.00th=[  137], 60.00th=[  139],
     | 70.00th=[  141], 80.00th=[  151], 90.00th=[  163], 95.00th=[  174],
     | 99.00th=[  190], 99.50th=[  196], 99.90th=[  260], 99.95th=[  281],
     | 99.99th=[  388]
   bw (  KiB/s): min=26520, max=33197, per=99.91%, avg=28923.74, stdev=2341.03, samples=72
   iops        : min= 6630, max= 8299, avg=7230.90, stdev=585.28, samples=72
  lat (usec)   : 100=0.01%, 250=99.88%, 500=0.11%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=3.32%, sys=9.41%, ctx=786433, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=28.3MiB/s (29.6MB/s), 28.3MiB/s-28.3MiB/s (29.6MB/s-29.6MB/s), io=1024MiB (1074MB), run=36219-36219msec

sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=34.2MiB/s][r=8752 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=2420: Mon Aug 30 08:39:20 2021
  read: IOPS=9065, BW=35.4MiB/s (37.1MB/s)(1024MiB/28918msec)
    clat (usec): min=40, max=2256, avg=109.40, stdev=12.36
     lat (usec): min=40, max=2256, avg=109.49, stdev=12.37
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[   96], 10.00th=[   96], 20.00th=[   97],
     | 30.00th=[   99], 40.00th=[  109], 50.00th=[  112], 60.00th=[  114],
     | 70.00th=[  115], 80.00th=[  118], 90.00th=[  126], 95.00th=[  127],
     | 99.00th=[  133], 99.50th=[  141], 99.90th=[  153], 99.95th=[  161],
     | 99.99th=[  289]
   bw (  KiB/s): min=33352, max=38720, per=100.00%, avg=36316.81, stdev=1575.48, samples=57
   iops        : min= 8338, max= 9680, avg=9079.19, stdev=393.86, samples=57
  lat (usec)   : 50=0.01%, 100=31.69%, 250=68.29%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 4=0.01%
  cpu          : usr=2.09%, sys=4.23%, ctx=262144, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=35.4MiB/s (37.1MB/s), 35.4MiB/s-35.4MiB/s (37.1MB/s-37.1MB/s), io=1024MiB (1074MB), run=28918-28918msec
  
Script ran 1870 seconds or 31 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1088 MiB / 1024 MiB / 32 MiB / 6 MiB ( 1024 MiB / 1088 MiB / 1024 MiB / 32 MiB / 6 MiB )
2 min passed...1024 MiB / 1088 MiB / 1024 MiB / 32 MiB / 7 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1216 MiB / 1024 MiB / 32 MiB / 9 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 10 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 1 MiB )
5 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 12 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
6 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 15 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1504 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 21 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 24 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 27 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 1760 MiB / 1024 MiB / 864 MiB / 839 MiB ( 0 MiB / 0 MiB / 0 MiB / 832 MiB / 812 MiB )
17 min passed...1024 MiB / 1792 MiB / 1024 MiB / 1056 MiB / 1054 MiB ( 0 MiB / 32 MiB / 0 MiB / 192 MiB / 215 MiB )
18 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1057 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1058 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
22 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1088 MiB / 1061 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 1 MiB )
23 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1088 MiB / 1063 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1088 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2400 MiB / 1024 MiB / 1088 MiB / 1066 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1067 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
27 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1069 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1070 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1072 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1073 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
31 min passed...1024 MiB / 2464 MiB / 1024 MiB / 1088 MiB / 1075 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
 
Round 2 - Run 10 - Test 2: sync 16K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=99.7MiB/s][w=6379 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1833: Mon Aug 30 09:35:23 2021
  write: IOPS=5815, BW=90.9MiB/s (95.3MB/s)(1024MiB/11270msec); 0 zone resets
    clat (usec): min=127, max=2218, avg=166.06, stdev=31.15
     lat (usec): min=127, max=2218, avg=166.20, stdev=31.15
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  135], 10.00th=[  137], 20.00th=[  139],
     | 30.00th=[  143], 40.00th=[  155], 50.00th=[  169], 60.00th=[  172],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  204], 95.00th=[  215],
     | 99.00th=[  245], 99.50th=[  253], 99.90th=[  285], 99.95th=[  306],
     | 99.99th=[  799]
   bw (  KiB/s): min=83136, max=102176, per=99.76%, avg=92813.50, stdev=8726.09, samples=22
   iops        : min= 5196, max= 6386, avg=5800.77, stdev=545.38, samples=22
  lat (usec)   : 250=99.31%, 500=0.67%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=4.85%, sys=6.47%, ctx=196610, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-95.3MB/s), io=1024MiB (1074MB), run=11270-11270msec

sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=102MiB/s][r=6496 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=10428: Mon Aug 30 09:50:34 2021
  read: IOPS=6566, BW=103MiB/s (108MB/s)(1024MiB/9980msec)
    clat (usec): min=66, max=3010, avg=151.01, stdev=24.04
     lat (usec): min=66, max=3010, avg=151.12, stdev=24.05
    clat percentiles (usec):
     |  1.00th=[  123],  5.00th=[  125], 10.00th=[  137], 20.00th=[  137],
     | 30.00th=[  143], 40.00th=[  151], 50.00th=[  153], 60.00th=[  155],
     | 70.00th=[  157], 80.00th=[  159], 90.00th=[  172], 95.00th=[  174],
     | 99.00th=[  182], 99.50th=[  188], 99.90th=[  200], 99.95th=[  206],
     | 99.99th=[  775]
   bw (  KiB/s): min=101280, max=110432, per=100.00%, avg=105215.26, stdev=2934.78, samples=19
   iops        : min= 6330, max= 6902, avg=6575.95, stdev=183.43, samples=19
  lat (usec)   : 100=0.01%, 250=99.98%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.14%, sys=5.01%, ctx=65537, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=1024MiB (1074MB), run=9980-9980msec

Script ran 1825 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1280 MiB / 1024 MiB / 0 MiB / 4 MiB ( 1024 MiB / 1280 MiB / 1024 MiB / 0 MiB / 4 MiB )
2 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 5 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 7 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
5 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
6 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1728 MiB / 1024 MiB / 0 MiB / 14 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1760 MiB / 1024 MiB / 0 MiB / 16 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1760 MiB / 1024 MiB / 0 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 1856 MiB / 1024 MiB / 32 MiB / 23 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 2112 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1025 MiB )
17 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
18 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2272 MiB / 1024 MiB / 1056 MiB / 1058 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 2 MiB )
22 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1059 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 1 MiB )
23 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1061 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1062 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1065 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
27 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1067 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2720 MiB / 1024 MiB / 1056 MiB / 1068 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1056 MiB / 1070 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1056 MiB / 1071 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Last edited:
Round 2 - Run 10 - Test 3: sync 4M sequential read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=252MiB/s][w=63 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=15019: Mon Aug 30 09:01:35 2021
  write: IOPS=70, BW=281MiB/s (295MB/s)(10.0GiB/36394msec); 0 zone resets
    clat (usec): min=9279, max=23046, avg=12454.69, stdev=1385.63
     lat (usec): min=9280, max=23047, avg=12455.58, stdev=1385.63
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[10945], 10.00th=[10945], 20.00th=[11076],
     | 30.00th=[11863], 40.00th=[12125], 50.00th=[12256], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13566], 90.00th=[13960], 95.00th=[14746],
     | 99.00th=[17695], 99.50th=[19268], 99.90th=[20317], 99.95th=[20579],
     | 99.99th=[22938]
   bw (  KiB/s): min=245760, max=303104, per=100.00%, avg=288507.72, stdev=12142.96, samples=72
   iops        : min=   60, max=   74, avg=70.39, stdev= 2.95, samples=72
  lat (msec)   : 10=0.08%, 20=99.77%, 50=0.16%
  cpu          : usr=12.68%, sys=0.94%, ctx=12803, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io=10.0GiB (10.7GB), run=36394-36394msec

sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=501MiB/s][r=125 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=23786: Mon Aug 30 09:16:56 2021
  read: IOPS=126, BW=504MiB/s (529MB/s)(10.0GiB/20312msec)
    clat (usec): min=7779, max=9713, avg=7929.49, stdev=135.02
     lat (usec): min=7780, max=9714, avg=7930.02, stdev=135.02
    clat percentiles (usec):
     |  1.00th=[ 7832],  5.00th=[ 7832], 10.00th=[ 7898], 20.00th=[ 7898],
     | 30.00th=[ 7898], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 7898],
     | 70.00th=[ 7898], 80.00th=[ 7963], 90.00th=[ 7963], 95.00th=[ 7963],
     | 99.00th=[ 8225], 99.50th=[ 9241], 99.90th=[ 9503], 99.95th=[ 9634],
     | 99.99th=[ 9765]
   bw (  KiB/s): min=507904, max=524288, per=99.99%, avg=516197.70, stdev=2293.61, samples=40
   iops        : min=  124, max=  128, avg=125.95, stdev= 0.64, samples=40
  lat (msec)   : 10=100.00%
  cpu          : usr=0.13%, sys=3.00%, ctx=2562, majf=0, minf=1033
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=504MiB/s (529MB/s), 504MiB/s-504MiB/s (529MB/s-529MB/s), io=10.0GiB (10.7GB), run=20312-20312msec

Script ran 1861 seconds or 31 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...10240 MiB / 11008 MiB / 10240 MiB / 0 MiB / 3 MiB ( 10240 MiB / 11008 MiB / 10240 MiB / 0 MiB / 3 MiB )
2 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 5 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 2 MiB )
3 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 6 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
4 min passed...10240 MiB / 11200 MiB / 10240 MiB / 32 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 2 MiB )
5 min passed...10240 MiB / 11200 MiB / 10240 MiB / 32 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
6 min passed...10240 MiB / 11200 MiB / 10240 MiB / 32 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
7 min passed...10240 MiB / 11200 MiB / 10240 MiB / 32 MiB / 12 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
8 min passed...10240 MiB / 11360 MiB / 10240 MiB / 32 MiB / 14 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 2 MiB )
9 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 15 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 1 MiB )
10 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
11 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
12 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
13 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
14 min passed...10240 MiB / 11520 MiB / 10240 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
15 min passed...10240 MiB / 11776 MiB / 10240 MiB / 32 MiB / 24 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
16 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10266 MiB ( 0 MiB / 96 MiB / 0 MiB / 10240 MiB / 10242 MiB )
17 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10267 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
18 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
19 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10270 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
20 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10272 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
21 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10273 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
22 min passed...10240 MiB / 12160 MiB / 10240 MiB / 10272 MiB / 10275 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
23 min passed...10240 MiB / 12192 MiB / 10240 MiB / 10272 MiB / 10276 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 1 MiB )
24 min passed...10240 MiB / 12192 MiB / 10240 MiB / 10272 MiB / 10278 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
25 min passed...10240 MiB / 12192 MiB / 10240 MiB / 10304 MiB / 10279 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 1 MiB )
26 min passed...10240 MiB / 12192 MiB / 10240 MiB / 10304 MiB / 10281 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
27 min passed...10240 MiB / 12192 MiB / 10240 MiB / 10304 MiB / 10282 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
28 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10304 MiB / 10284 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
29 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10304 MiB / 10285 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 1 MiB )
30 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10304 MiB / 10287 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
31 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10304 MiB / 10288 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Round 2 - Run 10 - Test 4: async 4K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][-.-%][w=276MiB/s][w=70.6k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=20013: Mon Aug 30 10:07:10 2021
  write: IOPS=70.1k, BW=274MiB/s (287MB/s)(1024MiB/3737msec); 0 zone resets
    slat (nsec): min=1443, max=463726, avg=3304.24, stdev=2642.58
    clat (usec): min=191, max=8379, avg=907.28, stdev=356.81
     lat (usec): min=230, max=8382, avg=910.66, stdev=356.81
    clat percentiles (usec):
     |  1.00th=[  469],  5.00th=[  506], 10.00th=[  553], 20.00th=[  627],
     | 30.00th=[  709], 40.00th=[  791], 50.00th=[  865], 60.00th=[  947],
     | 70.00th=[ 1029], 80.00th=[ 1123], 90.00th=[ 1237], 95.00th=[ 1352],
     | 99.00th=[ 2278], 99.50th=[ 2769], 99.90th=[ 3982], 99.95th=[ 4948],
     | 99.99th=[ 5604]
   bw (  KiB/s): min=277744, max=282568, per=100.00%, avg=280638.14, stdev=1768.60, samples=7
   iops        : min=69436, max=70642, avg=70159.43, stdev=442.03, samples=7
  lat (usec)   : 250=0.01%, 500=4.21%, 750=31.08%, 1000=31.21%
  lat (msec)   : 2=32.03%, 4=1.37%, 10=0.10%
  cpu          : usr=20.24%, sys=32.95%, ctx=253163, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=274MiB/s (287MB/s), 274MiB/s-274MiB/s (287MB/s-287MB/s), io=1024MiB (1074MB), run=3737-3737msec

async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=327MiB/s][r=83.7k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=28489: Mon Aug 30 10:22:14 2021
  read: IOPS=84.0k, BW=332MiB/s (348MB/s)(1024MiB/3085msec)
    slat (nsec): min=1555, max=90924, avg=2853.18, stdev=1398.07
    clat (usec): min=123, max=5544, avg=749.64, stdev=308.72
     lat (usec): min=126, max=5547, avg=752.57, stdev=308.72
    clat percentiles (usec):
     |  1.00th=[  281],  5.00th=[  367], 10.00th=[  424], 20.00th=[  510],
     | 30.00th=[  586], 40.00th=[  652], 50.00th=[  725], 60.00th=[  791],
     | 70.00th=[  865], 80.00th=[  938], 90.00th=[ 1045], 95.00th=[ 1172],
     | 99.00th=[ 1926], 99.50th=[ 2343], 99.90th=[ 3163], 99.95th=[ 3523],
     | 99.99th=[ 4228]
   bw (  KiB/s): min=333904, max=342424, per=99.83%, avg=339327.67, stdev=4099.81, samples=6
   iops        : min=83476, max=85606, avg=84831.83, stdev=1024.91, samples=6
  lat (usec)   : 250=0.44%, 500=18.32%, 750=35.43%, 1000=32.06%
  lat (msec)   : 2=12.87%, 4=0.86%, 10=0.02%
  cpu          : usr=15.01%, sys=39.20%, ctx=248093, majf=0, minf=71
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=332MiB/s (348MB/s), 332MiB/s-332MiB/s (348MB/s-348MB/s), io=1024MiB (1074MB), run=3085-3085msec

Script ran 1811 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1088 MiB / 1024 MiB / 32 MiB / 4 MiB ( 1024 MiB / 1088 MiB / 1024 MiB / 32 MiB / 4 MiB )
2 min passed...1024 MiB / 1120 MiB / 1024 MiB / 32 MiB / 5 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 7 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
5 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
6 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1568 MiB / 1024 MiB / 32 MiB / 16 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 17 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 2016 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 256 MiB / 0 MiB / 1024 MiB / 1025 MiB )
17 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1052 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
18 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1088 MiB / 1058 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 2 MiB )
22 min passed...1024 MiB / 2144 MiB / 1024 MiB / 1088 MiB / 1059 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
23 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1061 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1062 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1065 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
27 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1067 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1068 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2560 MiB / 1024 MiB / 1088 MiB / 1070 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1088 MiB / 1071 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Last edited:
Round 2 - Run 10 - Test 5: async 16K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=254MiB/s][w=16.2k IOPS][eta 00m:00s]
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=7549: Mon Aug 30 10:42:30 2021
  write: IOPS=16.4k, BW=256MiB/s (268MB/s)(1024MiB/4005msec); 0 zone resets
    slat (usec): min=2, max=1596, avg= 7.04, stdev=10.75
    clat (usec): min=824, max=20229, avg=3896.17, stdev=1674.50
     lat (usec): min=833, max=20237, avg=3903.40, stdev=1674.53
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1680], 10.00th=[ 2040], 20.00th=[ 2638],
     | 30.00th=[ 2966], 40.00th=[ 3261], 50.00th=[ 3621], 60.00th=[ 4047],
     | 70.00th=[ 4490], 80.00th=[ 5080], 90.00th=[ 5800], 95.00th=[ 6456],
     | 99.00th=[ 9765], 99.50th=[11469], 99.90th=[15270], 99.95th=[16319],
     | 99.99th=[19530]
   bw (  KiB/s): min=255968, max=266880, per=99.93%, avg=261636.00, stdev=4557.26, samples=8
   iops        : min=15998, max=16680, avg=16352.25, stdev=284.83, samples=8
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=9.33%, 4=49.69%, 10=39.99%, 20=0.92%, 50=0.01%
  cpu          : usr=17.68%, sys=14.16%, ctx=64110, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=1024MiB (1074MB), run=4005-4005msec

async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=16115: Mon Aug 30 10:57:33 2021
  read: IOPS=29.1k, BW=455MiB/s (477MB/s)(1024MiB/2251msec)
    slat (nsec): min=2249, max=51597, avg=5789.92, stdev=890.14
    clat (usec): min=370, max=11719, avg=2190.42, stdev=821.53
     lat (usec): min=376, max=11725, avg=2196.35, stdev=821.52
    clat percentiles (usec):
     |  1.00th=[  914],  5.00th=[ 1123], 10.00th=[ 1287], 20.00th=[ 1516],
     | 30.00th=[ 1729], 40.00th=[ 1926], 50.00th=[ 2114], 60.00th=[ 2311],
     | 70.00th=[ 2507], 80.00th=[ 2737], 90.00th=[ 3032], 95.00th=[ 3359],
     | 99.00th=[ 5276], 99.50th=[ 6194], 99.90th=[ 8029], 99.95th=[ 8586],
     | 99.99th=[10028]
   bw (  KiB/s): min=465632, max=466784, per=100.00%, avg=466280.00, stdev=477.77, samples=4
   iops        : min=29102, max=29174, avg=29142.50, stdev=29.86, samples=4
  lat (usec)   : 500=0.01%, 750=0.19%, 1000=1.81%
  lat (msec)   : 2=41.89%, 4=53.78%, 10=2.31%, 20=0.01%
  cpu          : usr=11.16%, sys=26.18%, ctx=64689, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=1024MiB (1074MB), run=2251-2251msec

Script ran 1811 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1408 MiB / 1024 MiB / 0 MiB / 4 MiB ( 1024 MiB / 1408 MiB / 1024 MiB / 0 MiB / 4 MiB )
2 min passed...1024 MiB / 1408 MiB / 1024 MiB / 0 MiB / 5 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1408 MiB / 1024 MiB / 0 MiB / 7 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1408 MiB / 1024 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
5 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 2 MiB )
6 min passed...1024 MiB / 1408 MiB / 1024 MiB / 32 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1536 MiB / 1024 MiB / 32 MiB / 13 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 14 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 2016 MiB / 1024 MiB / 32 MiB / 23 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 2080 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1025 MiB )
17 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
18 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2144 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2368 MiB / 1024 MiB / 1056 MiB / 1058 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 2 MiB )
22 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1059 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
23 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1061 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1062 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1088 MiB / 1065 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 1 MiB )
27 min passed...1024 MiB / 2592 MiB / 1024 MiB / 1088 MiB / 1067 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2752 MiB / 1024 MiB / 1088 MiB / 1068 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2752 MiB / 1024 MiB / 1088 MiB / 1070 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2752 MiB / 1024 MiB / 1088 MiB / 1071 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Round 2 - Run 10 - Test 6: async 4M sequential read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=252MiB/s][w=63 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=25830: Mon Aug 30 11:15:18 2021
  write: IOPS=75, BW=303MiB/s (318MB/s)(10.0GiB/33760msec); 0 zone resets
    slat (usec): min=37, max=31831, avg=11347.99, stdev=12069.83
    clat (msec): min=213, max=1007, avg=825.03, stdev=54.10
     lat (msec): min=213, max=1035, avg=836.38, stdev=54.17
    clat percentiles (msec):
     |  1.00th=[  542],  5.00th=[  793], 10.00th=[  810], 20.00th=[  818],
     | 30.00th=[  818], 40.00th=[  827], 50.00th=[  835], 60.00th=[  835],
     | 70.00th=[  844], 80.00th=[  852], 90.00th=[  852], 95.00th=[  860],
     | 99.00th=[  869], 99.50th=[  869], 99.90th=[  978], 99.95th=[ 1003],
     | 99.99th=[ 1011]
   bw (  KiB/s): min=221184, max=335872, per=99.61%, avg=309385.59, stdev=17612.44, samples=66
   iops        : min=   54, max=   82, avg=75.47, stdev= 4.27, samples=66
  lat (msec)   : 250=0.16%, 500=0.74%, 750=1.13%, 1000=97.89%
  cpu          : usr=13.43%, sys=0.86%, ctx=1236, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=303MiB/s (318MB/s), 303MiB/s-303MiB/s (318MB/s-318MB/s), io=10.0GiB (10.7GB), run=33760-33760msec

async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=724MiB/s][r=181 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=2246: Mon Aug 30 11:30:38 2021
  read: IOPS=135, BW=543MiB/s (569MB/s)(10.0GiB/18857msec)
    slat (usec): min=93, max=16343, avg=7315.59, stdev=7179.63
    clat (msec): min=110, max=569, avg=460.55, stdev=30.22
     lat (msec): min=111, max=569, avg=467.86, stdev=29.53
    clat percentiles (msec):
     |  1.00th=[  288],  5.00th=[  456], 10.00th=[  456], 20.00th=[  456],
     | 30.00th=[  456], 40.00th=[  456], 50.00th=[  464], 60.00th=[  472],
     | 70.00th=[  472], 80.00th=[  472], 90.00th=[  472], 95.00th=[  472],
     | 99.00th=[  472], 99.50th=[  481], 99.90th=[  558], 99.95th=[  558],
     | 99.99th=[  567]
   bw (  KiB/s): min=163840, max=557056, per=98.06%, avg=545262.27, stdev=64574.29, samples=37
   iops        : min=   40, max=  136, avg=133.08, stdev=15.76, samples=37
  lat (msec)   : 250=0.78%, 500=98.87%, 750=0.35%
  cpu          : usr=0.02%, sys=3.12%, ctx=1288, majf=0, minf=65544
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=543MiB/s (569MB/s), 543MiB/s-543MiB/s (569MB/s-569MB/s), io=10.0GiB (10.7GB), run=18857-18857msec

Script ran 1857 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...10240 MiB / 10944 MiB / 10240 MiB / 0 MiB / 3 MiB ( 10240 MiB / 10944 MiB / 10240 MiB / 0 MiB / 3 MiB )
2 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 5 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 2 MiB )
3 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 6 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
4 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
5 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
6 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
7 min passed...10240 MiB / 11200 MiB / 10240 MiB / 0 MiB / 12 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
8 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 14 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 2 MiB )
9 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 15 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
10 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
11 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
12 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
13 min passed...10240 MiB / 11520 MiB / 10240 MiB / 0 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
14 min passed...10240 MiB / 11744 MiB / 10240 MiB / 32 MiB / 23 MiB ( 0 MiB / 224 MiB / 0 MiB / 32 MiB / 2 MiB )
15 min passed...10240 MiB / 11872 MiB / 10240 MiB / 32 MiB / 24 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 1 MiB )
16 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10266 MiB ( 0 MiB / 0 MiB / 0 MiB / 10240 MiB / 10242 MiB )
17 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10267 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
18 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
19 min passed...10240 MiB / 11872 MiB / 10240 MiB / 10272 MiB / 10270 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
20 min passed...10240 MiB / 11968 MiB / 10240 MiB / 10272 MiB / 10272 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 2 MiB )
21 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10273 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
22 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10275 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
23 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10276 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
24 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10278 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
25 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10279 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
26 min passed...10240 MiB / 12224 MiB / 10240 MiB / 10272 MiB / 10281 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
27 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10272 MiB / 10282 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 1 MiB )
28 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10272 MiB / 10284 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
29 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10272 MiB / 10285 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
30 min passed...10240 MiB / 12544 MiB / 10240 MiB / 10272 MiB / 10287 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
 
Round 2 - Run 10 - Test 7: async 16K random read/write but 50% compressible

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=249MiB/s][w=15.0k IOPS][eta 00m:00s]
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=15647: Mon Aug 30 11:54:01 2021
  write: IOPS=16.3k, BW=255MiB/s (267MB/s)(1024MiB/4021msec); 0 zone resets
    slat (nsec): min=2540, max=69389, avg=6549.17, stdev=2373.50
    clat (usec): min=775, max=22378, avg=3914.55, stdev=1700.08
     lat (usec): min=783, max=22384, avg=3921.29, stdev=1700.10
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1680], 10.00th=[ 2024], 20.00th=[ 2606],
     | 30.00th=[ 2933], 40.00th=[ 3294], 50.00th=[ 3654], 60.00th=[ 4047],
     | 70.00th=[ 4555], 80.00th=[ 5145], 90.00th=[ 5866], 95.00th=[ 6587],
     | 99.00th=[ 9896], 99.50th=[11207], 99.90th=[14746], 99.95th=[15926],
     | 99.99th=[19792]
   bw (  KiB/s): min=246592, max=276000, per=99.97%, avg=260703.12, stdev=8272.89, samples=8
   iops        : min=15412, max=17250, avg=16293.88, stdev=517.07, samples=8
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=9.68%, 4=48.88%, 10=40.43%, 20=0.94%, 50=0.01%
  cpu          : usr=11.62%, sys=14.08%, ctx=64400, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=1024MiB (1074MB), run=4021-4021msec

async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=24153: Mon Aug 30 12:09:04 2021
  read: IOPS=29.1k, BW=455MiB/s (477MB/s)(1024MiB/2250msec)
    slat (nsec): min=1763, max=52646, avg=5286.89, stdev=1495.60
    clat (usec): min=482, max=11595, avg=2189.95, stdev=822.12
     lat (usec): min=489, max=11601, avg=2195.35, stdev=822.10
    clat percentiles (usec):
     |  1.00th=[  914],  5.00th=[ 1139], 10.00th=[ 1287], 20.00th=[ 1516],
     | 30.00th=[ 1729], 40.00th=[ 1926], 50.00th=[ 2114], 60.00th=[ 2311],
     | 70.00th=[ 2507], 80.00th=[ 2737], 90.00th=[ 3032], 95.00th=[ 3359],
     | 99.00th=[ 5276], 99.50th=[ 6194], 99.90th=[ 8160], 99.95th=[ 8848],
     | 99.99th=[10028]
   bw (  KiB/s): min=465184, max=466043, per=99.91%, avg=465614.75, stdev=461.81, samples=4
   iops        : min=29074, max=29127, avg=29100.75, stdev=28.65, samples=4
  lat (usec)   : 500=0.01%, 750=0.18%, 1000=1.84%
  lat (msec)   : 2=41.69%, 4=54.04%, 10=2.24%, 20=0.01%
  cpu          : usr=13.52%, sys=19.56%, ctx=64715, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=1024MiB (1074MB), run=2250-2250msec

Script ran 1811 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 3 MiB ( 1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 3 MiB )
2 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 5 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
3 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 6 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
4 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
5 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
6 min passed...1024 MiB / 1376 MiB / 1024 MiB / 0 MiB / 11 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
7 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 12 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
8 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
9 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 15 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
10 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
11 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
12 min passed...1024 MiB / 1664 MiB / 1024 MiB / 0 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
13 min passed...1024 MiB / 1824 MiB / 1024 MiB / 0 MiB / 21 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 1 MiB )
14 min passed...1024 MiB / 1984 MiB / 1024 MiB / 0 MiB / 23 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 2 MiB )
15 min passed...1024 MiB / 1984 MiB / 1024 MiB / 0 MiB / 24 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
16 min passed...1024 MiB / 1984 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 1056 MiB / 1026 MiB )
17 min passed...1024 MiB / 1984 MiB / 1024 MiB / 1056 MiB / 1051 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
18 min passed...1024 MiB / 1984 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
19 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1054 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 1 MiB )
20 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 2 MiB )
21 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1057 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
22 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1059 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
23 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
24 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1062 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
25 min passed...1024 MiB / 2400 MiB / 1024 MiB / 1056 MiB / 1063 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 1 MiB )
26 min passed...1024 MiB / 2688 MiB / 1024 MiB / 1056 MiB / 1065 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
27 min passed...1024 MiB / 2688 MiB / 1024 MiB / 1056 MiB / 1066 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
28 min passed...1024 MiB / 2688 MiB / 1024 MiB / 1056 MiB / 1068 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
29 min passed...1024 MiB / 2688 MiB / 1024 MiB / 1056 MiB / 1069 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
30 min passed...1024 MiB / 2688 MiB / 1024 MiB / 1056 MiB / 1071 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
 
Round 2 - Run 10 - Test 8: sync 32K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=142MiB/s][w=4540 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=9113: Mon Aug 30 12:40:13 2021
  write: IOPS=4602, BW=144MiB/s (151MB/s)(1024MiB/7120msec); 0 zone resets
    clat (usec): min=160, max=4003, avg=205.26, stdev=46.83
     lat (usec): min=160, max=4003, avg=205.43, stdev=46.83
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  169], 10.00th=[  172], 20.00th=[  182],
     | 30.00th=[  196], 40.00th=[  200], 50.00th=[  204], 60.00th=[  208],
     | 70.00th=[  215], 80.00th=[  221], 90.00th=[  235], 95.00th=[  247],
     | 99.00th=[  277], 99.50th=[  334], 99.90th=[  400], 99.95th=[  420],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=137600, max=164736, per=99.83%, avg=147014.00, stdev=10588.26, samples=14
   iops        : min= 4300, max= 5148, avg=4594.14, stdev=330.92, samples=14
  lat (usec)   : 250=96.05%, 500=3.91%, 750=0.01%, 1000=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%
  cpu          : usr=6.97%, sys=6.29%, ctx=98305, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=1024MiB (1074MB), run=7120-7120msec

sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=157MiB/s][r=5017 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=17732: Mon Aug 30 12:55:21 2021
  read: IOPS=5019, BW=157MiB/s (164MB/s)(1024MiB/6528msec)
    clat (usec): min=101, max=2746, avg=197.71, stdev=29.50
     lat (usec): min=101, max=2747, avg=197.82, stdev=29.50
    clat percentiles (usec):
     |  1.00th=[  178],  5.00th=[  188], 10.00th=[  190], 20.00th=[  190],
     | 30.00th=[  192], 40.00th=[  192], 50.00th=[  194], 60.00th=[  198],
     | 70.00th=[  204], 80.00th=[  208], 90.00th=[  208], 95.00th=[  210],
     | 99.00th=[  221], 99.50th=[  231], 99.90th=[  334], 99.95th=[  351],
     | 99.99th=[ 2212]
   bw (  KiB/s): min=159168, max=163008, per=100.00%, avg=160773.00, stdev=1096.17, samples=13
   iops        : min= 4974, max= 5094, avg=5024.15, stdev=34.26, samples=13
  lat (usec)   : 250=99.64%, 500=0.34%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.95%, sys=4.75%, ctx=32768, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=157MiB/s (164MB/s), 157MiB/s-157MiB/s (164MB/s-164MB/s), io=1024MiB (1074MB), run=6528-6528msec

Script ran 1818 seconds or 30 minutes.


SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1088 MiB / 1024 MiB / 0 MiB / 4 MiB ( 1024 MiB / 1088 MiB / 1024 MiB / 0 MiB / 4 MiB )
2 min passed...1024 MiB / 1088 MiB / 1024 MiB / 0 MiB / 5 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1088 MiB / 1024 MiB / 0 MiB / 7 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1088 MiB / 1024 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
5 min passed...1024 MiB / 1344 MiB / 1024 MiB / 0 MiB / 10 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 2 MiB )
6 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 11 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1568 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 20 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 1760 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1025 MiB )
17 min passed...1024 MiB / 1856 MiB / 1024 MiB / 1056 MiB / 1052 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 2 MiB )
18 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1058 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
22 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1059 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
23 min passed...1024 MiB / 2112 MiB / 1024 MiB / 1056 MiB / 1061 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1062 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1065 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
27 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1067 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1068 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1070 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2656 MiB / 1024 MiB / 1056 MiB / 1071 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Round 2 - Run 10 - Test 9: async 32K random read/write

Fio on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_d_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][-.-%][w=299MiB/s][w=9566 IOPS][eta 00m:00s]
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=3860: Mon Aug 30 13:28:15 2021
  write: IOPS=9536, BW=298MiB/s (312MB/s)(1024MiB/3436msec); 0 zone resets
    slat (nsec): min=3771, max=62382, avg=8510.29, stdev=1032.93
    clat (usec): min=1391, max=34275, avg=6684.15, stdev=2552.31
     lat (usec): min=1405, max=34284, avg=6692.85, stdev=2552.29
    clat percentiles (usec):
     |  1.00th=[ 2409],  5.00th=[ 3261], 10.00th=[ 3851], 20.00th=[ 4686],
     | 30.00th=[ 5342], 40.00th=[ 5932], 50.00th=[ 6521], 60.00th=[ 7111],
     | 70.00th=[ 7701], 80.00th=[ 8291], 90.00th=[ 9241], 95.00th=[10159],
     | 99.00th=[15926], 99.50th=[19006], 99.90th=[24773], 99.95th=[26608],
     | 99.99th=[31589]
   bw (  KiB/s): min=297600, max=316160, per=100.00%, avg=307096.83, stdev=6200.90, samples=6
   iops        : min= 9300, max= 9880, avg=9596.67, stdev=193.73, samples=6
  lat (msec)   : 2=0.12%, 4=11.80%, 10=82.53%, 20=5.15%, 50=0.40%
  cpu          : usr=18.63%, sys=11.06%, ctx=32098, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=298MiB/s (312MB/s), 298MiB/s-298MiB/s (312MB/s-312MB/s), io=1024MiB (1074MB), run=3436-3436msec

async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=12373: Mon Aug 30 13:43:18 2021
  read: IOPS=15.8k, BW=494MiB/s (518MB/s)(1024MiB/2074msec)
    slat (usec): min=3, max=1257, avg= 8.37, stdev=14.62
    clat (usec): min=1053, max=17987, avg=4038.09, stdev=1427.51
     lat (usec): min=1062, max=17996, avg=4046.64, stdev=1427.45
    clat percentiles (usec):
     |  1.00th=[ 1942],  5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2835],
     | 30.00th=[ 3195], 40.00th=[ 3556], 50.00th=[ 3916], 60.00th=[ 4293],
     | 70.00th=[ 4621], 80.00th=[ 5014], 90.00th=[ 5538], 95.00th=[ 5997],
     | 99.00th=[ 9372], 99.50th=[11207], 99.90th=[14222], 99.95th=[14877],
     | 99.99th=[17433]
   bw (  KiB/s): min=503552, max=508992, per=99.94%, avg=505296.00, stdev=2555.93, samples=4
   iops        : min=15736, max=15906, avg=15790.50, stdev=79.87, samples=4
  lat (msec)   : 2=1.40%, 4=50.92%, 10=46.91%, 20=0.77%
  cpu          : usr=7.28%, sys=18.04%, ctx=32045, majf=0, minf=520
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=1024MiB (1074MB), run=2074-2074msec

Script ran 1810 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 4 MiB ( 1024 MiB / 1312 MiB / 1024 MiB / 0 MiB / 4 MiB )
2 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 5 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 1 MiB )
3 min passed...1024 MiB / 1440 MiB / 1024 MiB / 0 MiB / 7 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
4 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 1 MiB )
5 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
6 min passed...1024 MiB / 1440 MiB / 1024 MiB / 32 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
7 min passed...1024 MiB / 1504 MiB / 1024 MiB / 32 MiB / 13 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
8 min passed...1024 MiB / 1696 MiB / 1024 MiB / 32 MiB / 14 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 1 MiB )
9 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 16 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
10 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
11 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
12 min passed...1024 MiB / 1760 MiB / 1024 MiB / 32 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
13 min passed...1024 MiB / 1792 MiB / 1024 MiB / 32 MiB / 22 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
14 min passed...1024 MiB / 2080 MiB / 1024 MiB / 32 MiB / 23 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
15 min passed...1024 MiB / 2080 MiB / 1024 MiB / 32 MiB / 25 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
16 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1025 MiB )
17 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1052 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
18 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1053 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
19 min passed...1024 MiB / 2080 MiB / 1024 MiB / 1056 MiB / 1055 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
20 min passed...1024 MiB / 2336 MiB / 1024 MiB / 1056 MiB / 1056 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
21 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1058 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 2 MiB )
22 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1059 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
23 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1061 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
24 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1062 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
25 min passed...1024 MiB / 2432 MiB / 1024 MiB / 1056 MiB / 1064 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
26 min passed...1024 MiB / 2560 MiB / 1024 MiB / 1088 MiB / 1065 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 1 MiB )
27 min passed...1024 MiB / 2752 MiB / 1024 MiB / 1088 MiB / 1067 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 2 MiB )
28 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1088 MiB / 1068 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 1 MiB )
29 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1088 MiB / 1070 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
30 min passed...1024 MiB / 2784 MiB / 1024 MiB / 1088 MiB / 1071 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Round 2 - Run 10 - Test X6: 4x async sequential read/write 32K

fio ran on host (part 1):
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_dx_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [W(4)][97.7%][w=300MiB/s][w=9592 IOPS][eta 00m:01s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=959: Mon Aug 30 14:21:43 2021
  write: IOPS=2301, BW=71.9MiB/s (75.4MB/s)(3072MiB/42716msec); 0 zone resets
    slat (nsec): min=6877, max=95132, avg=17268.15, stdev=7020.72
    clat (usec): min=86, max=8509, avg=401.16, stdev=464.97
     lat (usec): min=103, max=8522, avg=418.73, stdev=465.08
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  176], 10.00th=[  194], 20.00th=[  210],
     | 30.00th=[  225], 40.00th=[  227], 50.00th=[  235], 60.00th=[  239],
     | 70.00th=[  260], 80.00th=[  334], 90.00th=[ 1188], 95.00th=[ 1549],
     | 99.00th=[ 1876], 99.50th=[ 2278], 99.90th=[ 3949], 99.95th=[ 5538],
     | 99.99th=[ 7046]
   bw (  KiB/s): min=43456, max=83840, per=24.90%, avg=73350.65, stdev=7401.41, samples=85
   iops        : min= 1358, max= 2620, avg=2292.18, stdev=231.29, samples=85
  lat (usec)   : 100=0.07%, 250=67.46%, 500=18.96%, 750=1.42%, 1000=0.94%
  lat (msec)   : 2=10.38%, 4=0.67%, 10=0.10%
  cpu          : usr=6.04%, sys=5.74%, ctx=98318, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=960: Mon Aug 30 14:21:43 2021
  write: IOPS=2313, BW=72.3MiB/s (75.8MB/s)(3072MiB/42497msec); 0 zone resets
    slat (usec): min=7, max=117, avg=15.55, stdev= 6.21
    clat (usec): min=114, max=8577, avg=400.85, stdev=462.31
     lat (usec): min=126, max=8592, avg=416.67, stdev=462.42
    clat percentiles (usec):
     |  1.00th=[  167],  5.00th=[  180], 10.00th=[  204], 20.00th=[  223],
     | 30.00th=[  225], 40.00th=[  227], 50.00th=[  233], 60.00th=[  239],
     | 70.00th=[  255], 80.00th=[  322], 90.00th=[ 1188], 95.00th=[ 1549],
     | 99.00th=[ 1876], 99.50th=[ 2278], 99.90th=[ 3916], 99.95th=[ 5473],
     | 99.99th=[ 6980]
   bw (  KiB/s): min=42688, max=84800, per=25.12%, avg=73983.02, stdev=8038.05, samples=84
   iops        : min= 1334, max= 2650, avg=2311.94, stdev=251.19, samples=84
  lat (usec)   : 250=67.47%, 500=19.10%, 750=1.36%, 1000=0.97%
  lat (msec)   : 2=10.34%, 4=0.66%, 10=0.10%
  cpu          : usr=5.69%, sys=4.94%, ctx=98330, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=961: Mon Aug 30 14:21:43 2021
  write: IOPS=2309, BW=72.2MiB/s (75.7MB/s)(3072MiB/42569msec); 0 zone resets
    slat (usec): min=6, max=167, avg=16.31, stdev= 6.77
    clat (usec): min=90, max=8536, avg=400.79, stdev=463.68
     lat (usec): min=106, max=8548, avg=417.39, stdev=463.86
    clat percentiles (usec):
     |  1.00th=[  167],  5.00th=[  178], 10.00th=[  200], 20.00th=[  219],
     | 30.00th=[  225], 40.00th=[  227], 50.00th=[  233], 60.00th=[  239],
     | 70.00th=[  255], 80.00th=[  322], 90.00th=[ 1188], 95.00th=[ 1549],
     | 99.00th=[ 1876], 99.50th=[ 2278], 99.90th=[ 3916], 99.95th=[ 5538],
     | 99.99th=[ 6980]
   bw (  KiB/s): min=43520, max=84864, per=25.07%, avg=73845.41, stdev=7965.85, samples=85
   iops        : min= 1360, max= 2652, avg=2307.62, stdev=248.93, samples=85
  lat (usec)   : 100=0.01%, 250=67.74%, 500=18.84%, 750=1.34%, 1000=0.95%
  lat (msec)   : 2=10.36%, 4=0.67%, 10=0.10%
  cpu          : usr=5.46%, sys=5.74%, ctx=98322, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=962: Mon Aug 30 14:21:43 2021
  write: IOPS=2301, BW=71.9MiB/s (75.4MB/s)(3072MiB/42709msec); 0 zone resets
    slat (usec): min=7, max=147, avg=17.12, stdev= 7.04
    clat (usec): min=85, max=8497, avg=401.31, stdev=464.88
     lat (usec): min=105, max=8512, avg=418.72, stdev=464.99
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  176], 10.00th=[  196], 20.00th=[  212],
     | 30.00th=[  225], 40.00th=[  227], 50.00th=[  235], 60.00th=[  239],
     | 70.00th=[  260], 80.00th=[  330], 90.00th=[ 1205], 95.00th=[ 1549],
     | 99.00th=[ 1876], 99.50th=[ 2278], 99.90th=[ 3949], 99.95th=[ 5538],
     | 99.99th=[ 7046]
   bw (  KiB/s): min=43392, max=83456, per=24.91%, avg=73387.65, stdev=7544.79, samples=85
   iops        : min= 1356, max= 2608, avg=2293.33, stdev=235.78, samples=85
  lat (usec)   : 100=0.01%, 250=67.69%, 500=18.79%, 750=1.42%, 1000=0.93%
  lat (msec)   : 2=10.39%, 4=0.68%, 10=0.10%
  cpu          : usr=6.21%, sys=5.42%, ctx=98328, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=288MiB/s (302MB/s), 71.9MiB/s-72.3MiB/s (75.4MB/s-75.8MB/s), io=12.0GiB (12.9GB), run=42497-42716msec
 
Round 2 - Run 10 - Test X6: 4x async sequential read/write 32K

fio ran on host (part 2):
Code: 
async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=294MiB/s][r=9423 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=9890: Mon Aug 30 14:37:23 2021
  read: IOPS=2494, BW=77.9MiB/s (81.7MB/s)(3072MiB/39414msec)
    slat (usec): min=5, max=513, avg=19.58, stdev= 8.76
    clat (usec): min=2, max=8412, avg=378.40, stdev=207.12
     lat (usec): min=188, max=8446, avg=398.35, stdev=207.91
    clat percentiles (usec):
     |  1.00th=[  225],  5.00th=[  269], 10.00th=[  289], 20.00th=[  314],
     | 30.00th=[  334], 40.00th=[  347], 50.00th=[  359], 60.00th=[  371],
     | 70.00th=[  383], 80.00th=[  404], 90.00th=[  437], 95.00th=[  469],
     | 99.00th=[  906], 99.50th=[ 2409], 99.90th=[ 3064], 99.95th=[ 3163],
     | 99.99th=[ 3392]
   bw (  KiB/s): min=71488, max=87424, per=25.11%, avg=79867.73, stdev=4502.60, samples=78
   iops        : min= 2234, max= 2732, avg=2495.79, stdev=140.68, samples=78
  lat (usec)   : 4=0.01%, 250=3.49%, 500=93.66%, 750=1.59%, 1000=0.43%
  lat (msec)   : 2=0.23%, 4=0.60%, 10=0.01%
  cpu          : usr=2.93%, sys=8.46%, ctx=98320, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=9891: Mon Aug 30 14:37:23 2021
  read: IOPS=2490, BW=77.8MiB/s (81.6MB/s)(3072MiB/39473msec)
    slat (nsec): min=5522, max=72369, avg=19729.68, stdev=8614.84
    clat (usec): min=170, max=8112, avg=378.83, stdev=207.09
     lat (usec): min=186, max=8140, avg=398.91, stdev=207.77
    clat percentiles (usec):
     |  1.00th=[  225],  5.00th=[  269], 10.00th=[  289], 20.00th=[  314],
     | 30.00th=[  334], 40.00th=[  347], 50.00th=[  359], 60.00th=[  371],
     | 70.00th=[  383], 80.00th=[  404], 90.00th=[  437], 95.00th=[  469],
     | 99.00th=[  906], 99.50th=[ 2474], 99.90th=[ 3064], 99.95th=[ 3163],
     | 99.99th=[ 3392]
   bw (  KiB/s): min=71872, max=87424, per=25.07%, avg=79721.19, stdev=4107.43, samples=78
   iops        : min= 2246, max= 2732, avg=2491.28, stdev=128.36, samples=78
  lat (usec)   : 250=3.48%, 500=93.67%, 750=1.58%, 1000=0.43%
  lat (msec)   : 2=0.24%, 4=0.60%, 10=0.01%
  cpu          : usr=2.73%, sys=8.71%, ctx=98323, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=9892: Mon Aug 30 14:37:23 2021
  read: IOPS=2493, BW=77.9MiB/s (81.7MB/s)(3072MiB/39428msec)
    slat (usec): min=6, max=125, avg=20.03, stdev= 8.48
    clat (usec): min=162, max=6411, avg=377.96, stdev=205.10
     lat (usec): min=175, max=6444, avg=398.37, stdev=205.85
    clat percentiles (usec):
     |  1.00th=[  223],  5.00th=[  269], 10.00th=[  289], 20.00th=[  314],
     | 30.00th=[  334], 40.00th=[  347], 50.00th=[  359], 60.00th=[  371],
     | 70.00th=[  383], 80.00th=[  404], 90.00th=[  437], 95.00th=[  469],
     | 99.00th=[  898], 99.50th=[ 2409], 99.90th=[ 3064], 99.95th=[ 3163],
     | 99.99th=[ 3425]
   bw (  KiB/s): min=72768, max=87296, per=25.10%, avg=79843.53, stdev=4395.67, samples=78
   iops        : min= 2274, max= 2728, avg=2495.10, stdev=137.37, samples=78
  lat (usec)   : 250=3.49%, 500=93.67%, 750=1.58%, 1000=0.45%
  lat (msec)   : 2=0.21%, 4=0.60%, 10=0.01%
  cpu          : usr=2.93%, sys=8.85%, ctx=98318, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=9893: Mon Aug 30 14:37:23 2021
  read: IOPS=2484, BW=77.6MiB/s (81.4MB/s)(3072MiB/39562msec)
    slat (usec): min=7, max=167, avg=21.50, stdev= 8.54
    clat (usec): min=72, max=8219, avg=377.62, stdev=208.68
     lat (usec): min=89, max=8252, avg=399.51, stdev=209.39
    clat percentiles (usec):
     |  1.00th=[  194],  5.00th=[  260], 10.00th=[  289], 20.00th=[  314],
     | 30.00th=[  334], 40.00th=[  347], 50.00th=[  359], 60.00th=[  371],
     | 70.00th=[  388], 80.00th=[  408], 90.00th=[  441], 95.00th=[  474],
     | 99.00th=[  898], 99.50th=[ 2409], 99.90th=[ 3064], 99.95th=[ 3163],
     | 99.99th=[ 3359]
   bw (  KiB/s): min=72064, max=90112, per=24.87%, avg=79108.29, stdev=4064.32, samples=79
   iops        : min= 2252, max= 2816, avg=2472.08, stdev=126.99, samples=79
  lat (usec)   : 100=0.86%, 250=3.45%, 500=92.71%, 750=1.72%, 1000=0.43%
  lat (msec)   : 2=0.22%, 4=0.61%, 10=0.01%
  cpu          : usr=3.10%, sys=9.76%, ctx=98318, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=311MiB/s (326MB/s), 77.6MiB/s-77.9MiB/s (81.4MB/s-81.7MB/s), io=12.0GiB (12.9GB), run=39414-39562msec

Script ran 1887 seconds or 31 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...12288 MiB / 13056 MiB / 12288 MiB / 0 MiB / 6 MiB ( 12288 MiB / 13056 MiB / 12288 MiB / 0 MiB / 6 MiB )
2 min passed...12288 MiB / 13056 MiB / 12288 MiB / 0 MiB / 8 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
3 min passed...12288 MiB / 13056 MiB / 12288 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
4 min passed...12288 MiB / 13056 MiB / 12288 MiB / 0 MiB / 11 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
5 min passed...12288 MiB / 13344 MiB / 12288 MiB / 0 MiB / 12 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
6 min passed...12288 MiB / 13376 MiB / 12288 MiB / 0 MiB / 14 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
7 min passed...12288 MiB / 13376 MiB / 12288 MiB / 0 MiB / 15 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
8 min passed...12288 MiB / 13376 MiB / 12288 MiB / 0 MiB / 17 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
9 min passed...12288 MiB / 13376 MiB / 12288 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
10 min passed...12288 MiB / 13376 MiB / 12288 MiB / 0 MiB / 20 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
11 min passed...12288 MiB / 13664 MiB / 12288 MiB / 0 MiB / 21 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 1 MiB )
12 min passed...12288 MiB / 13728 MiB / 12288 MiB / 32 MiB / 23 MiB ( 0 MiB / 64 MiB / 0 MiB / 32 MiB / 2 MiB )
13 min passed...12288 MiB / 13728 MiB / 12288 MiB / 32 MiB / 24 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
14 min passed...12288 MiB / 13728 MiB / 12288 MiB / 32 MiB / 26 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
15 min passed...12288 MiB / 13728 MiB / 12288 MiB / 32 MiB / 27 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
16 min passed...12288 MiB / 13728 MiB / 12288 MiB / 4864 MiB / 4885 MiB ( 0 MiB / 0 MiB / 0 MiB / 4832 MiB / 4858 MiB )
17 min passed...12288 MiB / 14048 MiB / 12288 MiB / 12320 MiB / 12318 MiB ( 0 MiB / 320 MiB / 0 MiB / 7456 MiB / 7433 MiB )
18 min passed...12288 MiB / 14080 MiB / 12288 MiB / 12320 MiB / 12320 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 2 MiB )
19 min passed...12288 MiB / 14080 MiB / 12288 MiB / 12320 MiB / 12321 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
20 min passed...12288 MiB / 14080 MiB / 12288 MiB / 12320 MiB / 12323 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
21 min passed...12288 MiB / 14080 MiB / 12288 MiB / 12320 MiB / 12324 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
22 min passed...12288 MiB / 14080 MiB / 12288 MiB / 12320 MiB / 12326 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
23 min passed...12288 MiB / 14336 MiB / 12288 MiB / 12320 MiB / 12327 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
24 min passed...12288 MiB / 14400 MiB / 12288 MiB / 12320 MiB / 12329 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 2 MiB )
25 min passed...12288 MiB / 14400 MiB / 12288 MiB / 12320 MiB / 12330 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
26 min passed...12288 MiB / 14400 MiB / 12288 MiB / 12320 MiB / 12332 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
27 min passed...12288 MiB / 14400 MiB / 12288 MiB / 12320 MiB / 12333 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
28 min passed...12288 MiB / 14400 MiB / 12288 MiB / 12320 MiB / 12335 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
29 min passed...12288 MiB / 14656 MiB / 12288 MiB / 12320 MiB / 12336 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 1 MiB )
30 min passed...12288 MiB / 14752 MiB / 12288 MiB / 12320 MiB / 12338 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 2 MiB )
31 min passed...12288 MiB / 14752 MiB / 12288 MiB / 12320 MiB / 12339 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
 
Round 2 - Run 10 - Test X7: 4x sync sequential read/write 32K

fio ran on host (part 1):
Code: 
root@Hypervisor:~# bash /root/scripts/bench/benchmark_dx_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [W(4)][100.0%][w=240MiB/s][w=7677 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=20878: Mon Aug 30 14:56:12 2021
  write: IOPS=1823, BW=56.0MiB/s (59.8MB/s)(512MiB/8985msec); 0 zone resets
    clat (usec): min=187, max=6627, avg=532.08, stdev=313.80
     lat (usec): min=187, max=6627, avg=532.49, stdev=313.84
    clat percentiles (usec):
     |  1.00th=[  297],  5.00th=[  355], 10.00th=[  400], 20.00th=[  445],
     | 30.00th=[  461], 40.00th=[  478], 50.00th=[  490], 60.00th=[  502],
     | 70.00th=[  515], 80.00th=[  537], 90.00th=[  594], 95.00th=[  742],
     | 99.00th=[ 2278], 99.50th=[ 3064], 99.90th=[ 4228], 99.95th=[ 4621],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=14592, max=71153, per=25.96%, avg=58017.29, stdev=12229.36, samples=17
   iops        : min=  456, max= 2223, avg=1813.00, stdev=382.13, samples=17
  lat (usec)   : 250=0.15%, 500=59.10%, 750=35.95%, 1000=2.60%
  lat (msec)   : 2=1.08%, 4=0.99%, 10=0.13%
  cpu          : usr=6.49%, sys=6.68%, ctx=39610, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=20879: Mon Aug 30 14:56:12 2021
  write: IOPS=1781, BW=55.7MiB/s (58.4MB/s)(512MiB/9195msec); 0 zone resets
    clat (usec): min=197, max=6620, avg=543.89, stdev=321.92
     lat (usec): min=197, max=6620, avg=544.33, stdev=321.94
    clat percentiles (usec):
     |  1.00th=[  306],  5.00th=[  371], 10.00th=[  416], 20.00th=[  453],
     | 30.00th=[  469], 40.00th=[  482], 50.00th=[  494], 60.00th=[  506],
     | 70.00th=[  523], 80.00th=[  545], 90.00th=[  660], 95.00th=[  766],
     | 99.00th=[ 2376], 99.50th=[ 3097], 99.90th=[ 4228], 99.95th=[ 4621],
     | 99.99th=[ 5997]
   bw (  KiB/s): min=14592, max=63360, per=25.45%, avg=56872.44, stdev=11107.72, samples=18
   iops        : min=  456, max= 1980, avg=1777.22, stdev=347.12, samples=18
  lat (usec)   : 250=0.03%, 500=55.96%, 750=38.35%, 1000=3.42%
  lat (msec)   : 2=1.06%, 4=1.03%, 10=0.15%
  cpu          : usr=7.19%, sys=7.03%, ctx=38389, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=20880: Mon Aug 30 14:56:12 2021
  write: IOPS=1745, BW=54.6MiB/s (57.2MB/s)(512MiB/9385msec); 0 zone resets
    clat (usec): min=198, max=6632, avg=554.90, stdev=332.15
     lat (usec): min=198, max=6633, avg=555.39, stdev=332.18
    clat percentiles (usec):
     |  1.00th=[  212],  5.00th=[  355], 10.00th=[  420], 20.00th=[  457],
     | 30.00th=[  474], 40.00th=[  486], 50.00th=[  498], 60.00th=[  510],
     | 70.00th=[  529], 80.00th=[  562], 90.00th=[  725], 95.00th=[  799],
     | 99.00th=[ 2409], 99.50th=[ 3130], 99.90th=[ 4228], 99.95th=[ 4621],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=14528, max=61248, per=24.54%, avg=54846.11, stdev=10505.21, samples=18
   iops        : min=  454, max= 1914, avg=1713.89, stdev=328.28, samples=18
  lat (usec)   : 250=2.19%, 500=49.59%, 750=40.21%, 1000=5.68%
  lat (msec)   : 2=1.12%, 4=1.08%, 10=0.14%
  cpu          : usr=6.62%, sys=8.73%, ctx=35869, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=20881: Mon Aug 30 14:56:12 2021
  write: IOPS=1769, BW=55.3MiB/s (57.0MB/s)(512MiB/9259msec); 0 zone resets
    clat (usec): min=215, max=6628, avg=547.45, stdev=324.72
     lat (usec): min=215, max=6628, avg=547.93, stdev=324.74
    clat percentiles (usec):
     |  1.00th=[  302],  5.00th=[  367], 10.00th=[  416], 20.00th=[  453],
     | 30.00th=[  469], 40.00th=[  482], 50.00th=[  494], 60.00th=[  506],
     | 70.00th=[  529], 80.00th=[  545], 90.00th=[  685], 95.00th=[  783],
     | 99.00th=[ 2409], 99.50th=[ 3130], 99.90th=[ 4228], 99.95th=[ 4621],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=14528, max=70144, per=25.26%, avg=56452.44, stdev=11242.83, samples=18
   iops        : min=  454, max= 2192, avg=1764.11, stdev=351.33, samples=18
  lat (usec)   : 250=0.09%, 500=55.11%, 750=38.38%, 1000=4.12%
  lat (msec)   : 2=1.09%, 4=1.09%, 10=0.13%
  cpu          : usr=6.30%, sys=8.36%, ctx=37249, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=218MiB/s (229MB/s), 54.6MiB/s-56.0MiB/s (57.2MB/s-59.8MB/s), io=2048MiB (2147MB), run=8985-9385msec
 
Round 2 - Run 10 - Test X7: 4x sync sequential read/write 32K

fio ran on host (part 2):
Code: 
sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=322MiB/s][r=10.3k IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=29460: Mon Aug 30 15:11:19 2021
  read: IOPS=2557, BW=79.9MiB/s (83.8MB/s)(512MiB/6407msec)
    clat (usec): min=183, max=3285, avg=388.52, stdev=123.99
     lat (usec): min=183, max=3285, avg=388.81, stdev=123.99
    clat percentiles (usec):
     |  1.00th=[  243],  5.00th=[  277], 10.00th=[  306], 20.00th=[  343],
     | 30.00th=[  359], 40.00th=[  371], 50.00th=[  383], 60.00th=[  396],
     | 70.00th=[  412], 80.00th=[  433], 90.00th=[  461], 95.00th=[  486],
     | 99.00th=[  545], 99.50th=[  570], 99.90th=[ 2868], 99.95th=[ 3064],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=77632, max=84672, per=24.98%, avg=81728.00, stdev=1948.49, samples=12
   iops        : min= 2426, max= 2646, avg=2554.00, stdev=60.89, samples=12
  lat (usec)   : 250=1.50%, 500=95.07%, 750=3.22%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.18%
  cpu          : usr=2.28%, sys=5.85%, ctx=16389, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=29461: Mon Aug 30 15:11:19 2021
  read: IOPS=2564, BW=80.1MiB/s (84.0MB/s)(512MiB/6389msec)
    clat (usec): min=176, max=3227, avg=387.98, stdev=114.47
     lat (usec): min=176, max=3227, avg=388.21, stdev=114.47
    clat percentiles (usec):
     |  1.00th=[  247],  5.00th=[  281], 10.00th=[  306], 20.00th=[  343],
     | 30.00th=[  359], 40.00th=[  371], 50.00th=[  383], 60.00th=[  396],
     | 70.00th=[  412], 80.00th=[  429], 90.00th=[  461], 95.00th=[  486],
     | 99.00th=[  545], 99.50th=[  570], 99.90th=[ 2573], 99.95th=[ 2868],
     | 99.99th=[ 3228]
   bw (  KiB/s): min=79744, max=83520, per=25.14%, avg=82261.33, stdev=1199.30, samples=12
   iops        : min= 2492, max= 2610, avg=2570.67, stdev=37.48, samples=12
  lat (usec)   : 250=1.16%, 500=95.26%, 750=3.39%
  lat (msec)   : 2=0.02%, 4=0.17%
  cpu          : usr=1.00%, sys=4.27%, ctx=16386, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=29462: Mon Aug 30 15:11:19 2021
  read: IOPS=2556, BW=79.9MiB/s (83.8MB/s)(512MiB/6410msec)
    clat (usec): min=89, max=3261, avg=388.56, stdev=125.69
     lat (usec): min=89, max=3261, avg=388.87, stdev=125.71
    clat percentiles (usec):
     |  1.00th=[  241],  5.00th=[  277], 10.00th=[  302], 20.00th=[  343],
     | 30.00th=[  359], 40.00th=[  371], 50.00th=[  383], 60.00th=[  396],
     | 70.00th=[  412], 80.00th=[  433], 90.00th=[  461], 95.00th=[  490],
     | 99.00th=[  553], 99.50th=[  578], 99.90th=[ 2933], 99.95th=[ 3130],
     | 99.99th=[ 3261]
   bw (  KiB/s): min=77824, max=84928, per=24.93%, avg=81546.67, stdev=1874.04, samples=12
   iops        : min= 2432, max= 2654, avg=2548.33, stdev=58.56, samples=12
  lat (usec)   : 100=0.09%, 250=1.50%, 500=94.49%, 750=3.72%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.18%
  cpu          : usr=2.08%, sys=6.99%, ctx=16386, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=29463: Mon Aug 30 15:11:19 2021
  read: IOPS=2560, BW=80.0MiB/s (83.9MB/s)(512MiB/6400msec)
    clat (usec): min=189, max=3302, avg=388.04, stdev=121.42
     lat (usec): min=189, max=3302, avg=388.35, stdev=121.42
    clat percentiles (usec):
     |  1.00th=[  247],  5.00th=[  281], 10.00th=[  306], 20.00th=[  343],
     | 30.00th=[  359], 40.00th=[  371], 50.00th=[  383], 60.00th=[  396],
     | 70.00th=[  408], 80.00th=[  429], 90.00th=[  461], 95.00th=[  486],
     | 99.00th=[  545], 99.50th=[  570], 99.90th=[ 2802], 99.95th=[ 3064],
     | 99.99th=[ 3261]
   bw (  KiB/s): min=78848, max=85184, per=25.02%, avg=81866.67, stdev=1933.69, samples=12
   iops        : min= 2464, max= 2662, avg=2558.33, stdev=60.50, samples=12
  lat (usec)   : 250=1.20%, 500=95.34%, 750=3.27%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.17%
  cpu          : usr=2.41%, sys=6.14%, ctx=16387, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=320MiB/s (335MB/s), 79.9MiB/s-80.1MiB/s (83.8MB/s-84.0MB/s), io=2048MiB (2147MB), run=6389-6410msec

Script ran 1820 seconds or 30 minutes.

SMART/iostat on host:
Code: 
root@Hypervisor:~# bash /root/scripts/bench/smartctl_12.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2048 MiB / 2304 MiB / 2048 MiB / 0 MiB / 7 MiB ( 2048 MiB / 2304 MiB / 2048 MiB / 0 MiB / 7 MiB )
2 min passed...2048 MiB / 2304 MiB / 2048 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
3 min passed...2048 MiB / 2304 MiB / 2048 MiB / 0 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
4 min passed...2048 MiB / 2304 MiB / 2048 MiB / 0 MiB / 12 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
5 min passed...2048 MiB / 2336 MiB / 2048 MiB / 0 MiB / 13 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 1 MiB )
6 min passed...2048 MiB / 2624 MiB / 2048 MiB / 0 MiB / 15 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 2 MiB )
7 min passed...2048 MiB / 2624 MiB / 2048 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
8 min passed...2048 MiB / 2624 MiB / 2048 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
9 min passed...2048 MiB / 2624 MiB / 2048 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
10 min passed...2048 MiB / 2624 MiB / 2048 MiB / 0 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
11 min passed...2048 MiB / 2720 MiB / 2048 MiB / 0 MiB / 22 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 1 MiB )
12 min passed...2048 MiB / 2944 MiB / 2048 MiB / 0 MiB / 24 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 2 MiB )
13 min passed...2048 MiB / 2944 MiB / 2048 MiB / 0 MiB / 25 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
14 min passed...2048 MiB / 2944 MiB / 2048 MiB / 0 MiB / 27 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
15 min passed...2048 MiB / 2944 MiB / 2048 MiB / 0 MiB / 28 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
16 min passed...2048 MiB / 2944 MiB / 2048 MiB / 2080 MiB / 2078 MiB ( 0 MiB / 0 MiB / 0 MiB / 2080 MiB / 2050 MiB )
17 min passed...2048 MiB / 3072 MiB / 2048 MiB / 2080 MiB / 2079 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 1 MiB )
18 min passed...2048 MiB / 3296 MiB / 2048 MiB / 2080 MiB / 2081 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 2 MiB )
19 min passed...2048 MiB / 3296 MiB / 2048 MiB / 2080 MiB / 2082 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
20 min passed...2048 MiB / 3296 MiB / 2048 MiB / 2080 MiB / 2084 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
21 min passed...2048 MiB / 3296 MiB / 2048 MiB / 2080 MiB / 2085 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
22 min passed...2048 MiB / 3296 MiB / 2048 MiB / 2080 MiB / 2087 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
23 min passed...2048 MiB / 3488 MiB / 2048 MiB / 2080 MiB / 2088 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 1 MiB )
24 min passed...2048 MiB / 3616 MiB / 2048 MiB / 2080 MiB / 2090 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 2 MiB )
25 min passed...2048 MiB / 3616 MiB / 2048 MiB / 2080 MiB / 2091 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
26 min passed...2048 MiB / 3616 MiB / 2048 MiB / 2080 MiB / 2093 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
27 min passed...2048 MiB / 3616 MiB / 2048 MiB / 2080 MiB / 2094 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 1 MiB )
28 min passed...2048 MiB / 3616 MiB / 2048 MiB / 2080 MiB / 2096 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 2 MiB )
29 min passed...2048 MiB / 3840 MiB / 2048 MiB / 2080 MiB / 2097 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 1 MiB )
30 min passed...2048 MiB / 3968 MiB / 2048 MiB / 2080 MiB / 2099 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 2 MiB )
 
Dunuin said:
I wanted to see if adding a SLOG might reduce the write amplification:

Total Write Amplification (factor):
5 disk raidz1 without slog5 disk raidz1 + 1 disk slog
T1: sync 4K random44,8835,28
T2: sync 16K random13,7911,66
T8: sync 32K random11,488,66
T3: sync 4M sequential3,042,76
TX7: 4x sync 32K sequential on host8,347,63

So looks like adding a SLOG helps to lower write amplification. And yes, the writes to the slog are included in the total write amplification.

For performance I'm not sure how comparable they are. The tests wouthout slog where created of a raidz1 of 4x S3710 + 1x S3700 and the tests with slog used 6x S3710. And the S3710 is on paper faster than the S3700.

Write Performance (MiB/s):
5 disk raidz1 without slog5 disk raidz1 + 1 disk slog
T1: sync 4K random1,171,26
T2: sync 16K random5,505,40
T8: sync 32K random12,4011,60
T3: sync 4M sequential262,00183,00
TX7: 4x sync 32K sequential on host34,1032,20

So here a SLOG doesn't help with performance at all.
Click to expand...
yeah for performance the benefit would be if e.g. your main storage is a spindle, and the slog is a ssd. But it is good you confirmed is a write amplification improvement meaning there is a 2nd reason to consider for adding one.
 
chrcoluk said:
yeah for performance the benefit would be if e.g. your main storage is a spindle, and the slog is a ssd. But it is good you confirmed is a write amplification improvement meaning there is a 2nd reason to consider for adding one.
Click to expand...
Yes, for write amification it really made a difference.

I had a really hard time deciding how I should setup my storage.

I've got only 10 SATA ports or 9 SATA + 1 M.2 and I have no free PCIe slots to add any HBAs.
For boot, root and swap I got two S3700 100GB SSDs running as a LUKS encrypted mdadm raid1. And it was hard to set it up that way so I wanted to keep it that way.
So I only got 8 ports for my other storages but I got 6x S3710 200GB SSD, 2x S3700 200GB SSD and 2x 3TB HDDs.
I really would like to keep the two HDDs inside the server because maybe I want to setup some VMs later that need alot of storage. So my idea was to somehow use the 6x S3710 200GB as a VM storage. That way I could use the 2x S3700 SSDs for something else like as mirrored special devices for my TrueNAS server to speed up the spinning rust there.

Right now my VMs consume about 200GB of space without snapshots and I want some free space so the VMs can grow so I would need atleast 400GB or better 500GB of usable storage.

So the options would be:

1.) 6x SSDs as striped mirror with 16K volblocksize.
Should give better IOPS but terrible write amplification and only 432GB of usable storage. Benefit would be that 16K volblocksize should be fine for all my MySQL DBs so I don't loose too much performance because of read overhead.

2.) 5x SSDs as raidz1 with 32K volblocksize + 1x LUKS encrypted LVMthin.
The raidz1 would give me 576GB of usable storage for VMs where I really care about data integrity. Write amplification woukd still be bad but way better than a striped mirror. It also would be faster than a striped mirror for sequential writes. What I dont like is the 32K volblocksize what basically cuts the read and write performance for MySQL DBs in half but with only 16K volblocksize I would loose alot of storage due to padding.
But the LUKS encrypted LVMthin would be really great for my Graylog LXC and Zabbix VM because these two do most of my writes. A encrypted LVMthin got a waaaaay lower write amplification (for example only factor 8,41 instead of factor 45 (raidz1) or factor 59 (striped mirror) for 4K sync writes) but it wouldn't be able to prevent data corruption and there is no redundancy if the SSD fails. So I woudn't want to run everything on it but for zabbix and graylog it should be fine because they only store metrics and logs, PBS is doing a daily backup and it really doesn't matter if some metrics or logs may corrupt or I loose up to a day of metrics/logs.

3.) 4x SSDs as striped mirror with 8K volblocksize + 2x SSDs as luks encrypted mdadm raid1 for lvmthin
8K volblocksize would be great and performance should be good but I would only be able to use 288GB and that is a little bit low if I already need 200GB. The mirrored lvmthin would give a little bit more redundancy but that doesn't help much because I still don't like to run important VMs on a lvmthin because of the missing bit rot prevention.

4.) 4x SSDs as striped mirror with 8K volblocksize + 1 SSD as SLOG + 1x SSDs as luks encrypted lvmthin
Same as option 3 but using one SSD as slog to lower the write amplification on the striped mirror instead of using it as a mirror for the lvmthin.
I really like that setup but with only 288GB that won't last long and I maybe need to redo everything again in one year.

5.) 3x SSDs as raidz1 with 16K volblocksize + 2x SSD as ZFS mirror with 4K volblocksize + 1x SSDs as luks encrypted lvmthin
Here I would get 288 + 144 = 432GB of reliable ZFS storage + 200GB unreliable LVM thin. A little bit annoying to have 3 separate storages but atleast it would combine low vollbocksizes with enough storage and a LVMthin for write-heavy but unimportant stuff.

I went with option nr 2 and will look how good that will work. Most of the time my IO delay is below 1% so I think I dont need all the IOPS of a striped mirror. But if I ever need more performance I could destroy the raidz1, keep the LVMthin and try it with a option number 5. Shoudn't be that much of a problem because of the dedicated root/boot disks and how easiy PVE makes it to backup/restore VMs/LXCs.

Edit:
Here are some updated graphs showing ZFS vs LVMthin:
WA_T1.pngWA_T2.png
So ZFS causes reaaaaly bad write amplification compared to LVMthin.

Total Write Amplification (factor):
4k sync: 8,41 LVMthin vs 35,28 - 62,44 ZFS
16k sync: 4,63 LVMthin vs 11,66 - 20,12 ZFS
4M sync: 1,17 LVMthin vs 2,76 - 5,02 ZFS
4K async: 3,28 LVMthin vs 5,44 - 11,5 ZFS
 
Last edited:
  • Like
Reactions: xxmail
interesting that mirror has higher amplification especially when taking into account the padding on zvols with small blocks on raidz.

As for the gap to lvm thin, I wonder if the impact of things like writing checksum and other zfs only metadata is quite big with 4k ashift.
 
chrcoluk said:
interesting that mirror has higher amplification especially when taking into account the padding on zvols with small blocks on raidz.
Click to expand...
Padding is no problem if you use the right volblocksize. My 5 disk raidz1 with 32k volblocksize should only loose 20% of raw capacity to parity and 0% to padding. A striped mirror would loose 50% to parity and so needs to write everything twice. My raidz1 don't need to write it twice, it only writes it 1.25 times. I think that is why raidz got a lower write amplification.
chrcoluk said:
As for the gap to lvm thin, I wonder if the impact of things like writing checksum and other zfs only metadata is quite big with 4k ashift.
Click to expand...
Yes, with 4K sync writes the difference between ZFS and LVMthin it the biggest. But even with big writes like 4096K ZFSs write amplification is still a multiple of that of LVMthin.
 
Last edited:
chrcoluk said:
So you added the write to second mirrored device in the amplification figures?
Click to expand...
Yes, its data that fio wrote amplified to total data written across all drives. That way it is possible to compare numbers using different setups with different amount of drives.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom