how to best benchmark SSDs?

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test 8: sync 32K random read/write

Fio test on host:

Code:

root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=9696KiB/s][w=303 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=8027: Sun Aug 15 01:14:45 2021
  write: IOPS=357, BW=11.2MiB/s (11.7MB/s)(1024MiB/91608msec); 0 zone resets
    clat (usec): min=1126, max=44826, avg=2775.92, stdev=1180.05
     lat (usec): min=1127, max=44827, avg=2776.52, stdev=1180.06
    clat percentiles (usec):
     |  1.00th=[ 2008],  5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2311],
     | 30.00th=[ 2376], 40.00th=[ 2442], 50.00th=[ 2507], 60.00th=[ 2573],
     | 70.00th=[ 2671], 80.00th=[ 2769], 90.00th=[ 3785], 95.00th=[ 4752],
     | 99.00th=[ 5473], 99.50th=[ 9765], 99.90th=[16581], 99.95th=[21103],
     | 99.99th=[28967]
   bw (  KiB/s): min= 8320, max=13184, per=100.00%, avg=11446.52, stdev=966.57, samples=183
   iops        : min=  260, max=  412, avg=357.70, stdev=30.21, samples=183
  lat (msec)   : 2=0.95%, 4=90.24%, 10=8.34%, 20=0.39%, 50=0.07%
  cpu          : usr=1.09%, sys=3.06%, ctx=65569, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=11.2MiB/s (11.7MB/s), 11.2MiB/s-11.2MiB/s (11.7MB/s-11.7MB/s), io=1024MiB (1074MB), run=91608-91608msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=80.5MiB/s][r=2576 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=31114: Sun Aug 15 01:29:58 2021
  read: IOPS=2704, BW=84.5MiB/s (88.6MB/s)(1024MiB/12117msec)
    clat (usec): min=253, max=1220, avg=367.23, stdev=41.68
     lat (usec): min=253, max=1221, avg=367.47, stdev=41.74
    clat percentiles (usec):
     |  1.00th=[  293],  5.00th=[  310], 10.00th=[  318], 20.00th=[  334],
     | 30.00th=[  343], 40.00th=[  355], 50.00th=[  363], 60.00th=[  371],
     | 70.00th=[  383], 80.00th=[  400], 90.00th=[  424], 95.00th=[  445],
     | 99.00th=[  486], 99.50th=[  498], 99.90th=[  523], 99.95th=[  529],
     | 99.99th=[  619]
   bw (  KiB/s): min=80640, max=90880, per=100.00%, avg=86635.17, stdev=2883.92, samples=24
   iops        : min= 2520, max= 2840, avg=2707.33, stdev=90.10, samples=24
  lat (usec)   : 500=99.52%, 750=0.47%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=2.34%, sys=6.17%, ctx=32770, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=84.5MiB/s (88.6MB/s), 84.5MiB/s-84.5MiB/s (88.6MB/s-88.6MB/s), io=1024MiB (1074MB), run=12117-12117msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1908 seconds or 31 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...7424 MiB / 7808 MiB / 7388 MiB / 352 MiB / 333 MiB ( 7424 MiB / 7808 MiB / 7388 MiB / 352 MiB / 333 MiB )
2 min passed...11680 MiB / 12768 MiB / 11665 MiB / 480 MiB / 512 MiB ( 4256 MiB / 4960 MiB / 4277 MiB / 128 MiB / 179 MiB )
3 min passed...11680 MiB / 13184 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...11680 MiB / 13248 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...11680 MiB / 13248 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...11680 MiB / 13248 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11680 MiB / 13376 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...11680 MiB / 13472 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...11680 MiB / 13472 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...11680 MiB / 13472 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11680 MiB / 13472 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...11680 MiB / 13504 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...11680 MiB / 13696 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...11680 MiB / 13824 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...11680 MiB / 13824 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...11680 MiB / 13824 MiB / 11665 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...11680 MiB / 13824 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1024 MiB )
18 min passed...11680 MiB / 13952 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...11680 MiB / 14080 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11680 MiB / 14112 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...11680 MiB / 14112 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...11680 MiB / 14112 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...11680 MiB / 14144 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11680 MiB / 14656 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...11680 MiB / 14944 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...11680 MiB / 14944 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...11680 MiB / 14944 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11680 MiB / 15072 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...11680 MiB / 15424 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...11680 MiB / 15744 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...11680 MiB / 15840 MiB / 11665 MiB / 1504 MiB / 1536 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test 9: async 32K random read/write

Fio test on host:

Code:

root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=0)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=23386: Sun Aug 15 01:48:10 2021
  write: IOPS=40.8k, BW=1275MiB/s (1337MB/s)(1024MiB/803msec); 0 zone resets
    slat (usec): min=7, max=22638, avg=14.66, stdev=184.56
    clat (usec): min=79, max=29802, avg=1541.46, stdev=1753.54
     lat (usec): min=93, max=29816, avg=1556.27, stdev=1765.77
    clat percentiles (usec):
     |  1.00th=[ 1106],  5.00th=[ 1123], 10.00th=[ 1156], 20.00th=[ 1188],
     | 30.00th=[ 1205], 40.00th=[ 1221], 50.00th=[ 1254], 60.00th=[ 1319],
     | 70.00th=[ 1385], 80.00th=[ 1467], 90.00th=[ 1647], 95.00th=[ 1958],
     | 99.00th=[10814], 99.50th=[16581], 99.90th=[23987], 99.95th=[23987],
     | 99.99th=[28705]
   bw (  MiB/s): min= 1539, max= 1539, per=100.00%, avg=1539.75, stdev= 0.00, samples=1
   iops        : min=49272, max=49272, avg=49272.00, stdev= 0.00, samples=1
  lat (usec)   : 100=0.01%, 250=0.02%, 500=0.05%, 750=0.06%, 1000=0.15%
  lat (msec)   : 2=95.01%, 4=3.19%, 10=0.48%, 20=0.80%, 50=0.25%
  cpu          : usr=36.41%, sys=51.00%, ctx=260, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=1275MiB/s (1337MB/s), 1275MiB/s-1275MiB/s (1337MB/s-1337MB/s), io=1024MiB (1074MB), run=803-803msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=29374: Sun Aug 15 02:03:12 2021
  read: IOPS=31.3k, BW=977MiB/s (1025MB/s)(1024MiB/1048msec)
    slat (usec): min=3, max=3234, avg=20.31, stdev=46.81
    clat (usec): min=497, max=9470, avg=2018.20, stdev=511.34
     lat (usec): min=504, max=9476, avg=2039.07, stdev=512.04
    clat percentiles (usec):
     |  1.00th=[  963],  5.00th=[ 1352], 10.00th=[ 1532], 20.00th=[ 1795],
     | 30.00th=[ 1893], 40.00th=[ 1942], 50.00th=[ 2008], 60.00th=[ 2057],
     | 70.00th=[ 2114], 80.00th=[ 2212], 90.00th=[ 2311], 95.00th=[ 2507],
     | 99.00th=[ 4293], 99.50th=[ 5080], 99.90th=[ 6718], 99.95th=[ 7046],
     | 99.99th=[ 8160]
   bw (  KiB/s): min=983232, max=1025024, per=100.00%, avg=1004128.00, stdev=29551.41, samples=2
   iops        : min=30726, max=32032, avg=31379.00, stdev=923.48, samples=2
  lat (usec)   : 500=0.01%, 750=0.35%, 1000=0.78%
  lat (msec)   : 2=48.75%, 4=48.86%, 10=1.25%
  cpu          : usr=10.70%, sys=74.79%, ctx=3304, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=977MiB/s (1025MB/s), 977MiB/s-977MiB/s (1025MB/s-1025MB/s), io=1024MiB (1074MB), run=1048-1048msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1806 seconds or 30 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2144 MiB / 2944 MiB / 2116 MiB / 0 MiB / 3 MiB ( 2144 MiB / 2944 MiB / 2116 MiB / 0 MiB / 3 MiB )
2 min passed...2144 MiB / 2976 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2144 MiB / 2976 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2144 MiB / 3136 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2144 MiB / 3680 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2144 MiB / 4032 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2144 MiB / 4032 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2144 MiB / 4032 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2144 MiB / 4064 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2144 MiB / 4384 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2144 MiB / 4768 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2144 MiB / 4768 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2144 MiB / 4768 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2144 MiB / 4768 MiB / 2116 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2144 MiB / 5088 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 320 MiB / 0 MiB / 1088 MiB / 1053 MiB )
16 min passed...2144 MiB / 5568 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2144 MiB / 5792 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2144 MiB / 5792 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2144 MiB / 5792 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2144 MiB / 5984 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2144 MiB / 6496 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2144 MiB / 6784 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2144 MiB / 6784 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2144 MiB / 6784 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2144 MiB / 6880 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2144 MiB / 7360 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2144 MiB / 7840 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2144 MiB / 7904 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2144 MiB / 7904 MiB / 2116 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2144 MiB / 7904 MiB / 2118 MiB / 1088 MiB / 1056 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test X6: 4x async sequential read/write 32K

Fio test on host (part 1):

Code:

root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=759MiB/s][w=24.3k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=24099: Sun Aug 15 02:21:39 2021
  write: IOPS=5076, BW=159MiB/s (166MB/s)(3072MiB/19364msec); 0 zone resets
    slat (usec): min=7, max=15203, avg=22.80, stdev=118.20
    clat (nsec): min=1138, max=1509.5M, avg=161544.85, stdev=7843251.89
     lat (usec): min=51, max=1509.5k, avg=184.72, stdev=7844.13
    clat percentiles (nsec):
     |  1.00th=[    1800],  5.00th=[   50432], 10.00th=[   54528],
     | 20.00th=[   59136], 30.00th=[   63744], 40.00th=[   68096],
     | 50.00th=[   73216], 60.00th=[   78336], 70.00th=[   88576],
     | 80.00th=[  111104], 90.00th=[  152576], 95.00th=[  195584],
     | 99.00th=[  423936], 99.50th=[  765952], 99.90th=[ 5537792],
     | 99.95th=[ 8847360], 99.99th=[23199744]
   bw (  KiB/s): min=59374, max=327536, per=29.37%, avg=190138.25, stdev=83437.48, samples=32
   iops        : min= 1855, max=10235, avg=5941.69, stdev=2607.40, samples=32
  lat (usec)   : 2=1.79%, 4=0.86%, 10=0.02%, 20=0.08%, 50=1.96%
  lat (usec)   : 100=70.82%, 250=21.69%, 500=1.97%, 750=0.30%, 1000=0.08%
  lat (msec)   : 2=0.15%, 4=0.12%, 10=0.11%, 20=0.03%, 50=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=6.29%, sys=9.27%, ctx=105640, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=24100: Sun Aug 15 02:21:39 2021
  write: IOPS=5097, BW=159MiB/s (167MB/s)(3072MiB/19285msec); 0 zone resets
    slat (usec): min=7, max=29512, avg=23.44, stdev=161.75
    clat (nsec): min=1111, max=1509.7M, avg=160425.66, stdev=7842161.70
     lat (usec): min=51, max=1509.7k, avg=184.29, stdev=7843.97
    clat percentiles (nsec):
     |  1.00th=[    1816],  5.00th=[   50432], 10.00th=[   54528],
     | 20.00th=[   59648], 30.00th=[   63744], 40.00th=[   68096],
     | 50.00th=[   73216], 60.00th=[   78336], 70.00th=[   88576],
     | 80.00th=[  111104], 90.00th=[  152576], 95.00th=[  195584],
     | 99.00th=[  419840], 99.50th=[  749568], 99.90th=[ 5537792],
     | 99.95th=[ 7831552], 99.99th=[22675456]
   bw (  KiB/s): min=57408, max=335808, per=29.57%, avg=191455.53, stdev=83375.89, samples=32
   iops        : min= 1794, max=10494, avg=5982.88, stdev=2605.56, samples=32
  lat (usec)   : 2=1.72%, 4=0.96%, 10=0.02%, 20=0.08%, 50=2.02%
  lat (usec)   : 100=70.91%, 250=21.58%, 500=1.91%, 750=0.29%, 1000=0.09%
  lat (msec)   : 2=0.15%, 4=0.11%, 10=0.12%, 20=0.02%, 50=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=7.06%, sys=8.49%, ctx=105710, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=24101: Sun Aug 15 02:21:39 2021
  write: IOPS=5058, BW=158MiB/s (166MB/s)(3072MiB/19435msec); 0 zone resets
    slat (usec): min=6, max=24260, avg=22.59, stdev=150.10
    clat (nsec): min=1112, max=1509.6M, avg=162248.24, stdev=7843792.08
     lat (usec): min=51, max=1509.7k, avg=185.20, stdev=7845.57
    clat percentiles (nsec):
     |  1.00th=[    1848],  5.00th=[   50944], 10.00th=[   54528],
     | 20.00th=[   59648], 30.00th=[   63744], 40.00th=[   68096],
     | 50.00th=[   73216], 60.00th=[   78336], 70.00th=[   88576],
     | 80.00th=[  111104], 90.00th=[  152576], 95.00th=[  195584],
     | 99.00th=[  423936], 99.50th=[  790528], 99.90th=[ 6127616],
     | 99.95th=[ 8847360], 99.99th=[22413312]
   bw (  KiB/s): min=58688, max=330880, per=29.18%, avg=188889.81, stdev=84268.53, samples=32
   iops        : min= 1834, max=10340, avg=5902.75, stdev=2633.38, samples=32
  lat (usec)   : 2=1.54%, 4=1.04%, 10=0.02%, 20=0.07%, 50=1.80%
  lat (usec)   : 100=70.95%, 250=21.87%, 500=1.91%, 750=0.28%, 1000=0.10%
  lat (msec)   : 2=0.15%, 4=0.11%, 10=0.12%, 20=0.03%, 50=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=6.53%, sys=8.94%, ctx=105598, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=24102: Sun Aug 15 02:21:39 2021
  write: IOPS=5072, BW=159MiB/s (166MB/s)(3072MiB/19380msec); 0 zone resets
    slat (usec): min=7, max=16007, avg=22.56, stdev=103.35
    clat (nsec): min=1163, max=1509.6M, avg=162299.88, stdev=7842365.31
     lat (usec): min=51, max=1509.6k, avg=185.20, stdev=7843.42
    clat percentiles (nsec):
     |  1.00th=[    1800],  5.00th=[   50432], 10.00th=[   54528],
     | 20.00th=[   59136], 30.00th=[   63744], 40.00th=[   68096],
     | 50.00th=[   73216], 60.00th=[   78336], 70.00th=[   88576],
     | 80.00th=[  112128], 90.00th=[  152576], 95.00th=[  195584],
     | 99.00th=[  432128], 99.50th=[  839680], 99.90th=[ 5734400],
     | 99.95th=[ 8355840], 99.99th=[24248320]
   bw (  KiB/s): min=58432, max=333376, per=29.31%, avg=189785.97, stdev=84906.53, samples=32
   iops        : min= 1826, max=10418, avg=5930.72, stdev=2653.33, samples=32
  lat (usec)   : 2=1.83%, 4=1.03%, 10=0.02%, 20=0.08%, 50=1.80%
  lat (usec)   : 100=70.46%, 250=21.97%, 500=1.95%, 750=0.32%, 1000=0.10%
  lat (msec)   : 2=0.16%, 4=0.12%, 10=0.12%, 20=0.02%, 50=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=6.37%, sys=9.10%, ctx=105617, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=632MiB/s (663MB/s), 158MiB/s-159MiB/s (166MB/s-167MB/s), io=12.0GiB (12.9GB), run=19285-19435msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test X6: 4x async sequential read/write 32K

Fio test on host (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=350MiB/s][r=11.2k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=26125: Sun Aug 15 02:37:14 2021
  read: IOPS=2860, BW=89.4MiB/s (93.7MB/s)(3072MiB/34362msec)
    slat (usec): min=5, max=135, avg=13.30, stdev= 3.79
    clat (usec): min=122, max=4302, avg=333.64, stdev=45.75
     lat (usec): min=132, max=4316, avg=347.24, stdev=46.10
    clat percentiles (usec):
     |  1.00th=[  265],  5.00th=[  281], 10.00th=[  293], 20.00th=[  306],
     | 30.00th=[  314], 40.00th=[  322], 50.00th=[  330], 60.00th=[  338],
     | 70.00th=[  351], 80.00th=[  359], 90.00th=[  379], 95.00th=[  396],
     | 99.00th=[  429], 99.50th=[  445], 99.90th=[  494], 99.95th=[  553],
     | 99.99th=[ 2040]
   bw (  KiB/s): min=88640, max=96000, per=25.03%, avg=91533.26, stdev=1582.82, samples=68
   iops        : min= 2770, max= 3000, avg=2860.41, stdev=49.46, samples=68
  lat (usec)   : 250=0.12%, 500=99.79%, 750=0.07%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.12%, sys=5.12%, ctx=98318, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=26126: Sun Aug 15 02:37:14 2021
  read: IOPS=2857, BW=89.3MiB/s (93.6MB/s)(3072MiB/34405msec)
    slat (usec): min=5, max=149, avg=13.30, stdev= 3.83
    clat (usec): min=133, max=4805, avg=334.07, stdev=48.27
     lat (usec): min=144, max=4815, avg=347.68, stdev=48.63
    clat percentiles (usec):
     |  1.00th=[  269],  5.00th=[  281], 10.00th=[  293], 20.00th=[  306],
     | 30.00th=[  314], 40.00th=[  322], 50.00th=[  330], 60.00th=[  338],
     | 70.00th=[  351], 80.00th=[  363], 90.00th=[  379], 95.00th=[  396],
     | 99.00th=[  433], 99.50th=[  445], 99.90th=[  498], 99.95th=[  562],
     | 99.99th=[ 2278]
   bw (  KiB/s): min=88448, max=95360, per=25.00%, avg=91437.54, stdev=1479.11, samples=68
   iops        : min= 2764, max= 2980, avg=2857.41, stdev=46.22, samples=68
  lat (usec)   : 250=0.10%, 500=99.80%, 750=0.07%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.19%, sys=5.06%, ctx=98321, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=26127: Sun Aug 15 02:37:14 2021
  read: IOPS=2861, BW=89.4MiB/s (93.8MB/s)(3072MiB/34354msec)
    slat (usec): min=5, max=102, avg=13.32, stdev= 3.72
    clat (usec): min=103, max=4123, avg=333.56, stdev=43.93
     lat (usec): min=117, max=4137, avg=347.18, stdev=44.31
    clat percentiles (usec):
     |  1.00th=[  265],  5.00th=[  281], 10.00th=[  293], 20.00th=[  306],
     | 30.00th=[  314], 40.00th=[  322], 50.00th=[  330], 60.00th=[  338],
     | 70.00th=[  351], 80.00th=[  363], 90.00th=[  379], 95.00th=[  396],
     | 99.00th=[  429], 99.50th=[  445], 99.90th=[  490], 99.95th=[  537],
     | 99.99th=[ 1680]
   bw (  KiB/s): min=88064, max=95616, per=25.03%, avg=91557.84, stdev=1619.65, samples=68
   iops        : min= 2752, max= 2988, avg=2861.18, stdev=50.61, samples=68
  lat (usec)   : 250=0.35%, 500=99.57%, 750=0.06%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.49%, sys=4.79%, ctx=98315, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=26128: Sun Aug 15 02:37:14 2021
  read: IOPS=2859, BW=89.3MiB/s (93.7MB/s)(3072MiB/34382msec)
    slat (usec): min=5, max=112, avg=13.34, stdev= 3.77
    clat (usec): min=92, max=4432, avg=333.84, stdev=51.73
     lat (usec): min=105, max=4448, avg=347.47, stdev=52.10
    clat percentiles (usec):
     |  1.00th=[  260],  5.00th=[  281], 10.00th=[  293], 20.00th=[  306],
     | 30.00th=[  314], 40.00th=[  322], 50.00th=[  330], 60.00th=[  338],
     | 70.00th=[  351], 80.00th=[  363], 90.00th=[  379], 95.00th=[  396],
     | 99.00th=[  433], 99.50th=[  449], 99.90th=[  506], 99.95th=[  644],
     | 99.99th=[ 2507]
   bw (  KiB/s): min=87744, max=97984, per=25.02%, avg=91488.21, stdev=1996.22, samples=68
   iops        : min= 2742, max= 3062, avg=2859.00, stdev=62.38, samples=68
  lat (usec)   : 100=0.01%, 250=0.63%, 500=99.25%, 750=0.08%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.01%
  cpu          : usr=1.86%, sys=5.41%, ctx=98319, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=357MiB/s (375MB/s), 89.3MiB/s-89.4MiB/s (93.6MB/s-93.8MB/s), io=12.0GiB (12.9GB), run=34354-34405msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1858 seconds or 30 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...25344 MiB / 26880 MiB / 25316 MiB / 128 MiB / 140 MiB ( 25344 MiB / 26880 MiB / 25316 MiB / 128 MiB / 140 MiB )
2 min passed...25344 MiB / 26880 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...25344 MiB / 27168 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...25344 MiB / 27552 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...25344 MiB / 27904 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...25344 MiB / 27904 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...25344 MiB / 27904 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...25344 MiB / 28128 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...25344 MiB / 28704 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...25344 MiB / 29120 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...25344 MiB / 29120 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...25344 MiB / 29120 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...25344 MiB / 29280 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...25344 MiB / 29696 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...25344 MiB / 30016 MiB / 25316 MiB / 128 MiB / 140 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...25344 MiB / 30048 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 32 MiB / 0 MiB / 12256 MiB / 12264 MiB )
17 min passed...25344 MiB / 30048 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...25344 MiB / 30176 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...25344 MiB / 30432 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...25344 MiB / 30720 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...25344 MiB / 30720 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...25344 MiB / 30720 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...25344 MiB / 30816 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...25344 MiB / 31360 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...25344 MiB / 31776 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...25344 MiB / 31776 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...25344 MiB / 31776 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...25344 MiB / 31840 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...25344 MiB / 32192 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...25344 MiB / 32640 MiB / 25316 MiB / 12384 MiB / 12404 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test X7: 4x sync sequential read/write 32K

Fio test on host (part 1):

Code:

root@Hypervisor:~/scripts/bench# bash /root/scripts/bench/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=16.5MiB/s][w=527 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11214: Sun Aug 15 02:59:19 2021
  write: IOPS=153, BW=4901KiB/s (5019kB/s)(512MiB/106975msec); 0 zone resets
    clat (usec): min=2358, max=60002, avg=6509.43, stdev=2623.43
     lat (usec): min=2359, max=60003, avg=6510.10, stdev=2623.43
    clat percentiles (usec):
     |  1.00th=[ 4359],  5.00th=[ 4752], 10.00th=[ 4948], 20.00th=[ 5211],
     | 30.00th=[ 5407], 40.00th=[ 5604], 50.00th=[ 5866], 60.00th=[ 6259],
     | 70.00th=[ 6849], 80.00th=[ 7635], 90.00th=[ 8225], 95.00th=[ 9241],
     | 99.00th=[16450], 99.50th=[25297], 99.90th=[35914], 99.95th=[45876],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3200, max= 5632, per=24.99%, avg=4899.48, stdev=494.95, samples=213
   iops        : min=  100, max=  176, avg=153.09, stdev=15.47, samples=213
  lat (msec)   : 4=0.16%, 10=96.70%, 20=2.23%, 50=0.87%, 100=0.03%
  cpu          : usr=0.54%, sys=1.91%, ctx=65653, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11215: Sun Aug 15 02:59:19 2021
  write: IOPS=153, BW=4903KiB/s (5021kB/s)(512MiB/106923msec); 0 zone resets
    clat (usec): min=3094, max=64011, avg=6506.22, stdev=2641.27
     lat (usec): min=3095, max=64012, avg=6506.88, stdev=2641.37
    clat percentiles (usec):
     |  1.00th=[ 4359],  5.00th=[ 4752], 10.00th=[ 4948], 20.00th=[ 5211],
     | 30.00th=[ 5342], 40.00th=[ 5604], 50.00th=[ 5866], 60.00th=[ 6259],
     | 70.00th=[ 6849], 80.00th=[ 7635], 90.00th=[ 8225], 95.00th=[ 9110],
     | 99.00th=[16581], 99.50th=[25560], 99.90th=[36439], 99.95th=[46400],
     | 99.99th=[52691]
   bw (  KiB/s): min= 3264, max= 5632, per=25.01%, avg=4902.29, stdev=495.97, samples=213
   iops        : min=  102, max=  176, avg=153.19, stdev=15.51, samples=213
  lat (msec)   : 4=0.15%, 10=96.76%, 20=2.19%, 50=0.87%, 100=0.04%
  cpu          : usr=0.70%, sys=1.76%, ctx=65669, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11216: Sun Aug 15 02:59:19 2021
  write: IOPS=153, BW=4903KiB/s (5021kB/s)(512MiB/106931msec); 0 zone resets
    clat (usec): min=3078, max=69384, avg=6506.84, stdev=2628.22
     lat (usec): min=3078, max=69384, avg=6507.49, stdev=2628.23
    clat percentiles (usec):
     |  1.00th=[ 4359],  5.00th=[ 4752], 10.00th=[ 4948], 20.00th=[ 5211],
     | 30.00th=[ 5407], 40.00th=[ 5604], 50.00th=[ 5866], 60.00th=[ 6259],
     | 70.00th=[ 6849], 80.00th=[ 7635], 90.00th=[ 8225], 95.00th=[ 9110],
     | 99.00th=[16581], 99.50th=[25035], 99.90th=[38011], 99.95th=[41157],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3200, max= 5568, per=25.00%, avg=4901.63, stdev=495.60, samples=213
   iops        : min=  100, max=  174, avg=153.16, stdev=15.49, samples=213
  lat (msec)   : 4=0.18%, 10=96.84%, 20=2.09%, 50=0.87%, 100=0.02%
  cpu          : usr=0.58%, sys=1.91%, ctx=65620, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11217: Sun Aug 15 02:59:19 2021
  write: IOPS=153, BW=4901KiB/s (5019kB/s)(512MiB/106967msec); 0 zone resets
    clat (usec): min=2332, max=56515, avg=6508.86, stdev=2606.99
     lat (usec): min=2333, max=56516, avg=6509.56, stdev=2607.01
    clat percentiles (usec):
     |  1.00th=[ 4359],  5.00th=[ 4752], 10.00th=[ 4948], 20.00th=[ 5211],
     | 30.00th=[ 5342], 40.00th=[ 5604], 50.00th=[ 5866], 60.00th=[ 6259],
     | 70.00th=[ 6849], 80.00th=[ 7635], 90.00th=[ 8225], 95.00th=[ 9110],
     | 99.00th=[17433], 99.50th=[25560], 99.90th=[35914], 99.95th=[43254],
     | 99.99th=[52691]
   bw (  KiB/s): min= 3264, max= 5632, per=24.99%, avg=4899.64, stdev=495.42, samples=213
   iops        : min=  102, max=  176, avg=153.09, stdev=15.49, samples=213
  lat (msec)   : 4=0.15%, 10=96.74%, 20=2.20%, 50=0.89%, 100=0.02%
  cpu          : usr=0.67%, sys=1.80%, ctx=65678, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=19.1MiB/s (20.1MB/s), 4901KiB/s-4903KiB/s (5019kB/s-5021kB/s), io=2048MiB (2147MB), run=106923-106975msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 15, 2021

Round 2 - Run 5 - Test X7: 4x sync sequential read/write 32K

Fio test on host (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=359MiB/s][r=11.5k IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=10974: Sun Aug 15 03:14:25 2021
  read: IOPS=2860, BW=89.4MiB/s (93.7MB/s)(512MiB/5728msec)
    clat (usec): min=237, max=2633, avg=347.50, stdev=45.65
     lat (usec): min=237, max=2634, avg=347.72, stdev=45.67
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  302], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  396], 95.00th=[  412],
     | 99.00th=[  449], 99.50th=[  465], 99.90th=[  537], 99.95th=[  627],
     | 99.99th=[ 2114]
   bw (  KiB/s): min=87872, max=98048, per=24.99%, avg=91438.55, stdev=2708.51, samples=11
   iops        : min= 2746, max= 3064, avg=2857.45, stdev=84.64, samples=11
  lat (usec)   : 250=0.05%, 500=99.78%, 750=0.13%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.38%, sys=4.68%, ctx=16387, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=10975: Sun Aug 15 03:14:25 2021
  read: IOPS=2860, BW=89.4MiB/s (93.7MB/s)(512MiB/5728msec)
    clat (usec): min=237, max=2758, avg=347.50, stdev=48.00
     lat (usec): min=237, max=2758, avg=347.72, stdev=48.00
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  306], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  396], 95.00th=[  412],
     | 99.00th=[  445], 99.50th=[  461], 99.90th=[  502], 99.95th=[  627],
     | 99.99th=[ 2737]
   bw (  KiB/s): min=87488, max=98304, per=25.00%, avg=91456.00, stdev=2746.49, samples=11
   iops        : min= 2734, max= 3072, avg=2858.00, stdev=85.83, samples=11
  lat (usec)   : 250=0.04%, 500=99.86%, 750=0.09%
  lat (msec)   : 4=0.02%
  cpu          : usr=1.64%, sys=4.38%, ctx=16387, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=10976: Sun Aug 15 03:14:25 2021
  read: IOPS=2858, BW=89.3MiB/s (93.7MB/s)(512MiB/5732msec)
    clat (usec): min=220, max=2766, avg=347.75, stdev=48.38
     lat (usec): min=221, max=2766, avg=347.96, stdev=48.39
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  306], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  396], 95.00th=[  412],
     | 99.00th=[  449], 99.50th=[  461], 99.90th=[  510], 99.95th=[  545],
     | 99.99th=[ 2704]
   bw (  KiB/s): min=88064, max=98176, per=24.99%, avg=91432.73, stdev=2698.84, samples=11
   iops        : min= 2752, max= 3068, avg=2857.27, stdev=84.34, samples=11
  lat (usec)   : 250=0.07%, 500=99.80%, 750=0.10%, 1000=0.01%
  lat (msec)   : 4=0.02%
  cpu          : usr=2.16%, sys=3.89%, ctx=16385, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=10977: Sun Aug 15 03:14:25 2021
  read: IOPS=2860, BW=89.4MiB/s (93.7MB/s)(512MiB/5727msec)
    clat (usec): min=237, max=2221, avg=347.45, stdev=40.50
     lat (usec): min=237, max=2221, avg=347.68, stdev=40.52
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  302], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  396], 95.00th=[  412],
     | 99.00th=[  445], 99.50th=[  461], 99.90th=[  510], 99.95th=[  627],
     | 99.99th=[ 1418]
   bw (  KiB/s): min=88576, max=97920, per=25.00%, avg=91456.00, stdev=2545.56, samples=11
   iops        : min= 2768, max= 3060, avg=2858.00, stdev=79.55, samples=11
  lat (usec)   : 250=0.04%, 500=99.84%, 750=0.10%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.51%, sys=3.51%, ctx=16392, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=357MiB/s (375MB/s), 89.3MiB/s-89.4MiB/s (93.7MB/s-93.7MB/s), io=2048MiB (2147MB), run=5727-5732msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1917 seconds or 31 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~/scripts/bench# bash smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...11520 MiB / 12160 MiB / 11551 MiB / 352 MiB / 331 MiB ( 11520 MiB / 12160 MiB / 11551 MiB / 352 MiB / 331 MiB )
2 min passed...21440 MiB / 22752 MiB / 21450 MiB / 640 MiB / 607 MiB ( 9920 MiB / 10592 MiB / 9899 MiB / 288 MiB / 276 MiB )
3 min passed...21440 MiB / 23072 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...21440 MiB / 23584 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...21440 MiB / 23776 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...21440 MiB / 23776 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...21440 MiB / 23776 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...21440 MiB / 23936 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...21440 MiB / 24320 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...21440 MiB / 24384 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...21440 MiB / 24384 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...21440 MiB / 24384 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...21440 MiB / 24576 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...21440 MiB / 25152 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...21440 MiB / 25312 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...21440 MiB / 25312 MiB / 21450 MiB / 640 MiB / 607 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...21440 MiB / 25312 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 0 MiB / 0 MiB / 2048 MiB / 2048 MiB )
18 min passed...21440 MiB / 25440 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...21440 MiB / 25856 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...21440 MiB / 26112 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...21440 MiB / 26112 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...21440 MiB / 26112 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...21440 MiB / 26240 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...21440 MiB / 26720 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...21440 MiB / 27104 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...21440 MiB / 27168 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...21440 MiB / 27168 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...21440 MiB / 27232 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...21440 MiB / 27552 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...21440 MiB / 28064 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...21440 MiB / 28064 MiB / 21450 MiB / 2688 MiB / 2655 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 15, 2021

Here are the results comparing fio tests inside VM (run 4) vs fio tests run directly on host (run5). Both write to ext4 filesystems, stored on zvols (8K volblocksize) stored on a 4 disks striped mirror pool (ashift=12):

Write Performance (MiB/s):

	Fio run in VM (run 4)	Fio run on host (run 5)
T1: sync 4K random	1,10	1,47
T2: sync 16K random	5,49	6,47
T3: sync 4M sequential	189,00	179,00
T4: async 4K random	94,30	219,00
T5: async 16K random	456,00	907,00
T6: async 4M sequential	379,00	535,00
T7: async 16K random but 50% compressible	490,00	927,00
T8: sync 32K random	9,89	11,20
T9: async 32K random	769,00	1.275,00
TX6: 4x async 32K sequential	506,00	632,00
TX7: 4x sync 32K sequential	18,20	19,10

Keep in mind that not all performances are very accurate. Most async tests finish within 1 or 2 seconds so there are not that much samples to calculate an average. Tests TX6 + TX7 I have added to get some better performance comparisons. These two tests won't write that much in parallel and are writng more data so they run longer and should give more reliable results.

Read Performance (MiB/s):

	Fio run in VM (run 4)	Fio run on host (run 5)
T1: sync 4K random	8,47	15,30
T2: sync 16K random	30,10	39,60
T3: sync 4M sequential	740,00	404,00
T4: async 4K random	284,00	320,00
T5: async 16K random	884,00	886,00
T6: async 4M sequential	1.393,00	1.406,00
T7: async 16K random but 50% compressible	898,00	1.296,00
T8: sync 32K random	56,40	84,50
T9: async 32K random	879,00	977,00
TX6: 4x async 32K sequential	248,00	357,00
TX7: 4x sync 32K sequential	271,00	357,00

Total Write Amplification (factor):

	Fio run in VM (run 4)	Fio run on host (run 5)
T1: sync 4K random	58,47	56,72
T2: sync 16K random	16,56	17,06
T3: sync 4M sequential	4,64	4,70
T4: async 4K random	6,56	6,00
T5: async 16K random	6,09	4,94
T6: async 4M sequential	3,43	2,33
T7: async 16K random but 50% compressible	5,19	2,94
T8: sync 32K random	14,72	13,50
T9: async 32K random	5,19	4,66
TX6: 4x async 32K sequential	2,61	2,44
TX7: 4x sync 32K sequential	12,63	12,36

Conclusion:
As expected, running same tasks inside a VM is slower compared to running them directly on the host because you get additional virtualization overhead. Write amplification is also most of the time lower if not run inside the VM. So because LXCs are run on the host and are not virtualized a LXCs performance and write amplification should be similar to running them directly on the host. But write amplification for sync writes isn't much different for VM vs host so it won'T help much to run write heavy Databases inside a LXC instead of VM. But sync performance would still be better inside a LXC.

Dunuin · Aug 15, 2021

I also tested test TX6 TX7 directly running on the host but on a 4k volblocksize zvol on a 2 disk mirror. I still don't see better performance for sync writes if striping is used:

Write Performance (MiB/s):

	TX6: 4x async 32K sequential	TX7: 4x sync 32K sequential
VM writes to 4K zvol on 2 disk mirror	311,00 MiB/s	23,1 MiB/s
Host writes to 4K zvol on 2 disk mirror	287,00 MiB/s	32,50 MiB/s
VM writes to 8K zvol on 4 disk striped mirror	506,00 MiB/s	18,20 MiB/s
Host writes to 8K zvol on 4 disk striped mirror	632,00 MiB/s	19,10 MiB/s

So basically striping will increase the async but decrease the sync performance. Even if ran diretly on the host and not just inside the guest. I would have thought that atleast on the host I should see better sync write performance on a striped mirror.

TX6 used this fio test for the writes:

fio --directory=/mnt/test/ --name=async_seq_32K --rw=write --bs=32K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=3G

And TX7 this:

fio --directory=/mnt/test/ --name=sync_seq_32K --rw=write --bs=32K --direct=1 --sync=1 --numjobs=4 --ioengine=psync --iodepth=1 --refill_buffers --size=512M

Dunuin · Aug 16, 2021

I also tested "logbias=throughput" vs "logbias=latency". I ran the fio tests (Test 1 + 3 + TX7) directly on the host to a ext4 ontop of a zvol (8k volblocksize) ontop of a 4 disk striped mirror (ashift=12).

Write Performance:

	logbias=throughput	logbias=latency
sync 4k random	1,45 MiB/s	1,47 MiB/s
sync 4M sequential	184,00 MiB/s	179,00 MiB/s
4x sync 32K sequential	18,80 MiB/s	19,10 MiB/s

Total Write Amplification:

	logbias=throughput	logbias=latency
sync 4k random	54,78 x	56,72 x
sync 4M sequential	4,52 x	4,70 x
4x sync 32K sequential	12,03 x	12,36 x

So there is neither a big performance difference nor a write amplification difference.

Dunuin · Aug 16, 2021

This is the pool and VM that will be used in Round 2 Run 6:

Host is a striped mirror of 6x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=16K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 4 are bold)

root@Hypervisor:~# zpool status

Code:

  pool: VMpool12
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool12                                        ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN  ONLINE       0     0     0

errors: No known data errors

root@Hypervisor:~# zpool get all

Code:

NAME      PROPERTY                       VALUE                          SOURCE
VMpool12  size                           558G                           -
VMpool12  capacity                       0%                             -
VMpool12  altroot                        -                              default
VMpool12  health                         ONLINE                         -
VMpool12  guid                           1730272729246680865            -
VMpool12  version                        -                              default
VMpool12  bootfs                         -                              default
VMpool12  delegation                     on                             default
VMpool12  autoreplace                    off                            default
VMpool12  cachefile                      -                              default
VMpool12  failmode                       wait                           default
VMpool12  listsnapshots                  off                            default
VMpool12  autoexpand                     off                            default
VMpool12  dedupratio                     1.00x                          -
VMpool12  free                           556G                           -
VMpool12  allocated                      1.88G                          -
VMpool12  readonly                       off                            -
VMpool12  ashift                         12                             local
VMpool12  comment                        -                              default
VMpool12  expandsize                     -                              -
VMpool12  freeing                        0                              -
VMpool12  fragmentation                  0%                             -
VMpool12  leaked                         0                              -
VMpool12  multihost                      off                            default
VMpool12  checkpoint                     -                              -
VMpool12  load_guid                      18209421268392521673           -
VMpool12  autotrim                       off                            default
VMpool12  feature@async_destroy          enabled                        local
VMpool12  feature@empty_bpobj            active                         local
VMpool12  feature@lz4_compress           active                         local
VMpool12  feature@multi_vdev_crash_dump  enabled                        local
VMpool12  feature@spacemap_histogram     active                         local
VMpool12  feature@enabled_txg            active                         local
VMpool12  feature@hole_birth             active                         local
VMpool12  feature@extensible_dataset     active                         local
VMpool12  feature@embedded_data          active                         local
VMpool12  feature@bookmarks              enabled                        local
VMpool12  feature@filesystem_limits      enabled                        local
VMpool12  feature@large_blocks           enabled                        local
VMpool12  feature@large_dnode            enabled                        local
VMpool12  feature@sha512                 enabled                        local
VMpool12  feature@skein                  enabled                        local
VMpool12  feature@edonr                  enabled                        local
VMpool12  feature@userobj_accounting     active                         local
VMpool12  feature@encryption             active                         local
VMpool12  feature@project_quota          active                         local
VMpool12  feature@device_removal         enabled                        local
VMpool12  feature@obsolete_counts        enabled                        local
VMpool12  feature@zpool_checkpoint       enabled                        local
VMpool12  feature@spacemap_v2            active                         local
VMpool12  feature@allocation_classes     enabled                        local
VMpool12  feature@resilver_defer         enabled                        local
VMpool12  feature@bookmark_v2            enabled                        local
VMpool12  feature@redaction_bookmarks    enabled                        local
VMpool12  feature@redacted_datasets      enabled                        local
VMpool12  feature@bookmark_written       enabled                        local
VMpool12  feature@log_spacemap           active                         local
VMpool12  feature@livelist               enabled                        local
VMpool12  feature@device_rebuild         enabled                        local
VMpool12  feature@zstd_compress          enabled                        local

root@Hypervisor:~# zfs get all VMpool12/VLT/VM

Code:

NAME             PROPERTY              VALUE                  SOURCE
VMpool12/VLT/VM  type                  filesystem             -
VMpool12/VLT/VM  creation              Mon Aug 16 15:58 2021  -
VMpool12/VLT/VM  used                  1.88G                  -
VMpool12/VLT/VM  available             430G                   -
VMpool12/VLT/VM  referenced            192K                   -
VMpool12/VLT/VM  compressratio         1.33x                  -
VMpool12/VLT/VM  mounted               yes                    -
VMpool12/VLT/VM  quota                 none                   default
VMpool12/VLT/VM  reservation           none                   default
VMpool12/VLT/VM  recordsize            128K                   default
VMpool12/VLT/VM  mountpoint            /VMpool12/VLT/VM       default
VMpool12/VLT/VM  sharenfs              off                    default
VMpool12/VLT/VM  checksum              on                     default
VMpool12/VLT/VM  compression           lz4                    inherited from VMpool12
VMpool12/VLT/VM  atime                 off                    inherited from VMpool12
VMpool12/VLT/VM  devices               on                     default
VMpool12/VLT/VM  exec                  on                     default
VMpool12/VLT/VM  setuid                on                     default
VMpool12/VLT/VM  readonly              off                    default
VMpool12/VLT/VM  zoned                 off                    default
VMpool12/VLT/VM  snapdir               hidden                 default
VMpool12/VLT/VM  aclmode               discard                default
VMpool12/VLT/VM  aclinherit            restricted             default
VMpool12/VLT/VM  createtxg             30                     -
VMpool12/VLT/VM  canmount              on                     local
VMpool12/VLT/VM  xattr                 on                     default
VMpool12/VLT/VM  copies                1                      default
VMpool12/VLT/VM  version               5                      -
VMpool12/VLT/VM  utf8only              off                    -
VMpool12/VLT/VM  normalization         none                   -
VMpool12/VLT/VM  casesensitivity       sensitive              -
VMpool12/VLT/VM  vscan                 off                    default
VMpool12/VLT/VM  nbmand                off                    default
VMpool12/VLT/VM  sharesmb              off                    default
VMpool12/VLT/VM  refquota              none                   default
VMpool12/VLT/VM  refreservation        none                   default
VMpool12/VLT/VM  guid                  3556412500475304937    -
VMpool12/VLT/VM  primarycache          metadata               inherited from VMpool12
VMpool12/VLT/VM  secondarycache        all                    default
VMpool12/VLT/VM  usedbysnapshots       0B                     -
VMpool12/VLT/VM  usedbydataset         192K                   -
VMpool12/VLT/VM  usedbychildren        1.88G                  -
VMpool12/VLT/VM  usedbyrefreservation  0B                     -
VMpool12/VLT/VM  logbias               latency                default
VMpool12/VLT/VM  objsetid              388                    -
VMpool12/VLT/VM  dedup                 off                    default
VMpool12/VLT/VM  mlslabel              none                   default
VMpool12/VLT/VM  sync                  standard               inherited from VMpool12
VMpool12/VLT/VM  dnodesize             legacy                 default
VMpool12/VLT/VM  refcompressratio      1.00x                  -
VMpool12/VLT/VM  written               192K                   -
VMpool12/VLT/VM  logicalused           2.49G                  -
VMpool12/VLT/VM  logicalreferenced     69K                    -
VMpool12/VLT/VM  volmode               default                default
VMpool12/VLT/VM  filesystem_limit      none                   default
VMpool12/VLT/VM  snapshot_limit        none                   default
VMpool12/VLT/VM  filesystem_count      none                   default
VMpool12/VLT/VM  snapshot_count        none                   default
VMpool12/VLT/VM  snapdev               hidden                 default
VMpool12/VLT/VM  acltype               off                    default
VMpool12/VLT/VM  context               none                   default
VMpool12/VLT/VM  fscontext             none                   default
VMpool12/VLT/VM  defcontext            none                   default
VMpool12/VLT/VM  rootcontext           none                   default
VMpool12/VLT/VM  relatime              off                    default
VMpool12/VLT/VM  redundant_metadata    all                    default
VMpool12/VLT/VM  overlay               on                     default
VMpool12/VLT/VM  encryption            aes-256-gcm            -
VMpool12/VLT/VM  keylocation           none                   default
VMpool12/VLT/VM  keyformat             passphrase             -
VMpool12/VLT/VM  pbkdf2iters           350000                 -
VMpool12/VLT/VM  encryptionroot        VMpool12/VLT           -
VMpool12/VLT/VM  keystatus             available              -
VMpool12/VLT/VM  special_small_blocks  0                      default

Commands used to create the pool:

Code:

# create mirror of 6 disks (6x Intel S3710 200GB):
zpool create -f -o ashift=12 VMpool12 mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN
# set ZFS options
zpool set autoexpand=off VMpool12
zpool set feature@encryption=enabled VMpool12
zfs set compression=lz4 VMpool12
zfs set sync=standard VMpool12
zfs set canmount=off VMpool12
zfs set atime=off VMpool12
# set quota to 80% so pool can't get full/slow:
zfs set quota=432G VMpool12
# create encrypted dataset (256bit aes):
zfs create -o encryption=on -o keyformat=passphrase VMpool12/VLT
zfs set canmount=off VMpool12/VLT
# create dataset for encrypted VMs
zfs create -o encryption=on VMpool12/VLT/VM
zfs set canmount=on VMpool12/VLT/VM
# add dataset to PVE:
pvesm add zfspool VMpool12_VLT_VM -pool VMpool12/VLT/VM --content images,rootdir --sparse 1 --mountpoint /VMpool12/VLT/VM
# disable ARC caching:
zfs set primarycache=metadata VMpool12
# don't forget to set block size from 8K to 16K using WebUI

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1233KiB/s][w=308 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=771: Mon Aug 16 16:19:40 2021
  write: IOPS=295, BW=1181KiB/s (1209kB/s)(1024MiB/887913msec); 0 zone resets
    clat (usec): min=1270, max=35364, avg=3372.06, stdev=984.91
     lat (usec): min=1271, max=35366, avg=3374.27, stdev=984.92
    clat percentiles (usec):
     |  1.00th=[ 2114],  5.00th=[ 2409], 10.00th=[ 2573], 20.00th=[ 2802],
     | 30.00th=[ 2999], 40.00th=[ 3163], 50.00th=[ 3326], 60.00th=[ 3458],
     | 70.00th=[ 3589], 80.00th=[ 3720], 90.00th=[ 3916], 95.00th=[ 4359],
     | 99.00th=[ 6128], 99.50th=[ 6587], 99.90th=[15270], 99.95th=[20841],
     | 99.99th=[28967]
   bw (  KiB/s): min=  928, max= 1752, per=100.00%, avg=1180.82, stdev=104.94, samples=1775
   iops        : min=  232, max=  438, avg=295.20, stdev=26.24, samples=1775
  lat (msec)   : 2=0.46%, 4=91.89%, 10=7.41%, 20=0.19%, 50=0.05%
  cpu          : usr=0.71%, sys=4.14%, ctx=640072, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1181KiB/s (1209kB/s), 1181KiB/s-1181KiB/s (1209kB/s-1209kB/s), io=1024MiB (1074MB), run=887913-887913msec

Disk stats (read/write):
  sda: ios=3/797416, merge=0/552895, ticks=3/842002, in_queue=895396, util=99.95%
0 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 5272 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5272 MiB / 17 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=8904KiB/s][r=2226 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=985: Mon Aug 16 16:37:01 2021
  read: IOPS=1869, BW=7479KiB/s (7659kB/s)(1024MiB/140195msec)
    clat (usec): min=233, max=7792, avg=524.02, stdev=125.65
     lat (usec): min=234, max=7794, avg=526.01, stdev=125.74
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  363], 10.00th=[  388], 20.00th=[  424],
     | 30.00th=[  457], 40.00th=[  486], 50.00th=[  519], 60.00th=[  553],
     | 70.00th=[  586], 80.00th=[  627], 90.00th=[  660], 95.00th=[  685],
     | 99.00th=[  717], 99.50th=[  725], 99.90th=[ 1811], 99.95th=[ 2671],
     | 99.99th=[ 3228]
   bw (  KiB/s): min= 5944, max=11432, per=99.96%, avg=7476.21, stdev=1032.87, samples=280
   iops        : min= 1486, max= 2858, avg=1869.05, stdev=258.22, samples=280
  lat (usec)   : 250=0.01%, 500=44.52%, 750=55.22%, 1000=0.09%
  lat (msec)   : 2=0.08%, 4=0.09%, 10=0.01%
  cpu          : usr=3.13%, sys=9.79%, ctx=262312, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=7479KiB/s (7659kB/s), 7479KiB/s-7479KiB/s (7659kB/s-7659kB/s), io=1024MiB (1074MB), run=140195-140195msec

Disk stats (read/write):
  sda: ios=262025/12, merge=0/9, ticks=127694/13, in_queue=140252, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5272 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Script ran 2830 seconds or 47 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3488 MiB / 4352 MiB / 3525 MiB / 288 MiB / 286 MiB ( 3488 MiB / 4352 MiB / 3525 MiB / 288 MiB / 286 MiB )
2 min passed...7008 MiB / 8192 MiB / 7030 MiB / 640 MiB / 644 MiB ( 3520 MiB / 3840 MiB / 3505 MiB / 352 MiB / 358 MiB )
3 min passed...10560 MiB / 11872 MiB / 10524 MiB / 1056 MiB / 1052 MiB ( 3552 MiB / 3680 MiB / 3494 MiB / 416 MiB / 408 MiB )
4 min passed...13888 MiB / 15488 MiB / 13925 MiB / 1504 MiB / 1495 MiB ( 3328 MiB / 3616 MiB / 3401 MiB / 448 MiB / 443 MiB )
5 min passed...17312 MiB / 19744 MiB / 17335 MiB / 1952 MiB / 1978 MiB ( 3424 MiB / 4256 MiB / 3410 MiB / 448 MiB / 483 MiB )
6 min passed...20736 MiB / 23776 MiB / 20774 MiB / 2496 MiB / 2501 MiB ( 3424 MiB / 4032 MiB / 3439 MiB / 544 MiB / 523 MiB )
7 min passed...24160 MiB / 27648 MiB / 24164 MiB / 3040 MiB / 3045 MiB ( 3424 MiB / 3872 MiB / 3390 MiB / 544 MiB / 544 MiB )
8 min passed...27488 MiB / 31200 MiB / 27512 MiB / 3616 MiB / 3616 MiB ( 3328 MiB / 3552 MiB / 3348 MiB / 576 MiB / 571 MiB )
9 min passed...30912 MiB / 34848 MiB / 30950 MiB / 4192 MiB / 4202 MiB ( 3424 MiB / 3648 MiB / 3438 MiB / 576 MiB / 586 MiB )
10 min passed...34304 MiB / 38528 MiB / 34295 MiB / 4768 MiB / 4769 MiB ( 3392 MiB / 3680 MiB / 3345 MiB / 576 MiB / 567 MiB )
11 min passed...37632 MiB / 42752 MiB / 37642 MiB / 5344 MiB / 5327 MiB ( 3328 MiB / 4224 MiB / 3347 MiB / 576 MiB / 558 MiB )
12 min passed...40896 MiB / 46496 MiB / 40922 MiB / 5888 MiB / 5877 MiB ( 3264 MiB / 3744 MiB / 3280 MiB / 544 MiB / 550 MiB )
13 min passed...44352 MiB / 50080 MiB / 44326 MiB / 6464 MiB / 6452 MiB ( 3456 MiB / 3584 MiB / 3404 MiB / 576 MiB / 575 MiB )
14 min passed...47616 MiB / 53888 MiB / 47639 MiB / 7008 MiB / 7008 MiB ( 3264 MiB / 3808 MiB / 3313 MiB / 544 MiB / 556 MiB )
15 min passed...50016 MiB / 56448 MiB / 50013 MiB / 7360 MiB / 7357 MiB ( 2400 MiB / 2560 MiB / 2374 MiB / 352 MiB / 349 MiB )
16 min passed...50016 MiB / 56800 MiB / 50016 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 352 MiB / 3 MiB / 0 MiB / 1 MiB )
17 min passed...50016 MiB / 57568 MiB / 50016 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...50016 MiB / 57728 MiB / 50016 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...50016 MiB / 57728 MiB / 50016 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...50016 MiB / 57728 MiB / 50016 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...50016 MiB / 57792 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 64 MiB / 6 MiB / 0 MiB / 0 MiB )
22 min passed...50016 MiB / 58432 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...50016 MiB / 58944 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...50016 MiB / 58944 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...50016 MiB / 59008 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...50016 MiB / 59136 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...50016 MiB / 59488 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...50016 MiB / 60064 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...50016 MiB / 60416 MiB / 50022 MiB / 7360 MiB / 7358 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...50016 MiB / 60416 MiB / 50022 MiB / 8064 MiB / 8094 MiB ( 0 MiB / 0 MiB / 0 MiB / 704 MiB / 736 MiB )
31 min passed...50016 MiB / 60480 MiB / 50030 MiB / 9824 MiB / 9836 MiB ( 0 MiB / 64 MiB / 8 MiB / 1760 MiB / 1742 MiB )
32 min passed...50016 MiB / 60672 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 192 MiB / 0 MiB / 1632 MiB / 1618 MiB )
33 min passed...50016 MiB / 61184 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...50016 MiB / 61472 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...50016 MiB / 61760 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...50016 MiB / 61824 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...50016 MiB / 61952 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...50016 MiB / 62592 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...50016 MiB / 63008 MiB / 50030 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...50016 MiB / 63264 MiB / 50032 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
41 min passed...50048 MiB / 63584 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 32 MiB / 320 MiB / 4 MiB / 0 MiB / 0 MiB )
42 min passed...50048 MiB / 63616 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...50048 MiB / 63904 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...50048 MiB / 64672 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...50048 MiB / 64864 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...50048 MiB / 64992 MiB / 50036 MiB / 11456 MiB / 11454 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
47 min passed...50080 MiB / 65184 MiB / 50061 MiB / 11456 MiB / 11454 MiB ( 32 MiB / 192 MiB / 25 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=6614KiB/s][w=413 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1217: Mon Aug 16 16:57:33 2021
  write: IOPS=362, BW=5797KiB/s (5936kB/s)(1024MiB/180894msec); 0 zone resets
    clat (usec): min=1519, max=38197, avg=2740.16, stdev=774.52
     lat (usec): min=1520, max=38200, avg=2742.40, stdev=774.55
    clat percentiles (usec):
     |  1.00th=[ 1926],  5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2409],
     | 30.00th=[ 2507], 40.00th=[ 2573], 50.00th=[ 2671], 60.00th=[ 2769],
     | 70.00th=[ 2835], 80.00th=[ 2933], 90.00th=[ 3097], 95.00th=[ 3261],
     | 99.00th=[ 5014], 99.50th=[ 5473], 99.90th=[12125], 99.95th=[16319],
     | 99.99th=[27395]
   bw (  KiB/s): min= 4608, max= 7776, per=100.00%, avg=5796.70, stdev=468.94, samples=361
   iops        : min=  288, max=  486, avg=362.29, stdev=29.31, samples=361
  lat (msec)   : 2=1.93%, 4=95.39%, 10=2.49%, 20=0.16%, 50=0.03%
  cpu          : usr=1.04%, sys=4.66%, ctx=163197, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5797KiB/s (5936kB/s), 5797KiB/s-5797KiB/s (5936kB/s-5936kB/s), io=1024MiB (1074MB), run=180894-180894msec

Disk stats (read/write):
  sda: ios=0/196792, merge=0/135405, ticks=0/165597, in_queue=180632, util=99.94%
0 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 2068 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=29.5MiB/s][r=1886 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1420: Mon Aug 16 17:13:10 2021
  read: IOPS=1791, BW=27.0MiB/s (29.3MB/s)(1024MiB/36586msec)
    clat (usec): min=276, max=3519, avg=547.67, stdev=138.56
     lat (usec): min=277, max=3521, avg=549.63, stdev=138.61
    clat percentiles (usec):
     |  1.00th=[  351],  5.00th=[  388], 10.00th=[  416], 20.00th=[  453],
     | 30.00th=[  486], 40.00th=[  519], 50.00th=[  545], 60.00th=[  570],
     | 70.00th=[  603], 80.00th=[  635], 90.00th=[  668], 95.00th=[  693],
     | 99.00th=[  725], 99.50th=[  758], 99.90th=[ 2704], 99.95th=[ 2999],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=24768, max=36896, per=100.00%, avg=28661.04, stdev=3184.84, samples=73
   iops        : min= 1548, max= 2306, avg=1791.29, stdev=199.07, samples=73
  lat (usec)   : 500=34.40%, 750=65.08%, 1000=0.15%
  lat (msec)   : 2=0.18%, 4=0.19%
  cpu          : usr=3.22%, sys=9.12%, ctx=65655, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=27.0MiB/s (29.3MB/s), 27.0MiB/s-27.0MiB/s (29.3MB/s-29.3MB/s), io=1024MiB (1074MB), run=36586-36586msec

Disk stats (read/write):
  sda: ios=65218/0, merge=0/0, ticks=33308/0, in_queue=36476, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 2019 seconds or 33 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4032 MiB / 4704 MiB / 4038 MiB / 192 MiB / 235 MiB ( 4032 MiB / 4704 MiB / 4038 MiB / 192 MiB / 235 MiB )
2 min passed...8352 MiB / 9440 MiB / 8408 MiB / 512 MiB / 508 MiB ( 4320 MiB / 4736 MiB / 4370 MiB / 320 MiB / 273 MiB )
3 min passed...12544 MiB / 14400 MiB / 12609 MiB / 736 MiB / 702 MiB ( 4192 MiB / 4960 MiB / 4201 MiB / 224 MiB / 194 MiB )
4 min passed...12704 MiB / 14560 MiB / 12718 MiB / 736 MiB / 705 MiB ( 160 MiB / 160 MiB / 109 MiB / 0 MiB / 3 MiB )
5 min passed...12704 MiB / 14976 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...12704 MiB / 15456 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...12704 MiB / 15744 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...12704 MiB / 15904 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...12704 MiB / 16128 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...12704 MiB / 16352 MiB / 12718 MiB / 736 MiB / 705 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...12704 MiB / 16736 MiB / 12720 MiB / 736 MiB / 705 MiB ( 0 MiB / 384 MiB / 2 MiB / 0 MiB / 0 MiB )
12 min passed...12704 MiB / 17184 MiB / 12723 MiB / 736 MiB / 705 MiB ( 0 MiB / 448 MiB / 3 MiB / 0 MiB / 0 MiB )
13 min passed...12704 MiB / 17312 MiB / 12723 MiB / 736 MiB / 705 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...12704 MiB / 17600 MiB / 12723 MiB / 736 MiB / 705 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...12704 MiB / 17888 MiB / 12725 MiB / 736 MiB / 705 MiB ( 0 MiB / 288 MiB / 2 MiB / 0 MiB / 0 MiB )
16 min passed...12704 MiB / 18304 MiB / 12726 MiB / 736 MiB / 705 MiB ( 0 MiB / 416 MiB / 1 MiB / 0 MiB / 0 MiB )
17 min passed...12704 MiB / 18496 MiB / 12726 MiB / 736 MiB / 705 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...12704 MiB / 18880 MiB / 12726 MiB / 800 MiB / 870 MiB ( 0 MiB / 384 MiB / 0 MiB / 64 MiB / 165 MiB )
19 min passed...12704 MiB / 19104 MiB / 12726 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 224 MiB / 0 MiB / 928 MiB / 859 MiB )
20 min passed...12704 MiB / 19328 MiB / 12726 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...12704 MiB / 19680 MiB / 12730 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 352 MiB / 4 MiB / 0 MiB / 0 MiB )
22 min passed...12704 MiB / 20064 MiB / 12732 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 384 MiB / 2 MiB / 0 MiB / 0 MiB )
23 min passed...12704 MiB / 20288 MiB / 12736 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 224 MiB / 4 MiB / 0 MiB / 0 MiB )
24 min passed...12704 MiB / 20512 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 224 MiB / 1 MiB / 0 MiB / 0 MiB )
25 min passed...12704 MiB / 20864 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...12704 MiB / 21088 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...12704 MiB / 21760 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...12704 MiB / 22112 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...12704 MiB / 22176 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...12704 MiB / 22432 MiB / 12737 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...12704 MiB / 22944 MiB / 12744 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 512 MiB / 7 MiB / 0 MiB / 0 MiB )
32 min passed...12704 MiB / 23136 MiB / 12744 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...12704 MiB / 23744 MiB / 12744 MiB / 1728 MiB / 1729 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=260MiB/s][w=65 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1655: Mon Aug 16 17:32:34 2021
  write: IOPS=52, BW=211MiB/s (222MB/s)(10.0GiB/48425msec); 0 zone resets
    clat (msec): min=11, max=149, avg=17.17, stdev= 7.92
     lat (msec): min=11, max=149, avg=17.18, stdev= 7.92
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   13], 10.00th=[   13], 20.00th=[   14],
     | 30.00th=[   14], 40.00th=[   14], 50.00th=[   14], 60.00th=[   14],
     | 70.00th=[   15], 80.00th=[   25], 90.00th=[   28], 95.00th=[   29],
     | 99.00th=[   35], 99.50th=[   53], 99.90th=[  115], 99.95th=[  120],
     | 99.99th=[  150]
   bw (  KiB/s): min=106496, max=278528, per=99.71%, avg=215908.14, stdev=58070.67, samples=96
   iops        : min=   26, max=   68, avg=52.67, stdev=14.17, samples=96
  lat (msec)   : 20=74.18%, 50=25.31%, 100=0.35%, 250=0.16%
  cpu          : usr=8.71%, sys=2.81%, ctx=6524, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=211MiB/s (222MB/s), 211MiB/s-211MiB/s (222MB/s-222MB/s), io=10.0GiB (10.7GB), run=48425-48425msec

Disk stats (read/write):
  sda: ios=0/15360, merge=0/5284, ticks=0/59686, in_queue=65644, util=98.15%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 10281 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=997MiB/s][r=249 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1858: Mon Aug 16 17:47:45 2021
  read: IOPS=248, BW=994MiB/s (1042MB/s)(10.0GiB/10300msec)
    clat (usec): min=3417, max=7701, avg=4013.25, stdev=262.47
     lat (usec): min=3419, max=7703, avg=4015.07, stdev=262.49
    clat percentiles (usec):
     |  1.00th=[ 3654],  5.00th=[ 3752], 10.00th=[ 3785], 20.00th=[ 3851],
     | 30.00th=[ 3884], 40.00th=[ 3916], 50.00th=[ 3982], 60.00th=[ 4015],
     | 70.00th=[ 4080], 80.00th=[ 4146], 90.00th=[ 4293], 95.00th=[ 4424],
     | 99.00th=[ 4948], 99.50th=[ 5276], 99.90th=[ 6259], 99.95th=[ 7308],
     | 99.99th=[ 7701]
   bw (  KiB/s): min=999424, max=1032192, per=100.00%, avg=1018163.40, stdev=9189.70, samples=20
   iops        : min=  244, max=  252, avg=248.55, stdev= 2.24, samples=20
  lat (msec)   : 4=58.32%, 10=41.68%
  cpu          : usr=0.37%, sys=6.09%, ctx=2561, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=994MiB/s (1042MB/s), 994MiB/s-994MiB/s (1042MB/s-1042MB/s), io=10.0GiB (10.7GB), run=10300-10300msec

Disk stats (read/write):
  sda: ios=10088/2, merge=0/2, ticks=34282/3, in_queue=36804, util=99.52%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1860 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...42400 MiB / 45600 MiB / 42439 MiB / 0 MiB / 21 MiB ( 42400 MiB / 45600 MiB / 42439 MiB / 0 MiB / 21 MiB )
2 min passed...42400 MiB / 46144 MiB / 42440 MiB / 0 MiB / 21 MiB ( 0 MiB / 544 MiB / 1 MiB / 0 MiB / 0 MiB )
3 min passed...42400 MiB / 46144 MiB / 42440 MiB / 0 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...42400 MiB / 46176 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 32 MiB / 6 MiB / 0 MiB / 0 MiB )
5 min passed...42400 MiB / 46944 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...42400 MiB / 47296 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...42400 MiB / 47712 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...42400 MiB / 48096 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...42400 MiB / 48096 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...42400 MiB / 48224 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...42400 MiB / 48480 MiB / 42446 MiB / 0 MiB / 21 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...42400 MiB / 48864 MiB / 42449 MiB / 0 MiB / 21 MiB ( 0 MiB / 384 MiB / 3 MiB / 0 MiB / 0 MiB )
13 min passed...42400 MiB / 49152 MiB / 42456 MiB / 0 MiB / 21 MiB ( 0 MiB / 288 MiB / 7 MiB / 0 MiB / 0 MiB )
14 min passed...42464 MiB / 49376 MiB / 42467 MiB / 0 MiB / 21 MiB ( 64 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
15 min passed...42464 MiB / 49376 MiB / 42478 MiB / 0 MiB / 21 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...42496 MiB / 49472 MiB / 42489 MiB / 10560 MiB / 10569 MiB ( 32 MiB / 96 MiB / 11 MiB / 10560 MiB / 10548 MiB )
17 min passed...42560 MiB / 49792 MiB / 42499 MiB / 10560 MiB / 10569 MiB ( 64 MiB / 320 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...42560 MiB / 49952 MiB / 42508 MiB / 10560 MiB / 10569 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...42560 MiB / 50176 MiB / 42518 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 1 MiB )
20 min passed...42560 MiB / 50400 MiB / 42529 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...42560 MiB / 50464 MiB / 42538 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...42560 MiB / 50848 MiB / 42548 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 384 MiB / 10 MiB / 0 MiB / 0 MiB )
23 min passed...42560 MiB / 51104 MiB / 42558 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...42560 MiB / 51104 MiB / 42568 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
25 min passed...42592 MiB / 51456 MiB / 42580 MiB / 10560 MiB / 10570 MiB ( 32 MiB / 352 MiB / 12 MiB / 0 MiB / 0 MiB )
26 min passed...42592 MiB / 51616 MiB / 42589 MiB / 10560 MiB / 10570 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
27 min passed...42592 MiB / 51776 MiB / 42599 MiB / 10560 MiB / 10571 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 1 MiB )
28 min passed...42592 MiB / 52096 MiB / 42607 MiB / 10560 MiB / 10571 MiB ( 0 MiB / 320 MiB / 8 MiB / 0 MiB / 0 MiB )
29 min passed...42592 MiB / 52160 MiB / 42615 MiB / 10560 MiB / 10571 MiB ( 0 MiB / 64 MiB / 8 MiB / 0 MiB / 0 MiB )
30 min passed...42592 MiB / 52224 MiB / 42624 MiB / 10560 MiB / 10571 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
31 min passed...42592 MiB / 52576 MiB / 42638 MiB / 10560 MiB / 10571 MiB ( 0 MiB / 352 MiB / 14 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=123MiB/s][w=31.5k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2090: Mon Aug 16 18:07:45 2021
  write: IOPS=27.0k, BW=109MiB/s (115MB/s)(1024MiB/9369msec); 0 zone resets
    slat (usec): min=6, max=2919, avg=22.52, stdev=19.72
    clat (usec): min=621, max=84560, avg=2257.56, stdev=1925.16
     lat (usec): min=636, max=84599, avg=2281.49, stdev=1926.52
    clat percentiles (usec):
     |  1.00th=[ 1205],  5.00th=[ 1549], 10.00th=[ 1614], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1762], 50.00th=[ 1811], 60.00th=[ 1893],
     | 70.00th=[ 2089], 80.00th=[ 2245], 90.00th=[ 2769], 95.00th=[ 4686],
     | 99.00th=[ 9372], 99.50th=[10421], 99.90th=[20055], 99.95th=[25560],
     | 99.99th=[77071]
   bw (  KiB/s): min=47008, max=140432, per=98.91%, avg=110704.56, stdev=31469.35, samples=18
   iops        : min=11752, max=35108, avg=27676.06, stdev=7867.35, samples=18
  lat (usec)   : 750=0.02%, 1000=0.34%
  lat (msec)   : 2=65.06%, 4=28.85%, 10=5.04%, 20=0.58%, 50=0.08%
  lat (msec)   : 100=0.03%
  cpu          : usr=17.68%, sys=50.59%, ctx=209703, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=1024MiB (1074MB), run=9369-9369msec

Disk stats (read/write):
  sda: ios=0/254350, merge=0/1217, ticks=0/167837, in_queue=170060, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 0 MiB ( 1029 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=279MiB/s][r=71.5k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2338: Mon Aug 16 18:22:49 2021
  read: IOPS=69.5k, BW=271MiB/s (285MB/s)(1024MiB/3773msec)
    slat (usec): min=3, max=273, avg= 7.48, stdev= 3.96
    clat (usec): min=262, max=4966, avg=908.54, stdev=108.77
     lat (usec): min=268, max=4972, avg=917.24, stdev=109.57
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  791], 10.00th=[  816], 20.00th=[  840],
     | 30.00th=[  857], 40.00th=[  873], 50.00th=[  889], 60.00th=[  906],
     | 70.00th=[  922], 80.00th=[  955], 90.00th=[ 1020], 95.00th=[ 1156],
     | 99.00th=[ 1205], 99.50th=[ 1237], 99.90th=[ 1795], 99.95th=[ 2343],
     | 99.99th=[ 2933]
   bw (  KiB/s): min=252328, max=286896, per=99.89%, avg=277602.29, stdev=11685.58, samples=7
   iops        : min=63082, max=71724, avg=69400.57, stdev=2921.39, samples=7
  lat (usec)   : 500=0.01%, 750=1.02%, 1000=87.62%
  lat (msec)   : 2=11.29%, 4=0.07%, 10=0.01%
  cpu          : usr=42.07%, sys=57.90%, ctx=14, majf=0, minf=73
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=271MiB/s (285MB/s), 271MiB/s-271MiB/s (285MB/s-285MB/s), io=1024MiB (1074MB), run=3773-3773msec

Disk stats (read/write):
  sda: ios=253227/0, merge=0/0, ticks=74772/0, in_queue=76100, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1815 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4608 MiB / 5216 MiB / 4583 MiB / 1536 MiB / 1531 MiB ( 4608 MiB / 5216 MiB / 4583 MiB / 1536 MiB / 1531 MiB )
2 min passed...4608 MiB / 5248 MiB / 4597 MiB / 1536 MiB / 1531 MiB ( 0 MiB / 32 MiB / 14 MiB / 0 MiB / 0 MiB )
3 min passed...4608 MiB / 5312 MiB / 4607 MiB / 1536 MiB / 1531 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
4 min passed...4608 MiB / 5600 MiB / 4618 MiB / 1536 MiB / 1531 MiB ( 0 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...4608 MiB / 5632 MiB / 4628 MiB / 1536 MiB / 1531 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...4608 MiB / 5664 MiB / 4639 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 1 MiB )
7 min passed...4608 MiB / 6176 MiB / 4649 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 512 MiB / 10 MiB / 0 MiB / 0 MiB )
8 min passed...4608 MiB / 6272 MiB / 4659 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
9 min passed...4608 MiB / 6560 MiB / 4670 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...4608 MiB / 6688 MiB / 4682 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 128 MiB / 12 MiB / 0 MiB / 0 MiB )
11 min passed...4640 MiB / 6688 MiB / 4691 MiB / 1536 MiB / 1532 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...4640 MiB / 6976 MiB / 4701 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...4640 MiB / 7488 MiB / 4710 MiB / 1536 MiB / 1532 MiB ( 0 MiB / 512 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...4640 MiB / 7616 MiB / 4721 MiB / 1536 MiB / 1533 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 1 MiB )
15 min passed...4704 MiB / 7712 MiB / 4731 MiB / 1536 MiB / 1533 MiB ( 64 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
16 min passed...4704 MiB / 7712 MiB / 4741 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 0 MiB / 10 MiB / 4064 MiB / 4073 MiB )
17 min passed...4736 MiB / 7840 MiB / 4751 MiB / 5600 MiB / 5606 MiB ( 32 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...4736 MiB / 8128 MiB / 4760 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...4768 MiB / 8320 MiB / 4770 MiB / 5600 MiB / 5606 MiB ( 32 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
20 min passed...4800 MiB / 8320 MiB / 4779 MiB / 5600 MiB / 5606 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...4800 MiB / 8512 MiB / 4788 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...4800 MiB / 8704 MiB / 4799 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...4800 MiB / 8768 MiB / 4808 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
24 min passed...4800 MiB / 9184 MiB / 4818 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 416 MiB / 10 MiB / 0 MiB / 0 MiB )
25 min passed...4800 MiB / 9184 MiB / 4826 MiB / 5600 MiB / 5606 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
26 min passed...4800 MiB / 9184 MiB / 4838 MiB / 5600 MiB / 5607 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
27 min passed...4800 MiB / 9440 MiB / 4847 MiB / 5600 MiB / 5607 MiB ( 0 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
28 min passed...4800 MiB / 9632 MiB / 4861 MiB / 5600 MiB / 5607 MiB ( 0 MiB / 192 MiB / 14 MiB / 0 MiB / 0 MiB )
29 min passed...4800 MiB / 9664 MiB / 4871 MiB / 5600 MiB / 5607 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...4832 MiB / 10144 MiB / 4883 MiB / 5600 MiB / 5607 MiB ( 32 MiB / 480 MiB / 12 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2571: Mon Aug 16 18:39:49 2021
  write: IOPS=28.5k, BW=446MiB/s (468MB/s)(1024MiB/2296msec); 0 zone resets
    slat (usec): min=7, max=1494, avg=20.04, stdev=17.15
    clat (usec): min=68, max=32766, avg=2207.48, stdev=945.57
     lat (usec): min=79, max=32809, avg=2228.74, stdev=948.67
    clat percentiles (usec):
     |  1.00th=[ 1795],  5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1926],
     | 30.00th=[ 1975], 40.00th=[ 1991], 50.00th=[ 2024], 60.00th=[ 2073],
     | 70.00th=[ 2114], 80.00th=[ 2245], 90.00th=[ 2737], 95.00th=[ 3163],
     | 99.00th=[ 4293], 99.50th=[ 4686], 99.90th=[ 9896], 99.95th=[28443],
     | 99.99th=[30278]
   bw (  KiB/s): min=401312, max=516448, per=100.00%, avg=478168.00, stdev=52049.08, samples=4
   iops        : min=25082, max=32278, avg=29885.50, stdev=3253.07, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=40.80%, 4=57.85%, 10=1.22%, 20=0.01%, 50=0.09%
  cpu          : usr=23.97%, sys=54.38%, ctx=69014, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=446MiB/s (468MB/s), 446MiB/s-446MiB/s (468MB/s-468MB/s), io=1024MiB (1074MB), run=2296-2296msec

Disk stats (read/write):
  sda: ios=0/62520, merge=0/0, ticks=0/2813, in_queue=3716, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2785: Mon Aug 16 18:54:51 2021
  read: IOPS=61.7k, BW=963MiB/s (1010MB/s)(1024MiB/1063msec)
    slat (usec): min=4, max=275, avg= 8.90, stdev= 5.28
    clat (usec): min=286, max=6953, avg=1022.06, stdev=201.43
     lat (usec): min=303, max=6958, avg=1032.21, stdev=202.66
    clat percentiles (usec):
     |  1.00th=[  807],  5.00th=[  857], 10.00th=[  881], 20.00th=[  914],
     | 30.00th=[  938], 40.00th=[  955], 50.00th=[  979], 60.00th=[ 1004],
     | 70.00th=[ 1037], 80.00th=[ 1106], 90.00th=[ 1172], 95.00th=[ 1270],
     | 99.00th=[ 1762], 99.50th=[ 2147], 99.90th=[ 2835], 99.95th=[ 3523],
     | 99.99th=[ 5014]
   bw (  KiB/s): min=946400, max=1017152, per=99.53%, avg=981776.00, stdev=50029.22, samples=2
   iops        : min=59150, max=63572, avg=61361.00, stdev=3126.83, samples=2
  lat (usec)   : 500=0.04%, 750=0.36%, 1000=58.92%
  lat (msec)   : 2=40.10%, 4=0.56%, 10=0.03%
  cpu          : usr=36.16%, sys=63.37%, ctx=44, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=963MiB/s (1010MB/s), 963MiB/s-963MiB/s (1010MB/s-1010MB/s), io=1024MiB (1074MB), run=1063-1063msec

Disk stats (read/write):
  sda: ios=54648/0, merge=0/0, ticks=17078/0, in_queue=16144, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2336 MiB / 2592 MiB / 2317 MiB / 0 MiB / 1 MiB ( 2336 MiB / 2592 MiB / 2317 MiB / 0 MiB / 1 MiB )
2 min passed...2368 MiB / 2752 MiB / 2327 MiB / 0 MiB / 1 MiB ( 32 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
3 min passed...2368 MiB / 3296 MiB / 2338 MiB / 0 MiB / 1 MiB ( 0 MiB / 544 MiB / 11 MiB / 0 MiB / 0 MiB )
4 min passed...2432 MiB / 3712 MiB / 2347 MiB / 0 MiB / 1 MiB ( 64 MiB / 416 MiB / 9 MiB / 0 MiB / 0 MiB )
5 min passed...2432 MiB / 3712 MiB / 2358 MiB / 0 MiB / 1 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
6 min passed...2464 MiB / 3744 MiB / 2369 MiB / 0 MiB / 2 MiB ( 32 MiB / 32 MiB / 11 MiB / 0 MiB / 1 MiB )
7 min passed...2464 MiB / 3936 MiB / 2380 MiB / 0 MiB / 2 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
8 min passed...2464 MiB / 4160 MiB / 2390 MiB / 0 MiB / 2 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
9 min passed...2464 MiB / 4448 MiB / 2400 MiB / 0 MiB / 2 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
10 min passed...2464 MiB / 4512 MiB / 2410 MiB / 0 MiB / 2 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...2464 MiB / 4512 MiB / 2419 MiB / 0 MiB / 2 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...2464 MiB / 4608 MiB / 2427 MiB / 0 MiB / 2 MiB ( 0 MiB / 96 MiB / 8 MiB / 0 MiB / 0 MiB )
13 min passed...2464 MiB / 5024 MiB / 2437 MiB / 0 MiB / 2 MiB ( 0 MiB / 416 MiB / 10 MiB / 0 MiB / 0 MiB )
14 min passed...2464 MiB / 5376 MiB / 2446 MiB / 0 MiB / 3 MiB ( 0 MiB / 352 MiB / 9 MiB / 0 MiB / 1 MiB )
15 min passed...2464 MiB / 5472 MiB / 2456 MiB / 1056 MiB / 1027 MiB ( 0 MiB / 96 MiB / 10 MiB / 1056 MiB / 1024 MiB )
16 min passed...2464 MiB / 5472 MiB / 2465 MiB / 1056 MiB / 1027 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
17 min passed...2496 MiB / 5536 MiB / 2475 MiB / 1056 MiB / 1027 MiB ( 32 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...2496 MiB / 5920 MiB / 2484 MiB / 1056 MiB / 1027 MiB ( 0 MiB / 384 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...2528 MiB / 6528 MiB / 2495 MiB / 1056 MiB / 1027 MiB ( 32 MiB / 608 MiB / 11 MiB / 0 MiB / 0 MiB )
20 min passed...2528 MiB / 6720 MiB / 2504 MiB / 1056 MiB / 1027 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...2528 MiB / 6944 MiB / 2516 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 224 MiB / 12 MiB / 0 MiB / 1 MiB )
22 min passed...2560 MiB / 7104 MiB / 2526 MiB / 1056 MiB / 1028 MiB ( 32 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
23 min passed...2592 MiB / 7328 MiB / 2536 MiB / 1056 MiB / 1028 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...2624 MiB / 7616 MiB / 2547 MiB / 1056 MiB / 1028 MiB ( 32 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
25 min passed...2656 MiB / 8032 MiB / 2556 MiB / 1056 MiB / 1028 MiB ( 32 MiB / 416 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...2656 MiB / 8032 MiB / 2567 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
27 min passed...2656 MiB / 8320 MiB / 2577 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...2656 MiB / 8448 MiB / 2586 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
29 min passed...2656 MiB / 8512 MiB / 2596 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 1 MiB )
30 min passed...2656 MiB / 9056 MiB / 2610 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 544 MiB / 14 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 6: async 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3017: Mon Aug 16 19:12:19 2021
  write: IOPS=158, BW=636MiB/s (667MB/s)(10.0GiB/16104msec); 0 zone resets
    slat (usec): min=93, max=18887, avg=225.40, stdev=501.84
    clat (msec): min=16, max=3684, avg=400.21, stdev=770.33
     lat (msec): min=16, max=3684, avg=400.43, stdev=770.30
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   68], 10.00th=[   75], 20.00th=[   77],
     | 30.00th=[   79], 40.00th=[   83], 50.00th=[   85], 60.00th=[   97],
     | 70.00th=[  120], 80.00th=[  186], 90.00th=[ 2198], 95.00th=[ 2366],
     | 99.00th=[ 3004], 99.50th=[ 3641], 99.90th=[ 3641], 99.95th=[ 3675],
     | 99.99th=[ 3675]
   bw (  MiB/s): min=   86, max= 3048, per=100.00%, avg=1294.04, stdev=903.13, samples=15
   iops        : min=   21, max=  762, avg=323.40, stdev=225.88, samples=15
  lat (msec)   : 20=0.16%, 50=2.66%, 100=58.91%, 250=23.09%, 750=1.80%
  lat (msec)   : 1000=0.90%
  cpu          : usr=17.44%, sys=3.95%, ctx=1729, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=636MiB/s (667MB/s), 636MiB/s-636MiB/s (667MB/s-667MB/s), io=10.0GiB (10.7GB), run=16104-16104msec

Disk stats (read/write):
  sda: ios=0/10213, merge=0/96, ticks=0/2977530, in_queue=583836, util=34.87%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 10241 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1698MiB/s][r=424 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3244: Mon Aug 16 19:27:26 2021
  read: IOPS=395, BW=1581MiB/s (1658MB/s)(10.0GiB/6477msec)
    slat (usec): min=80, max=5819, avg=188.71, stdev=235.51
    clat (msec): min=12, max=251, avg=160.88, stdev=63.71
     lat (msec): min=12, max=251, avg=161.07, stdev=63.66
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   47], 10.00th=[   56], 20.00th=[   75],
     | 30.00th=[  148], 40.00th=[  176], 50.00th=[  188], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  213], 90.00th=[  220], 95.00th=[  226],
     | 99.00th=[  234], 99.50th=[  236], 99.90th=[  245], 99.95th=[  245],
     | 99.99th=[  251]
   bw (  MiB/s): min= 1208, max= 2408, per=98.88%, avg=1563.33, stdev=344.03, samples=12
   iops        : min=  302, max=  602, avg=390.83, stdev=86.01, samples=12
  lat (msec)   : 20=0.23%, 50=6.95%, 100=19.49%, 250=73.28%, 500=0.04%
  cpu          : usr=0.76%, sys=7.40%, ctx=2036, majf=0, minf=649
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1581MiB/s (1658MB/s), 1581MiB/s-1581MiB/s (1658MB/s-1658MB/s), io=10.0GiB (10.7GB), run=6477-6477msec

Disk stats (read/write):
  sda: ios=10105/0, merge=0/0, ticks=1177995/0, in_queue=1193680, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10242 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1824 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...23168 MiB / 24736 MiB / 23145 MiB / 224 MiB / 258 MiB ( 23168 MiB / 24704 MiB / 23145 MiB / 224 MiB / 258 MiB )
2 min passed...23200 MiB / 24960 MiB / 23156 MiB / 224 MiB / 258 MiB ( 32 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
3 min passed...23200 MiB / 25376 MiB / 23168 MiB / 224 MiB / 258 MiB ( 0 MiB / 416 MiB / 12 MiB / 0 MiB / 0 MiB )
4 min passed...23200 MiB / 25856 MiB / 23177 MiB / 224 MiB / 259 MiB ( 0 MiB / 480 MiB / 9 MiB / 0 MiB / 1 MiB )
5 min passed...23200 MiB / 25888 MiB / 23189 MiB / 224 MiB / 259 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
6 min passed...23200 MiB / 25984 MiB / 23199 MiB / 224 MiB / 259 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
7 min passed...23200 MiB / 26144 MiB / 23208 MiB / 224 MiB / 259 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
8 min passed...23200 MiB / 26272 MiB / 23217 MiB / 224 MiB / 259 MiB ( 0 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
9 min passed...23200 MiB / 26656 MiB / 23224 MiB / 224 MiB / 259 MiB ( 0 MiB / 384 MiB / 7 MiB / 0 MiB / 0 MiB )
10 min passed...23200 MiB / 26912 MiB / 23225 MiB / 224 MiB / 259 MiB ( 0 MiB / 256 MiB / 1 MiB / 0 MiB / 0 MiB )
11 min passed...23200 MiB / 26912 MiB / 23225 MiB / 224 MiB / 259 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...23200 MiB / 27168 MiB / 23225 MiB / 224 MiB / 259 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...23200 MiB / 27168 MiB / 23225 MiB / 224 MiB / 259 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...23232 MiB / 27200 MiB / 23232 MiB / 224 MiB / 259 MiB ( 32 MiB / 32 MiB / 7 MiB / 0 MiB / 0 MiB )
15 min passed...23232 MiB / 27392 MiB / 23233 MiB / 224 MiB / 259 MiB ( 0 MiB / 192 MiB / 1 MiB / 0 MiB / 0 MiB )
16 min passed...23232 MiB / 27392 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 0 MiB / 0 MiB / 9984 MiB / 9985 MiB )
17 min passed...23232 MiB / 27680 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...23232 MiB / 27936 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...23232 MiB / 27968 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...23232 MiB / 28096 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...23232 MiB / 28096 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...23232 MiB / 28224 MiB / 23233 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...23232 MiB / 28672 MiB / 23235 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 448 MiB / 2 MiB / 0 MiB / 0 MiB )
24 min passed...23264 MiB / 28800 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 32 MiB / 128 MiB / 4 MiB / 0 MiB / 0 MiB )
25 min passed...23264 MiB / 28800 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...23264 MiB / 28928 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...23264 MiB / 28992 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...23264 MiB / 29280 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...23264 MiB / 29504 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...23264 MiB / 29600 MiB / 23239 MiB / 10208 MiB / 10244 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=3476: Mon Aug 16 19:46:25 2021
  write: IOPS=30.2k, BW=472MiB/s (495MB/s)(1024MiB/2170msec); 0 zone resets
    slat (usec): min=7, max=3228, avg=20.43, stdev=21.19
    clat (usec): min=49, max=24203, avg=2087.14, stdev=674.42
     lat (usec): min=61, max=24230, avg=2108.92, stdev=677.96
    clat percentiles (usec):
     |  1.00th=[ 1582],  5.00th=[ 1729], 10.00th=[ 1778], 20.00th=[ 1811],
     | 30.00th=[ 1844], 40.00th=[ 1893], 50.00th=[ 1926], 60.00th=[ 1958],
     | 70.00th=[ 2024], 80.00th=[ 2147], 90.00th=[ 2606], 95.00th=[ 3032],
     | 99.00th=[ 4080], 99.50th=[ 6587], 99.90th=[10683], 99.95th=[11338],
     | 99.99th=[12125]
   bw (  KiB/s): min=372256, max=537312, per=100.00%, avg=486544.00, stdev=77185.60, samples=4
   iops        : min=23266, max=33582, avg=30409.00, stdev=4824.10, samples=4
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=66.18%, 4=32.74%, 10=0.89%, 20=0.13%, 50=0.01%
  cpu          : usr=18.07%, sys=58.00%, ctx=67540, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=472MiB/s (495MB/s), 472MiB/s-472MiB/s (495MB/s-495MB/s), io=1024MiB (1074MB), run=2170-2170msec

Disk stats (read/write):
  sda: ios=0/64637, merge=0/0, ticks=0/6335, in_queue=7000, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=3688: Mon Aug 16 20:01:27 2021
  read: IOPS=65.8k, BW=1028MiB/s (1078MB/s)(1024MiB/996msec)
    slat (usec): min=4, max=103, avg= 8.32, stdev= 4.22
    clat (usec): min=251, max=3602, avg=957.43, stdev=93.51
     lat (usec): min=256, max=3608, avg=966.94, stdev=94.33
    clat percentiles (usec):
     |  1.00th=[  832],  5.00th=[  865], 10.00th=[  881], 20.00th=[  906],
     | 30.00th=[  922], 40.00th=[  938], 50.00th=[  947], 60.00th=[  963],
     | 70.00th=[  979], 80.00th=[  996], 90.00th=[ 1029], 95.00th=[ 1057],
     | 99.00th=[ 1156], 99.50th=[ 1663], 99.90th=[ 1926], 99.95th=[ 2114],
     | 99.99th=[ 2835]
   bw (  MiB/s): min= 1027, max= 1027, per=99.89%, avg=1027.03, stdev= 0.00, samples=1
   iops        : min=65730, max=65730, avg=65730.00, stdev= 0.00, samples=1
  lat (usec)   : 500=0.03%, 750=0.03%, 1000=81.16%
  lat (msec)   : 2=18.71%, 4=0.08%
  cpu          : usr=39.30%, sys=60.60%, ctx=9, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1028MiB/s (1078MB/s), 1028MiB/s-1028MiB/s (1078MB/s-1078MB/s), io=1024MiB (1074MB), run=996-996msec

Disk stats (read/write):
  sda: ios=59068/0, merge=0/0, ticks=15641/0, in_queue=16112, util=99.90%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2048 MiB / 2496 MiB / 2060 MiB / 0 MiB / 0 MiB ( 2048 MiB / 2496 MiB / 2060 MiB / 0 MiB / 0 MiB )
2 min passed...2048 MiB / 2496 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2048 MiB / 2624 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2048 MiB / 3040 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2048 MiB / 3104 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2048 MiB / 3328 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2048 MiB / 3456 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2048 MiB / 3648 MiB / 2060 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2048 MiB / 4064 MiB / 2064 MiB / 0 MiB / 0 MiB ( 0 MiB / 416 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...2048 MiB / 4256 MiB / 2064 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2048 MiB / 4448 MiB / 2064 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2048 MiB / 4672 MiB / 2064 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2048 MiB / 4896 MiB / 2064 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2048 MiB / 5024 MiB / 2067 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 3 MiB / 0 MiB / 0 MiB )
15 min passed...2048 MiB / 5600 MiB / 2067 MiB / 768 MiB / 768 MiB ( 0 MiB / 576 MiB / 0 MiB / 768 MiB / 768 MiB )
16 min passed...2048 MiB / 5760 MiB / 2067 MiB / 768 MiB / 768 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2048 MiB / 5984 MiB / 2067 MiB / 768 MiB / 768 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2048 MiB / 6048 MiB / 2067 MiB / 768 MiB / 768 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2048 MiB / 6048 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
20 min passed...2048 MiB / 6528 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2048 MiB / 7008 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2048 MiB / 7296 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2048 MiB / 7424 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2048 MiB / 7456 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2048 MiB / 7584 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2048 MiB / 8192 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2048 MiB / 8576 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2048 MiB / 8736 MiB / 2070 MiB / 768 MiB / 768 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2048 MiB / 9024 MiB / 2074 MiB / 768 MiB / 768 MiB ( 0 MiB / 288 MiB / 4 MiB / 0 MiB / 0 MiB )
30 min passed...2048 MiB / 9376 MiB / 2084 MiB / 768 MiB / 768 MiB ( 0 MiB / 352 MiB / 10 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=9.98MiB/s][w=319 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=3973: Mon Aug 16 20:25:21 2021
  write: IOPS=312, BW=9.78MiB/s (10.3MB/s)(1024MiB/104731msec); 0 zone resets
    clat (usec): min=1911, max=36744, avg=3168.77, stdev=922.16
     lat (usec): min=1913, max=36747, avg=3171.16, stdev=922.21
    clat percentiles (usec):
     |  1.00th=[ 2311],  5.00th=[ 2540], 10.00th=[ 2671], 20.00th=[ 2802],
     | 30.00th=[ 2900], 40.00th=[ 2999], 50.00th=[ 3064], 60.00th=[ 3130],
     | 70.00th=[ 3228], 80.00th=[ 3326], 90.00th=[ 3490], 95.00th=[ 3982],
     | 99.00th=[ 5473], 99.50th=[ 8455], 99.90th=[13829], 99.95th=[18744],
     | 99.99th=[30540]
   bw (  KiB/s): min= 7744, max=12416, per=100.00%, avg=10013.29, stdev=745.88, samples=209
   iops        : min=  242, max=  388, avg=312.91, stdev=23.31, samples=209
  lat (msec)   : 2=0.03%, 4=95.06%, 10=4.56%, 20=0.31%, 50=0.04%
  cpu          : usr=1.08%, sys=4.41%, ctx=72803, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9.78MiB/s (10.3MB/s), 9.78MiB/s-9.78MiB/s (10.3MB/s-10.3MB/s), io=1024MiB (1074MB), run=104731-104731msec

Disk stats (read/write):
  sda: ios=0/98342, merge=0/66634, ticks=0/95523, in_queue=104664, util=99.94%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 1541 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=46.5MiB/s][r=1489 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4176: Mon Aug 16 20:40:44 2021
  read: IOPS=1495, BW=46.7MiB/s (48.0MB/s)(1024MiB/21917msec)
    clat (usec): min=249, max=6020, avg=657.25, stdev=244.96
     lat (usec): min=251, max=6022, avg=659.29, stdev=244.98
    clat percentiles (usec):
     |  1.00th=[  445],  5.00th=[  494], 10.00th=[  529], 20.00th=[  562],
     | 30.00th=[  594], 40.00th=[  611], 50.00th=[  635], 60.00th=[  660],
     | 70.00th=[  685], 80.00th=[  701], 90.00th=[  734], 95.00th=[  758],
     | 99.00th=[ 1811], 99.50th=[ 2835], 99.90th=[ 3326], 99.95th=[ 3425],
     | 99.99th=[ 5473]
   bw (  KiB/s): min=43712, max=59584, per=99.97%, avg=47828.47, stdev=3058.95, samples=43
   iops        : min= 1366, max= 1862, avg=1494.67, stdev=95.60, samples=43
  lat (usec)   : 250=0.01%, 500=5.79%, 750=88.22%, 1000=4.13%
  lat (msec)   : 2=0.96%, 4=0.87%, 10=0.03%
  cpu          : usr=2.40%, sys=9.64%, ctx=32770, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=46.7MiB/s (48.0MB/s), 46.7MiB/s-46.7MiB/s (48.0MB/s-48.0MB/s), io=1024MiB (1074MB), run=21917-21917msec

Disk stats (read/write):
  sda: ios=32739/0, merge=0/0, ticks=20066/0, in_queue=21956, util=99.89%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1928 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5952 MiB / 7104 MiB / 5962 MiB / 192 MiB / 202 MiB ( 5952 MiB / 7104 MiB / 5962 MiB / 192 MiB / 202 MiB )
2 min passed...11456 MiB / 12864 MiB / 11393 MiB / 384 MiB / 353 MiB ( 5504 MiB / 5760 MiB / 5431 MiB / 192 MiB / 151 MiB )
3 min passed...11456 MiB / 13152 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...11456 MiB / 13440 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...11456 MiB / 13696 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...11456 MiB / 14304 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11456 MiB / 14368 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...11456 MiB / 14528 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...11456 MiB / 14880 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...11456 MiB / 15008 MiB / 11393 MiB / 384 MiB / 353 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11456 MiB / 15456 MiB / 11396 MiB / 384 MiB / 353 MiB ( 0 MiB / 448 MiB / 3 MiB / 0 MiB / 0 MiB )
12 min passed...11456 MiB / 15808 MiB / 11399 MiB / 384 MiB / 353 MiB ( 0 MiB / 352 MiB / 3 MiB / 0 MiB / 0 MiB )
13 min passed...11456 MiB / 15808 MiB / 11399 MiB / 384 MiB / 353 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...11456 MiB / 16128 MiB / 11399 MiB / 384 MiB / 353 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...11456 MiB / 16544 MiB / 11399 MiB / 384 MiB / 353 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...11456 MiB / 16896 MiB / 11399 MiB / 384 MiB / 353 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...11456 MiB / 17344 MiB / 11399 MiB / 1248 MiB / 1239 MiB ( 0 MiB / 448 MiB / 0 MiB / 864 MiB / 886 MiB )
18 min passed...11456 MiB / 17440 MiB / 11399 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 96 MiB / 0 MiB / 128 MiB / 138 MiB )
19 min passed...11456 MiB / 17472 MiB / 11399 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11456 MiB / 18016 MiB / 11399 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...11456 MiB / 18368 MiB / 11399 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...11456 MiB / 18752 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 384 MiB / 4 MiB / 0 MiB / 0 MiB )
23 min passed...11456 MiB / 19040 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11456 MiB / 19072 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...11456 MiB / 19392 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...11456 MiB / 19776 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...11456 MiB / 20128 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11456 MiB / 20384 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...11456 MiB / 20576 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...11456 MiB / 20704 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...11456 MiB / 21152 MiB / 11403 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...11456 MiB / 21568 MiB / 11408 MiB / 1376 MiB / 1377 MiB ( 0 MiB / 416 MiB / 5 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 16, 2021

Round 2 - Run 6 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=4472: Mon Aug 16 22:55:35 2021
  write: IOPS=24.3k, BW=760MiB/s (797MB/s)(1024MiB/1347msec); 0 zone resets
    slat (usec): min=8, max=3313, avg=21.17, stdev=33.51
    clat (usec): min=82, max=13970, avg=2586.38, stdev=966.01
     lat (usec): min=105, max=13989, avg=2608.87, stdev=971.92
    clat percentiles (usec):
     |  1.00th=[ 2040],  5.00th=[ 2114], 10.00th=[ 2147], 20.00th=[ 2212],
     | 30.00th=[ 2278], 40.00th=[ 2311], 50.00th=[ 2376], 60.00th=[ 2409],
     | 70.00th=[ 2474], 80.00th=[ 2606], 90.00th=[ 3032], 95.00th=[ 3687],
     | 99.00th=[ 7701], 99.50th=[ 8848], 99.90th=[13042], 99.95th=[13566],
     | 99.99th=[13829]
   bw (  KiB/s): min=832792, max=850752, per=100.00%, avg=841772.00, stdev=12699.64, samples=2
   iops        : min=26024, max=26586, avg=26305.00, stdev=397.39, samples=2
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.38%, 4=95.80%, 10=3.39%, 20=0.36%
  cpu          : usr=32.17%, sys=47.47%, ctx=33192, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=760MiB/s (797MB/s), 760MiB/s-760MiB/s (797MB/s-797MB/s), io=1024MiB (1074MB), run=1347-1347msec

Disk stats (read/write):
  sda: ios=0/28546, merge=0/0, ticks=0/3802, in_queue=4256, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=4689: Mon Aug 16 23:10:37 2021
  read: IOPS=29.7k, BW=928MiB/s (973MB/s)(1024MiB/1103msec)
    slat (usec): min=5, max=477, avg=19.98, stdev=22.52
    clat (usec): min=685, max=7956, avg=2121.06, stdev=489.14
     lat (usec): min=696, max=7974, avg=2143.15, stdev=493.04
    clat percentiles (usec):
     |  1.00th=[  922],  5.00th=[ 1139], 10.00th=[ 1303], 20.00th=[ 1975],
     | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2212], 60.00th=[ 2245],
     | 70.00th=[ 2311], 80.00th=[ 2343], 90.00th=[ 2442], 95.00th=[ 2933],
     | 99.00th=[ 3195], 99.50th=[ 3687], 99.90th=[ 4883], 99.95th=[ 5211],
     | 99.99th=[ 6783]
   bw (  KiB/s): min=886720, max=909696, per=94.48%, avg=898208.00, stdev=16246.49, samples=2
   iops        : min=27710, max=28428, avg=28069.00, stdev=507.70, samples=2
  lat (usec)   : 750=0.02%, 1000=1.91%
  lat (msec)   : 2=18.89%, 4=78.78%, 10=0.40%
  cpu          : usr=38.02%, sys=60.53%, ctx=262, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=928MiB/s (973MB/s), 928MiB/s-928MiB/s (973MB/s-973MB/s), io=1024MiB (1074MB), run=1103-1103msec

Disk stats (read/write):
  sda: ios=24870/0, merge=0/0, ticks=10309/0, in_queue=10404, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1804 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2528 MiB / 3072 MiB / 2524 MiB / 0 MiB / 0 MiB ( 2528 MiB / 3072 MiB / 2524 MiB / 0 MiB / 0 MiB )
2 min passed...2528 MiB / 3360 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2528 MiB / 3488 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2528 MiB / 3680 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2528 MiB / 3872 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2528 MiB / 4224 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2528 MiB / 4736 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2528 MiB / 5120 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2528 MiB / 5504 MiB / 2524 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2528 MiB / 5728 MiB / 2528 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 4 MiB / 0 MiB / 0 MiB )
11 min passed...2528 MiB / 5984 MiB / 2528 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2528 MiB / 6272 MiB / 2528 MiB / 0 MiB / 0 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2528 MiB / 6720 MiB / 2528 MiB / 0 MiB / 0 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2528 MiB / 7040 MiB / 2528 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2528 MiB / 7264 MiB / 2528 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 224 MiB / 0 MiB / 1056 MiB / 1025 MiB )
16 min passed...2528 MiB / 7552 MiB / 2528 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2528 MiB / 7968 MiB / 2528 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2528 MiB / 8320 MiB / 2528 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2528 MiB / 8704 MiB / 2528 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2528 MiB / 8896 MiB / 2532 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 192 MiB / 4 MiB / 0 MiB / 0 MiB )
21 min passed...2528 MiB / 9152 MiB / 2532 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2528 MiB / 9568 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 416 MiB / 4 MiB / 0 MiB / 0 MiB )
23 min passed...2528 MiB / 9856 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2528 MiB / 9984 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2528 MiB / 10144 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2528 MiB / 10304 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2528 MiB / 10656 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2528 MiB / 11104 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2528 MiB / 11328 MiB / 2536 MiB / 1056 MiB / 1025 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2560 MiB / 11456 MiB / 2547 MiB / 1056 MiB / 1025 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 17, 2021

Round 2 - Run 6 - Test X6: 4x async sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=508MiB/s][w=16.3k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=4924: Mon Aug 16 23:30:23 2021
  write: IOPS=4367, BW=136MiB/s (143MB/s)(3072MiB/22506msec); 0 zone resets
    slat (usec): min=10, max=12855, avg=57.13, stdev=90.21
    clat (usec): min=2, max=55151, avg=140.54, stdev=525.97
     lat (usec): min=58, max=55171, avg=200.88, stdev=532.14
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   35], 20.00th=[   76],
     | 30.00th=[   90], 40.00th=[  103], 50.00th=[  117], 60.00th=[  133],
     | 70.00th=[  153], 80.00th=[  182], 90.00th=[  227], 95.00th=[  277],
     | 99.00th=[  449], 99.50th=[  578], 99.90th=[ 1647], 99.95th=[ 4146],
     | 99.99th=[32900]
   bw (  KiB/s): min=78222, max=223936, per=24.93%, avg=139361.59, stdev=25950.07, samples=44
   iops        : min= 2444, max= 6998, avg=4355.02, stdev=810.99, samples=44
  lat (usec)   : 4=7.60%, 10=1.48%, 20=0.25%, 50=1.52%, 100=27.38%
  lat (usec)   : 250=54.67%, 500=6.36%, 750=0.48%, 1000=0.09%
  lat (msec)   : 2=0.08%, 4=0.03%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=7.94%, sys=15.76%, ctx=144777, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=4925: Mon Aug 16 23:30:23 2021
  write: IOPS=4376, BW=137MiB/s (143MB/s)(3072MiB/22461msec); 0 zone resets
    slat (usec): min=10, max=4484, avg=56.39, stdev=72.56
    clat (usec): min=2, max=55181, avg=140.41, stdev=495.07
     lat (usec): min=58, max=55208, avg=199.91, stdev=497.97
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   42], 20.00th=[   77],
     | 30.00th=[   89], 40.00th=[  103], 50.00th=[  118], 60.00th=[  133],
     | 70.00th=[  153], 80.00th=[  180], 90.00th=[  227], 95.00th=[  277],
     | 99.00th=[  457], 99.50th=[  578], 99.90th=[ 1483], 99.95th=[ 4228],
     | 99.99th=[31589]
   bw (  KiB/s): min=88825, max=226176, per=24.98%, avg=139661.36, stdev=26862.91, samples=44
   iops        : min= 2775, max= 7068, avg=4364.39, stdev=839.52, samples=44
  lat (usec)   : 4=7.29%, 10=1.44%, 20=0.23%, 50=1.51%, 100=27.68%
  lat (usec)   : 250=54.91%, 500=6.18%, 750=0.48%, 1000=0.11%
  lat (msec)   : 2=0.08%, 4=0.03%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=8.50%, sys=15.62%, ctx=142901, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=4926: Mon Aug 16 23:30:23 2021
  write: IOPS=4376, BW=137MiB/s (143MB/s)(3072MiB/22463msec); 0 zone resets
    slat (usec): min=10, max=7679, avg=56.48, stdev=81.63
    clat (usec): min=2, max=55080, avg=140.89, stdev=501.12
     lat (usec): min=53, max=55101, avg=200.56, stdev=504.82
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   42], 20.00th=[   77],
     | 30.00th=[   90], 40.00th=[  102], 50.00th=[  117], 60.00th=[  133],
     | 70.00th=[  153], 80.00th=[  182], 90.00th=[  227], 95.00th=[  277],
     | 99.00th=[  457], 99.50th=[  594], 99.90th=[ 1647], 99.95th=[ 4424],
     | 99.99th=[27919]
   bw (  KiB/s): min=86095, max=228480, per=24.99%, avg=139710.70, stdev=26532.98, samples=44
   iops        : min= 2690, max= 7140, avg=4365.91, stdev=829.19, samples=44
  lat (usec)   : 4=7.04%, 10=1.64%, 20=0.28%, 50=1.53%, 100=27.75%
  lat (usec)   : 250=54.67%, 500=6.31%, 750=0.48%, 1000=0.12%
  lat (msec)   : 2=0.08%, 4=0.03%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=8.13%, sys=15.78%, ctx=143344, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=4927: Mon Aug 16 23:30:23 2021
  write: IOPS=4406, BW=138MiB/s (144MB/s)(3072MiB/22308msec); 0 zone resets
    slat (usec): min=10, max=7567, avg=55.80, stdev=78.04
    clat (usec): min=2, max=55008, avg=140.29, stdev=535.61
     lat (usec): min=57, max=55140, avg=199.18, stdev=539.71
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   46], 20.00th=[   77],
     | 30.00th=[   89], 40.00th=[  102], 50.00th=[  116], 60.00th=[  133],
     | 70.00th=[  151], 80.00th=[  178], 90.00th=[  225], 95.00th=[  273],
     | 99.00th=[  445], 99.50th=[  578], 99.90th=[ 1762], 99.95th=[ 4817],
     | 99.99th=[36963]
   bw (  KiB/s): min=84571, max=236096, per=25.17%, avg=140725.82, stdev=27703.90, samples=44
   iops        : min= 2642, max= 7378, avg=4397.61, stdev=865.77, samples=44
  lat (usec)   : 4=6.95%, 10=1.59%, 20=0.21%, 50=1.51%, 100=28.53%
  lat (usec)   : 250=54.25%, 500=6.23%, 750=0.43%, 1000=0.11%
  lat (msec)   : 2=0.09%, 4=0.03%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=8.33%, sys=15.58%, ctx=142734, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=546MiB/s (573MB/s), 136MiB/s-138MiB/s (143MB/s-144MB/s), io=12.0GiB (12.9GB), run=22308-22506msec

Disk stats (read/write):
  sda: ios=0/392061, merge=0/225, ticks=0/54635, in_queue=62964, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 12289 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member