how to best benchmark SSDs?

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=9.98MiB/s][w=319 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4785: Thu Aug 19 16:16:14 2021
  write: IOPS=317, BW=9.91MiB/s (10.4MB/s)(1024MiB/103314msec); 0 zone resets
    clat (usec): min=1534, max=43420, avg=3125.73, stdev=1030.08
     lat (usec): min=1535, max=43423, avg=3127.98, stdev=1030.08
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2474], 10.00th=[ 2606], 20.00th=[ 2769],
     | 30.00th=[ 2868], 40.00th=[ 2966], 50.00th=[ 3032], 60.00th=[ 3130],
     | 70.00th=[ 3195], 80.00th=[ 3294], 90.00th=[ 3458], 95.00th=[ 3687],
     | 99.00th=[ 5604], 99.50th=[ 8356], 99.90th=[14615], 99.95th=[23987],
     | 99.99th=[36439]
   bw (  KiB/s): min= 7536, max=13504, per=99.97%, avg=10145.92, stdev=774.83, samples=206
   iops        : min=  235, max=  422, avg=317.05, stdev=24.22, samples=206
  lat (msec)   : 2=0.29%, 4=96.18%, 10=3.18%, 20=0.27%, 50=0.07%
  cpu          : usr=1.14%, sys=4.63%, ctx=69863, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9.91MiB/s (10.4MB/s), 9.91MiB/s-9.91MiB/s (10.4MB/s-10.4MB/s), io=1024MiB (1074MB), run=103314-103314msec

Disk stats (read/write):
  sda: ios=0/98316, merge=0/65943, ticks=0/93656, in_queue=102860, util=99.67%
0 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 1539 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=60.3MiB/s][r=1930 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4991: Thu Aug 19 16:31:32 2021
  read: IOPS=1863, BW=58.2MiB/s (61.1MB/s)(1024MiB/17587msec)
    clat (usec): min=363, max=3061, avg=526.38, stdev=65.82
     lat (usec): min=364, max=3062, avg=528.28, stdev=65.92
    clat percentiles (usec):
     |  1.00th=[  420],  5.00th=[  441], 10.00th=[  453], 20.00th=[  469],
     | 30.00th=[  486], 40.00th=[  502], 50.00th=[  519], 60.00th=[  529],
     | 70.00th=[  553], 80.00th=[  578], 90.00th=[  619], 95.00th=[  652],
     | 99.00th=[  709], 99.50th=[  725], 99.90th=[  758], 99.95th=[  775],
     | 99.99th=[ 1020]
   bw (  KiB/s): min=54208, max=64128, per=99.94%, avg=59584.26, stdev=2637.60, samples=35
   iops        : min= 1694, max= 2004, avg=1862.00, stdev=82.42, samples=35
  lat (usec)   : 500=39.29%, 750=60.52%, 1000=0.18%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.65%, sys=10.03%, ctx=32770, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=58.2MiB/s (61.1MB/s), 58.2MiB/s-58.2MiB/s (61.1MB/s-61.1MB/s), io=1024MiB (1074MB), run=17587-17587msec

Disk stats (read/write):
  sda: ios=32390/0, merge=0/0, ticks=15867/0, in_queue=17496, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1539 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1922 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...6528 MiB / 6944 MiB / 6565 MiB / 192 MiB / 158 MiB ( 6528 MiB / 6944 MiB / 6565 MiB / 192 MiB / 158 MiB )
2 min passed...11744 MiB / 12768 MiB / 11774 MiB / 288 MiB / 269 MiB ( 5216 MiB / 5824 MiB / 5209 MiB / 96 MiB / 111 MiB )
3 min passed...11744 MiB / 12800 MiB / 11781 MiB / 288 MiB / 269 MiB ( 0 MiB / 32 MiB / 7 MiB / 0 MiB / 0 MiB )
4 min passed...11744 MiB / 13184 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 384 MiB / 5 MiB / 0 MiB / 0 MiB )
5 min passed...11744 MiB / 13760 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...11744 MiB / 13760 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11744 MiB / 13920 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...11744 MiB / 14112 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...11744 MiB / 14112 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...11744 MiB / 14560 MiB / 11786 MiB / 288 MiB / 269 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11744 MiB / 14944 MiB / 11788 MiB / 288 MiB / 269 MiB ( 0 MiB / 384 MiB / 2 MiB / 0 MiB / 0 MiB )
12 min passed...11776 MiB / 14944 MiB / 11793 MiB / 288 MiB / 269 MiB ( 32 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
13 min passed...11776 MiB / 14944 MiB / 11793 MiB / 288 MiB / 269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...11776 MiB / 14944 MiB / 11793 MiB / 288 MiB / 269 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...11776 MiB / 15072 MiB / 11793 MiB / 288 MiB / 269 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...11776 MiB / 15392 MiB / 11793 MiB / 288 MiB / 269 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...11776 MiB / 15392 MiB / 11793 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1024 MiB )
18 min passed...11776 MiB / 15456 MiB / 11793 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...11776 MiB / 15712 MiB / 11793 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11776 MiB / 15712 MiB / 11793 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...11776 MiB / 16000 MiB / 11797 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 288 MiB / 4 MiB / 0 MiB / 0 MiB )
22 min passed...11776 MiB / 16160 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 160 MiB / 3 MiB / 0 MiB / 0 MiB )
23 min passed...11776 MiB / 16160 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11776 MiB / 16160 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...11776 MiB / 16160 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...11776 MiB / 16160 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...11776 MiB / 16288 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11776 MiB / 16288 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...11776 MiB / 16320 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...11776 MiB / 16480 MiB / 11800 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...11808 MiB / 16480 MiB / 11806 MiB / 1312 MiB / 1293 MiB ( 32 MiB / 0 MiB / 6 MiB / 0 MiB / 0 MiB )
32 min passed...11808 MiB / 16512 MiB / 11812 MiB / 1312 MiB / 1293 MiB ( 0 MiB / 32 MiB / 6 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=5227: Thu Aug 19 16:56:19 2021
  write: IOPS=22.1k, BW=691MiB/s (725MB/s)(1024MiB/1482msec); 0 zone resets
    slat (usec): min=8, max=2148, avg=22.50, stdev=24.28
    clat (usec): min=114, max=36196, avg=2848.43, stdev=1893.13
     lat (usec): min=130, max=36215, avg=2872.24, stdev=1894.70
    clat percentiles (usec):
     |  1.00th=[ 1926],  5.00th=[ 2180], 10.00th=[ 2245], 20.00th=[ 2311],
     | 30.00th=[ 2343], 40.00th=[ 2376], 50.00th=[ 2409], 60.00th=[ 2474],
     | 70.00th=[ 2573], 80.00th=[ 3032], 90.00th=[ 3818], 95.00th=[ 4424],
     | 99.00th=[ 7308], 99.50th=[13960], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min=805376, max=833984, per=100.00%, avg=819680.00, stdev=20228.91, samples=2
   iops        : min=25168, max=26062, avg=25615.00, stdev=632.15, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=1.93%, 4=90.67%, 10=6.63%, 20=0.52%, 50=0.20%
  cpu          : usr=28.97%, sys=46.93%, ctx=31152, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=691MiB/s (725MB/s), 691MiB/s-691MiB/s (725MB/s-725MB/s), io=1024MiB (1074MB), run=1482-1482msec

Disk stats (read/write):
  sda: ios=0/31198, merge=0/0, ticks=0/12328, in_queue=13084, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process

async_rand_32K: (groupid=0, jobs=1): err= 0: pid=5443: Thu Aug 19 17:11:20 2021
  read: IOPS=40.2k, BW=1255MiB/s (1316MB/s)(1024MiB/816msec)
    slat (usec): min=4, max=657, avg=12.47, stdev=15.97
    clat (usec): min=549, max=10803, avg=1570.75, stdev=395.35
     lat (usec): min=555, max=10812, avg=1584.48, stdev=395.41
    clat percentiles (usec):
     |  1.00th=[  930],  5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1270],
     | 30.00th=[ 1352], 40.00th=[ 1434], 50.00th=[ 1516], 60.00th=[ 1598],
     | 70.00th=[ 1713], 80.00th=[ 1844], 90.00th=[ 2040], 95.00th=[ 2245],
     | 99.00th=[ 2802], 99.50th=[ 3097], 99.90th=[ 3982], 99.95th=[ 4490],
     | 99.99th=[ 7242]
   bw (  MiB/s): min= 1246, max= 1246, per=99.31%, avg=1246.19, stdev= 0.00, samples=1
   iops        : min=39878, max=39878, avg=39878.00, stdev= 0.00, samples=1
  lat (usec)   : 750=0.05%, 1000=2.49%
  lat (msec)   : 2=85.78%, 4=11.58%, 10=0.09%, 20=0.01%
  cpu          : usr=28.96%, sys=57.18%, ctx=1458, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1255MiB/s (1316MB/s), 1255MiB/s-1255MiB/s (1316MB/s-1316MB/s), io=1024MiB (1074MB), run=816-816msec

Disk stats (read/write):
  sda: ios=25923/0, merge=0/0, ticks=35733/0, in_queue=36144, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1025 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1804 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2560 MiB / 2784 MiB / 2561 MiB / 0 MiB / 0 MiB ( 2560 MiB / 2784 MiB / 2561 MiB / 0 MiB / 0 MiB )
2 min passed...2560 MiB / 3232 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2560 MiB / 3328 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2560 MiB / 3328 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2560 MiB / 3392 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2560 MiB / 3392 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2560 MiB / 3552 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2560 MiB / 3904 MiB / 2561 MiB / 0 MiB / 0 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2560 MiB / 3904 MiB / 2564 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
10 min passed...2560 MiB / 3904 MiB / 2569 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
11 min passed...2560 MiB / 3904 MiB / 2569 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2560 MiB / 3904 MiB / 2569 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2560 MiB / 4128 MiB / 2569 MiB / 0 MiB / 0 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2560 MiB / 4288 MiB / 2577 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 8 MiB / 0 MiB / 0 MiB )
15 min passed...2560 MiB / 4288 MiB / 2577 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 1056 MiB / 1060 MiB )
16 min passed...2560 MiB / 4288 MiB / 2577 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2560 MiB / 4288 MiB / 2577 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2560 MiB / 4320 MiB / 2577 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2560 MiB / 4640 MiB / 2579 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 320 MiB / 2 MiB / 0 MiB / 0 MiB )
20 min passed...2560 MiB / 4704 MiB / 2585 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 64 MiB / 6 MiB / 0 MiB / 0 MiB )
21 min passed...2560 MiB / 4704 MiB / 2587 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
22 min passed...2560 MiB / 4704 MiB / 2591 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
23 min passed...2560 MiB / 4704 MiB / 2591 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2560 MiB / 4704 MiB / 2591 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2560 MiB / 4832 MiB / 2592 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 128 MiB / 1 MiB / 0 MiB / 0 MiB )
26 min passed...2560 MiB / 5056 MiB / 2595 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 224 MiB / 3 MiB / 0 MiB / 0 MiB )
27 min passed...2560 MiB / 5056 MiB / 2595 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2560 MiB / 5056 MiB / 2595 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2560 MiB / 5056 MiB / 2597 MiB / 1056 MiB / 1060 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
30 min passed...2560 MiB / 5056 MiB / 2616 MiB / 1056 MiB / 1061 MiB ( 0 MiB / 0 MiB / 19 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test X6: 4x async sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 3 (f=3): [W(1),_(1),W(2)][100.0%][w=363MiB/s][w=11.6k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5692: Thu Aug 19 17:56:25 2021
  write: IOPS=4078, BW=127MiB/s (134MB/s)(3072MiB/24105msec); 0 zone resets
    slat (usec): min=11, max=20473, avg=51.71, stdev=106.52
    clat (usec): min=2, max=61201, avg=166.63, stdev=477.87
     lat (usec): min=68, max=61291, avg=221.10, stdev=493.83
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   72], 20.00th=[   90],
     | 30.00th=[  104], 40.00th=[  118], 50.00th=[  133], 60.00th=[  149],
     | 70.00th=[  172], 80.00th=[  204], 90.00th=[  260], 95.00th=[  322],
     | 99.00th=[  570], 99.50th=[  840], 99.90th=[ 4490], 99.95th=[ 8455],
     | 99.99th=[19268]
   bw (  KiB/s): min=81856, max=183872, per=25.04%, avg=130336.00, stdev=25765.06, samples=48
   iops        : min= 2558, max= 5746, avg=4072.94, stdev=805.22, samples=48
  lat (usec)   : 4=5.56%, 10=0.50%, 20=0.16%, 50=1.16%, 100=19.87%
  lat (usec)   : 250=61.45%, 500=9.96%, 750=0.76%, 1000=0.20%
  lat (msec)   : 2=0.19%, 4=0.07%, 10=0.08%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=7.68%, sys=14.58%, ctx=135652, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5693: Thu Aug 19 17:56:25 2021
  write: IOPS=4194, BW=131MiB/s (137MB/s)(3072MiB/23436msec); 0 zone resets
    slat (usec): min=10, max=20537, avg=51.53, stdev=105.94
    clat (usec): min=2, max=52783, avg=160.07, stdev=444.13
     lat (usec): min=65, max=52807, avg=214.37, stdev=455.08
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   68], 20.00th=[   87],
     | 30.00th=[  100], 40.00th=[  114], 50.00th=[  129], 60.00th=[  145],
     | 70.00th=[  167], 80.00th=[  198], 90.00th=[  253], 95.00th=[  318],
     | 99.00th=[  537], 99.50th=[  758], 99.90th=[ 3851], 99.95th=[ 7963],
     | 99.99th=[19792]
   bw (  KiB/s): min=83584, max=186496, per=25.80%, avg=134293.91, stdev=25173.34, samples=46
   iops        : min= 2612, max= 5828, avg=4196.65, stdev=786.69, samples=46
  lat (usec)   : 4=6.03%, 10=0.49%, 20=0.18%, 50=1.16%, 100=21.98%
  lat (usec)   : 250=59.79%, 500=9.17%, 750=0.70%, 1000=0.18%
  lat (msec)   : 2=0.15%, 4=0.08%, 10=0.06%, 20=0.02%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=7.41%, sys=15.12%, ctx=135888, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5694: Thu Aug 19 17:56:25 2021
  write: IOPS=4068, BW=127MiB/s (133MB/s)(3072MiB/24162msec); 0 zone resets
    slat (usec): min=11, max=24300, avg=52.78, stdev=115.35
    clat (usec): min=2, max=61320, avg=165.40, stdev=495.59
     lat (usec): min=69, max=61648, avg=220.82, stdev=507.30
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   70], 20.00th=[   89],
     | 30.00th=[  103], 40.00th=[  117], 50.00th=[  131], 60.00th=[  149],
     | 70.00th=[  172], 80.00th=[  202], 90.00th=[  260], 95.00th=[  322],
     | 99.00th=[  562], 99.50th=[  816], 99.90th=[ 4293], 99.95th=[ 8356],
     | 99.99th=[22676]
   bw (  KiB/s): min=82139, max=190400, per=24.98%, avg=129986.19, stdev=24745.58, samples=48
   iops        : min= 2566, max= 5950, avg=4061.98, stdev=773.37, samples=48
  lat (usec)   : 4=5.91%, 10=0.58%, 20=0.22%, 50=1.29%, 100=20.07%
  lat (usec)   : 250=60.88%, 500=9.74%, 750=0.72%, 1000=0.21%
  lat (msec)   : 2=0.18%, 4=0.08%, 10=0.07%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=7.64%, sys=14.58%, ctx=137887, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5695: Thu Aug 19 17:56:25 2021
  write: IOPS=4066, BW=127MiB/s (133MB/s)(3072MiB/24177msec); 0 zone resets
    slat (usec): min=11, max=20868, avg=52.82, stdev=125.53
    clat (usec): min=2, max=48865, avg=165.71, stdev=476.36
     lat (usec): min=69, max=48896, avg=221.16, stdev=490.32
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   70], 20.00th=[   89],
     | 30.00th=[  103], 40.00th=[  117], 50.00th=[  133], 60.00th=[  149],
     | 70.00th=[  172], 80.00th=[  202], 90.00th=[  260], 95.00th=[  322],
     | 99.00th=[  562], 99.50th=[  807], 99.90th=[ 4293], 99.95th=[ 8455],
     | 99.99th=[22152]
   bw (  KiB/s): min=80542, max=183744, per=24.96%, avg=129882.90, stdev=24544.73, samples=48
   iops        : min= 2516, max= 5742, avg=4058.79, stdev=767.06, samples=48
  lat (usec)   : 4=5.84%, 10=0.47%, 20=0.21%, 50=1.34%, 100=19.95%
  lat (usec)   : 250=61.10%, 500=9.77%, 750=0.75%, 1000=0.20%
  lat (msec)   : 2=0.17%, 4=0.09%, 10=0.08%, 20=0.03%, 50=0.01%
  cpu          : usr=7.94%, sys=14.21%, ctx=139318, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=508MiB/s (533MB/s), 127MiB/s-131MiB/s (133MB/s-137MB/s), io=12.0GiB (12.9GB), run=23436-24177msec

Disk stats (read/write):
  sda: ios=0/392886, merge=0/237, ticks=0/64090, in_queue=70956, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 12289 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test X6: 4x async sequential read/write 32K

Guest (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 2 (f=2): [_(2),R(2)][100.0%][r=246MiB/s][r=7864 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5897: Thu Aug 19 18:12:11 2021
  read: IOPS=2199, BW=68.7MiB/s (72.1MB/s)(3072MiB/44701msec)
    slat (usec): min=9, max=258, avg=29.47, stdev= 8.71
    clat (usec): min=85, max=4232, avg=416.58, stdev=61.67
     lat (usec): min=143, max=4271, avg=447.72, stdev=63.18
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  338], 10.00th=[  359], 20.00th=[  379],
     | 30.00th=[  396], 40.00th=[  404], 50.00th=[  416], 60.00th=[  429],
     | 70.00th=[  441], 80.00th=[  457], 90.00th=[  482], 95.00th=[  506],
     | 99.00th=[  553], 99.50th=[  578], 99.90th=[  685], 99.95th=[  742],
     | 99.99th=[ 1020]
   bw (  KiB/s): min=67200, max=116544, per=25.36%, avg=70385.39, stdev=6619.84, samples=89
   iops        : min= 2100, max= 3642, avg=2199.53, stdev=206.87, samples=89
  lat (usec)   : 100=0.01%, 250=2.44%, 500=91.86%, 750=5.66%, 1000=0.03%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.43%, sys=9.79%, ctx=98467, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5898: Thu Aug 19 18:12:11 2021
  read: IOPS=2194, BW=68.6MiB/s (71.9MB/s)(3072MiB/44788msec)
    slat (usec): min=9, max=173, avg=29.46, stdev= 8.78
    clat (usec): min=121, max=4231, avg=417.53, stdev=63.50
     lat (usec): min=138, max=4259, avg=448.65, stdev=64.90
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  338], 10.00th=[  359], 20.00th=[  379],
     | 30.00th=[  396], 40.00th=[  408], 50.00th=[  416], 60.00th=[  429],
     | 70.00th=[  445], 80.00th=[  461], 90.00th=[  486], 95.00th=[  506],
     | 99.00th=[  562], 99.50th=[  586], 99.90th=[  676], 99.95th=[  742],
     | 99.99th=[ 1139]
   bw (  KiB/s): min=66624, max=124800, per=25.31%, avg=70248.07, stdev=6959.18, samples=89
   iops        : min= 2082, max= 3900, avg=2195.24, stdev=217.48, samples=89
  lat (usec)   : 250=2.57%, 500=91.21%, 750=6.17%, 1000=0.03%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.87%, sys=9.33%, ctx=98466, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5899: Thu Aug 19 18:12:11 2021
  read: IOPS=2168, BW=67.8MiB/s (71.1MB/s)(3072MiB/45328msec)
    slat (usec): min=9, max=289, avg=29.67, stdev= 8.44
    clat (usec): min=120, max=4860, avg=422.68, stdev=54.64
     lat (usec): min=212, max=4892, avg=454.05, stdev=55.93
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  351], 10.00th=[  367], 20.00th=[  383],
     | 30.00th=[  396], 40.00th=[  408], 50.00th=[  420], 60.00th=[  433],
     | 70.00th=[  445], 80.00th=[  461], 90.00th=[  486], 95.00th=[  506],
     | 99.00th=[  553], 99.50th=[  586], 99.90th=[  701], 99.95th=[  791],
     | 99.99th=[ 1450]
   bw (  KiB/s): min=66432, max=76288, per=25.01%, avg=69429.10, stdev=1688.54, samples=90
   iops        : min= 2076, max= 2384, avg=2169.64, stdev=52.77, samples=90
  lat (usec)   : 250=0.06%, 500=93.99%, 750=5.88%, 1000=0.05%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.55%, sys=9.65%, ctx=98349, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5900: Thu Aug 19 18:12:11 2021
  read: IOPS=2171, BW=67.8MiB/s (71.1MB/s)(3072MiB/45280msec)
    slat (usec): min=9, max=281, avg=29.68, stdev= 8.71
    clat (usec): min=131, max=5138, avg=422.22, stdev=53.47
     lat (usec): min=172, max=5171, avg=453.56, stdev=54.80
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  351], 10.00th=[  367], 20.00th=[  383],
     | 30.00th=[  396], 40.00th=[  408], 50.00th=[  420], 60.00th=[  429],
     | 70.00th=[  445], 80.00th=[  461], 90.00th=[  482], 95.00th=[  506],
     | 99.00th=[  562], 99.50th=[  586], 99.90th=[  685], 99.95th=[  750],
     | 99.99th=[  979]
   bw (  KiB/s): min=66880, max=76352, per=25.03%, avg=69488.88, stdev=1783.28, samples=90
   iops        : min= 2090, max= 2386, avg=2171.53, stdev=55.76, samples=90
  lat (usec)   : 250=0.15%, 500=93.87%, 750=5.93%, 1000=0.04%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.65%, sys=9.57%, ctx=98432, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=271MiB/s (284MB/s), 67.8MiB/s-68.7MiB/s (71.1MB/s-72.1MB/s), io=12.0GiB (12.9GB), run=44701-45328msec

Disk stats (read/write):
  sda: ios=392571/0, merge=0/0, ticks=163674/0, in_queue=174508, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Script ran 1871 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...30432 MiB / 32256 MiB / 30432 MiB / 64 MiB / 75 MiB ( 30432 MiB / 32256 MiB / 30432 MiB / 64 MiB / 75 MiB )
2 min passed...30432 MiB / 32640 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...30432 MiB / 32896 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...30432 MiB / 32896 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...30432 MiB / 32896 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...30432 MiB / 32896 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...30432 MiB / 33024 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...30432 MiB / 33120 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...30432 MiB / 33120 MiB / 30432 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...30432 MiB / 33120 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
11 min passed...30432 MiB / 33120 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...30432 MiB / 33184 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...30432 MiB / 33440 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...30432 MiB / 33856 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...30432 MiB / 33856 MiB / 30437 MiB / 64 MiB / 75 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...30432 MiB / 33856 MiB / 30437 MiB / 11328 MiB / 11334 MiB ( 0 MiB / 0 MiB / 0 MiB / 11264 MiB / 11259 MiB )
17 min passed...30432 MiB / 33856 MiB / 30437 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 0 MiB / 0 MiB / 832 MiB / 827 MiB )
18 min passed...30432 MiB / 33920 MiB / 30437 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...30432 MiB / 34272 MiB / 30438 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 352 MiB / 1 MiB / 0 MiB / 0 MiB )
20 min passed...30432 MiB / 34432 MiB / 30443 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 160 MiB / 5 MiB / 0 MiB / 0 MiB )
21 min passed...30432 MiB / 34432 MiB / 30445 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
22 min passed...30432 MiB / 34432 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 0 MiB )
23 min passed...30432 MiB / 34656 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...30432 MiB / 34784 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...30432 MiB / 35072 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...30432 MiB / 35136 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...30432 MiB / 35136 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...30432 MiB / 35136 MiB / 30452 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...30432 MiB / 35456 MiB / 30454 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 320 MiB / 2 MiB / 0 MiB / 0 MiB )
30 min passed...30432 MiB / 35776 MiB / 30459 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 320 MiB / 5 MiB / 0 MiB / 0 MiB )
31 min passed...30432 MiB / 35808 MiB / 30462 MiB / 12160 MiB / 12161 MiB ( 0 MiB / 32 MiB / 3 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test X7: 4x sync sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=20.3MiB/s][w=648 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6141: Thu Aug 19 18:42:11 2021
  write: IOPS=155, BW=4978KiB/s (5097kB/s)(512MiB/105327msec); 0 zone resets
    clat (usec): min=3680, max=88150, avg=6401.88, stdev=2257.68
     lat (usec): min=3681, max=88152, avg=6404.30, stdev=2258.48
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5342], 10.00th=[ 5473], 20.00th=[ 5669],
     | 30.00th=[ 5800], 40.00th=[ 5932], 50.00th=[ 6063], 60.00th=[ 6194],
     | 70.00th=[ 6325], 80.00th=[ 6521], 90.00th=[ 7242], 95.00th=[ 8356],
     | 99.00th=[14615], 99.50th=[22938], 99.90th=[33817], 99.95th=[43254],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3200, max= 5696, per=24.99%, avg=4974.79, stdev=451.12, samples=210
   iops        : min=  100, max=  178, avg=155.44, stdev=14.10, samples=210
  lat (msec)   : 4=0.03%, 10=98.69%, 20=0.52%, 50=0.74%, 100=0.02%
  cpu          : usr=0.53%, sys=2.23%, ctx=49196, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6142: Thu Aug 19 18:42:11 2021
  write: IOPS=155, BW=4978KiB/s (5097kB/s)(512MiB/105331msec); 0 zone resets
    clat (usec): min=3368, max=87964, avg=6401.86, stdev=2281.84
     lat (usec): min=3370, max=87965, avg=6404.23, stdev=2281.80
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5342], 10.00th=[ 5473], 20.00th=[ 5669],
     | 30.00th=[ 5800], 40.00th=[ 5932], 50.00th=[ 6063], 60.00th=[ 6194],
     | 70.00th=[ 6325], 80.00th=[ 6521], 90.00th=[ 7308], 95.00th=[ 8356],
     | 99.00th=[14222], 99.50th=[22938], 99.90th=[34341], 99.95th=[44303],
     | 99.99th=[61604]
   bw (  KiB/s): min= 3200, max= 5632, per=24.98%, avg=4974.43, stdev=451.91, samples=210
   iops        : min=  100, max=  176, avg=155.43, stdev=14.12, samples=210
  lat (msec)   : 4=0.04%, 10=98.68%, 20=0.51%, 50=0.73%, 100=0.03%
  cpu          : usr=0.60%, sys=2.16%, ctx=49200, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6143: Thu Aug 19 18:42:11 2021
  write: IOPS=155, BW=4978KiB/s (5097kB/s)(512MiB/105327msec); 0 zone resets
    clat (usec): min=3625, max=88078, avg=6401.61, stdev=2273.64
     lat (usec): min=3626, max=88081, avg=6404.00, stdev=2273.64
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5342], 10.00th=[ 5473], 20.00th=[ 5669],
     | 30.00th=[ 5800], 40.00th=[ 5932], 50.00th=[ 6063], 60.00th=[ 6194],
     | 70.00th=[ 6325], 80.00th=[ 6521], 90.00th=[ 7308], 95.00th=[ 8356],
     | 99.00th=[14222], 99.50th=[22938], 99.90th=[34341], 99.95th=[44303],
     | 99.99th=[61604]
   bw (  KiB/s): min= 3200, max= 5632, per=24.99%, avg=4974.78, stdev=449.99, samples=210
   iops        : min=  100, max=  176, avg=155.44, stdev=14.06, samples=210
  lat (msec)   : 4=0.04%, 10=98.68%, 20=0.51%, 50=0.74%, 100=0.02%
  cpu          : usr=0.60%, sys=2.21%, ctx=49208, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6144: Thu Aug 19 18:42:11 2021
  write: IOPS=155, BW=4978KiB/s (5097kB/s)(512MiB/105325msec); 0 zone resets
    clat (usec): min=3741, max=69344, avg=6401.39, stdev=2232.45
     lat (usec): min=3853, max=69346, avg=6403.83, stdev=2232.42
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5342], 10.00th=[ 5473], 20.00th=[ 5669],
     | 30.00th=[ 5800], 40.00th=[ 5932], 50.00th=[ 6063], 60.00th=[ 6194],
     | 70.00th=[ 6325], 80.00th=[ 6521], 90.00th=[ 7242], 95.00th=[ 8356],
     | 99.00th=[14353], 99.50th=[23200], 99.90th=[34866], 99.95th=[45351],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3200, max= 5696, per=24.99%, avg=4974.79, stdev=448.68, samples=210
   iops        : min=  100, max=  178, avg=155.44, stdev=14.02, samples=210
  lat (msec)   : 4=0.04%, 10=98.70%, 20=0.48%, 50=0.75%, 100=0.02%
  cpu          : usr=0.51%, sys=2.34%, ctx=49202, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=19.4MiB/s (20.4MB/s), 4978KiB/s-4978KiB/s (5097kB/s-5097kB/s), io=2048MiB (2147MB), run=105325-105331msec

Disk stats (read/write):
  sda: ios=0/130870, merge=0/130847, ticks=0/120691, in_queue=129820, util=99.56%
0 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 2817 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2817 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test X7: 4x sync sequential read/write 32K

Guest (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=273MiB/s][r=8738 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6347: Thu Aug 19 18:57:20 2021
  read: IOPS=2171, BW=67.9MiB/s (71.2MB/s)(512MiB/7544msec)
    clat (usec): min=316, max=1060, avg=451.94, stdev=49.05
     lat (usec): min=317, max=1062, avg=453.62, stdev=49.15
    clat percentiles (usec):
     |  1.00th=[  359],  5.00th=[  383], 10.00th=[  396], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  461],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  515], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  619], 99.90th=[  709], 99.95th=[  775],
     | 99.99th=[  971]
   bw (  KiB/s): min=67712, max=73984, per=24.99%, avg=69482.00, stdev=1545.05, samples=15
   iops        : min= 2116, max= 2312, avg=2171.27, stdev=48.31, samples=15
  lat (usec)   : 500=85.29%, 750=14.64%, 1000=0.06%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.49%, sys=8.50%, ctx=16387, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6348: Thu Aug 19 18:57:20 2021
  read: IOPS=2175, BW=67.0MiB/s (71.3MB/s)(512MiB/7532msec)
    clat (usec): min=298, max=1010, avg=451.18, stdev=48.76
     lat (usec): min=300, max=1011, avg=452.84, stdev=48.85
    clat percentiles (usec):
     |  1.00th=[  359],  5.00th=[  379], 10.00th=[  396], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  457],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  619], 99.90th=[  725], 99.95th=[  799],
     | 99.99th=[  947]
   bw (  KiB/s): min=68032, max=74304, per=25.04%, avg=69605.67, stdev=1563.52, samples=15
   iops        : min= 2126, max= 2322, avg=2175.13, stdev=48.87, samples=15
  lat (usec)   : 500=85.97%, 750=13.95%, 1000=0.07%
  lat (msec)   : 2=0.01%
  cpu          : usr=2.28%, sys=9.69%, ctx=16394, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6349: Thu Aug 19 18:57:20 2021
  read: IOPS=2175, BW=67.0MiB/s (71.3MB/s)(512MiB/7530msec)
    clat (usec): min=312, max=2670, avg=451.16, stdev=51.26
     lat (usec): min=313, max=2671, avg=452.81, stdev=51.33
    clat percentiles (usec):
     |  1.00th=[  359],  5.00th=[  379], 10.00th=[  396], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  457],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  693], 99.95th=[  766],
     | 99.99th=[ 1237]
   bw (  KiB/s): min=67840, max=74240, per=25.04%, avg=69610.00, stdev=1569.42, samples=15
   iops        : min= 2120, max= 2320, avg=2175.27, stdev=49.07, samples=15
  lat (usec)   : 500=86.06%, 750=13.89%, 1000=0.04%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.82%, sys=9.00%, ctx=16385, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6350: Thu Aug 19 18:57:20 2021
  read: IOPS=2176, BW=68.0MiB/s (71.3MB/s)(512MiB/7528msec)
    clat (usec): min=306, max=1090, avg=451.06, stdev=48.53
     lat (usec): min=307, max=1092, avg=452.72, stdev=48.63
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  379], 10.00th=[  396], 20.00th=[  412],
     | 30.00th=[  424], 40.00th=[  437], 50.00th=[  449], 60.00th=[  457],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  515], 95.00th=[  537],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  676], 99.95th=[  734],
     | 99.99th=[ 1029]
   bw (  KiB/s): min=67520, max=74560, per=25.05%, avg=69648.40, stdev=1628.40, samples=15
   iops        : min= 2110, max= 2330, avg=2176.47, stdev=50.92, samples=15
  lat (usec)   : 500=85.26%, 750=14.71%, 1000=0.02%
  lat (msec)   : 2=0.01%
  cpu          : usr=2.14%, sys=9.62%, ctx=16388, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=271MiB/s (285MB/s), 67.9MiB/s-68.0MiB/s (71.2MB/s-71.3MB/s), io=2048MiB (2147MB), run=7528-7544msec

Disk stats (read/write):
  sda: ios=64351/0, merge=0/0, ticks=26997/0, in_queue=28756, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2817 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Script ran 1914 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...11584 MiB / 12352 MiB / 11614 MiB / 128 MiB / 140 MiB ( 11584 MiB / 12352 MiB / 11614 MiB / 128 MiB / 140 MiB )
2 min passed...21376 MiB / 23136 MiB / 21402 MiB / 256 MiB / 256 MiB ( 9792 MiB / 10784 MiB / 9788 MiB / 128 MiB / 116 MiB )
3 min passed...21376 MiB / 23488 MiB / 21402 MiB / 256 MiB / 256 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...21376 MiB / 23808 MiB / 21402 MiB / 256 MiB / 256 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...21376 MiB / 24096 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 288 MiB / 5 MiB / 0 MiB / 0 MiB )
6 min passed...21376 MiB / 24128 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...21376 MiB / 24128 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...21376 MiB / 24512 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...21376 MiB / 25056 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...21376 MiB / 25280 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...21376 MiB / 25472 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...21376 MiB / 25472 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...21376 MiB / 25536 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...21376 MiB / 26016 MiB / 21407 MiB / 256 MiB / 256 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...21376 MiB / 26176 MiB / 21411 MiB / 256 MiB / 256 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
16 min passed...21376 MiB / 26464 MiB / 21411 MiB / 256 MiB / 256 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...21376 MiB / 26496 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 32 MiB / 0 MiB / 2048 MiB / 2048 MiB )
18 min passed...21376 MiB / 26496 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...21376 MiB / 26720 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...21376 MiB / 26912 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...21376 MiB / 27168 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...21376 MiB / 27392 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...21376 MiB / 27392 MiB / 21411 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...21376 MiB / 27488 MiB / 21415 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 4 MiB / 0 MiB / 0 MiB )
25 min passed...21376 MiB / 27648 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
26 min passed...21376 MiB / 27904 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...21376 MiB / 28256 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...21376 MiB / 28256 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...21376 MiB / 28256 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...21376 MiB / 28416 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...21376 MiB / 28512 MiB / 21419 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

This is the pool and VM that will be used in Round 2 Run 8:

Host is a striped raidz1 (2x raidz1 of 3 SSDs) of 6x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=32K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 7 are bold)

root@Hypervisor:~# zpool status

Code:

 pool: VMpool13
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool13                                        ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN  ONLINE       0     0     0
          raidz1-1                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN  ONLINE       0     0     0

errors: No known data errors

root@Hypervisor:~# zpool get all

Code:

NAME      PROPERTY                       VALUE                          SOURCE
VMpool13  size                           1.09T                          -
VMpool13  capacity                       0%                             -
VMpool13  altroot                        -                              default
VMpool13  health                         ONLINE                         -
VMpool13  guid                           6114855189765257253            -
VMpool13  version                        -                              default
VMpool13  bootfs                         -                              default
VMpool13  delegation                     on                             default
VMpool13  autoreplace                    off                            default
VMpool13  cachefile                      -                              default
VMpool13  failmode                       wait                           default
VMpool13  listsnapshots                  off                            default
VMpool13  autoexpand                     off                            default
VMpool13  dedupratio                     1.00x                          -
VMpool13  free                           1.09T                          -
VMpool13  allocated                      9.30M                          -
VMpool13  readonly                       off                            -
VMpool13  ashift                         12                             local
VMpool13  comment                        -                              default
VMpool13  expandsize                     -                              -
VMpool13  freeing                        0                              -
VMpool13  fragmentation                  0%                             -
VMpool13  leaked                         0                              -
VMpool13  multihost                      off                            default
VMpool13  checkpoint                     -                              -
VMpool13  load_guid                      4511644241695424845            -
VMpool13  autotrim                       off                            default
VMpool13  feature@async_destroy          enabled                        local
VMpool13  feature@empty_bpobj            active                         local
VMpool13  feature@lz4_compress           active                         local
VMpool13  feature@multi_vdev_crash_dump  enabled                        local
VMpool13  feature@spacemap_histogram     active                         local
VMpool13  feature@enabled_txg            active                         local
VMpool13  feature@hole_birth             active                         local
VMpool13  feature@extensible_dataset     active                         local
VMpool13  feature@embedded_data          active                         local
VMpool13  feature@bookmarks              enabled                        local
VMpool13  feature@filesystem_limits      enabled                        local
VMpool13  feature@large_blocks           enabled                        local
VMpool13  feature@large_dnode            enabled                        local
VMpool13  feature@sha512                 enabled                        local
VMpool13  feature@skein                  enabled                        local
VMpool13  feature@edonr                  enabled                        local
VMpool13  feature@userobj_accounting     active                         local
VMpool13  feature@encryption             active                         local
VMpool13  feature@project_quota          active                         local
VMpool13  feature@device_removal         enabled                        local
VMpool13  feature@obsolete_counts        enabled                        local
VMpool13  feature@zpool_checkpoint       enabled                        local
VMpool13  feature@spacemap_v2            active                         local
VMpool13  feature@allocation_classes     enabled                        local
VMpool13  feature@resilver_defer         enabled                        local
VMpool13  feature@bookmark_v2            enabled                        local
VMpool13  feature@redaction_bookmarks    enabled                        local
VMpool13  feature@redacted_datasets      enabled                        local
VMpool13  feature@bookmark_written       enabled                        local
VMpool13  feature@log_spacemap           active                         local
VMpool13  feature@livelist               enabled                        local
VMpool13  feature@device_rebuild         enabled                        local
VMpool13  feature@zstd_compress          enabled                        local

root@Hypervisor:~# zfs get all VMpool13/VLT/VM

Code:

NAME             PROPERTY              VALUE                  SOURCE
VMpool13/VLT/VM  type                  filesystem             -
VMpool13/VLT/VM  creation              Sat Aug 21  3:25 2021  -
VMpool13/VLT/VM  used                  32.3G                  -
VMpool13/VLT/VM  available             544G                   -
VMpool13/VLT/VM  referenced            234K                   -
VMpool13/VLT/VM  compressratio         5.77x                  -
VMpool13/VLT/VM  mounted               yes                    -
VMpool13/VLT/VM  quota                 none                   default
VMpool13/VLT/VM  reservation           none                   default
VMpool13/VLT/VM  recordsize            128K                   default
VMpool13/VLT/VM  mountpoint            /VMpool13/VLT/VM       default
VMpool13/VLT/VM  sharenfs              off                    default
VMpool13/VLT/VM  checksum              on                     default
VMpool13/VLT/VM  compression           lz4                    inherited from VMpool13
VMpool13/VLT/VM  atime                 off                    inherited from VMpool13
VMpool13/VLT/VM  devices               on                     default
VMpool13/VLT/VM  exec                  on                     default
VMpool13/VLT/VM  setuid                on                     default
VMpool13/VLT/VM  readonly              off                    default
VMpool13/VLT/VM  zoned                 off                    default
VMpool13/VLT/VM  snapdir               hidden                 default
VMpool13/VLT/VM  aclmode               discard                default
VMpool13/VLT/VM  aclinherit            restricted             default
VMpool13/VLT/VM  createtxg             39                     -
VMpool13/VLT/VM  canmount              on                     local
VMpool13/VLT/VM  xattr                 on                     default
VMpool13/VLT/VM  copies                1                      default
VMpool13/VLT/VM  version               5                      -
VMpool13/VLT/VM  utf8only              off                    -
VMpool13/VLT/VM  normalization         none                   -
VMpool13/VLT/VM  casesensitivity       sensitive              -
VMpool13/VLT/VM  vscan                 off                    default
VMpool13/VLT/VM  nbmand                off                    default
VMpool13/VLT/VM  sharesmb              off                    default
VMpool13/VLT/VM  refquota              none                   default
VMpool13/VLT/VM  refreservation        none                   default
VMpool13/VLT/VM  guid                  14481661454458275882   -
VMpool13/VLT/VM  primarycache          metadata               inherited from VMpool13
VMpool13/VLT/VM  secondarycache        all                    default
VMpool13/VLT/VM  usedbysnapshots       0B                     -
VMpool13/VLT/VM  usedbydataset         234K                   -
VMpool13/VLT/VM  usedbychildren        32.3G                  -
VMpool13/VLT/VM  usedbyrefreservation  0B                     -
VMpool13/VLT/VM  logbias               latency                default
VMpool13/VLT/VM  objsetid              143                    -
VMpool13/VLT/VM  dedup                 off                    default
VMpool13/VLT/VM  mlslabel              none                   default
VMpool13/VLT/VM  sync                  standard               inherited from VMpool13
VMpool13/VLT/VM  dnodesize             legacy                 default
VMpool13/VLT/VM  refcompressratio      1.00x                  -
VMpool13/VLT/VM  written               234K                   -
VMpool13/VLT/VM  logicalused           5.36M                  -
VMpool13/VLT/VM  logicalreferenced     69K                    -
VMpool13/VLT/VM  volmode               default                default
VMpool13/VLT/VM  filesystem_limit      none                   default
VMpool13/VLT/VM  snapshot_limit        none                   default
VMpool13/VLT/VM  filesystem_count      none                   default
VMpool13/VLT/VM  snapshot_count        none                   default
VMpool13/VLT/VM  snapdev               hidden                 default
VMpool13/VLT/VM  acltype               off                    default
VMpool13/VLT/VM  context               none                   default
VMpool13/VLT/VM  fscontext             none                   default
VMpool13/VLT/VM  defcontext            none                   default
VMpool13/VLT/VM  rootcontext           none                   default
VMpool13/VLT/VM  relatime              off                    default
VMpool13/VLT/VM  redundant_metadata    all                    default
VMpool13/VLT/VM  overlay               on                     default
VMpool13/VLT/VM  encryption            aes-256-gcm            -
VMpool13/VLT/VM  keylocation           none                   default
VMpool13/VLT/VM  keyformat             passphrase             -
VMpool13/VLT/VM  pbkdf2iters           350000                 -
VMpool13/VLT/VM  encryptionroot        VMpool13/VLT           -
VMpool13/VLT/VM  keystatus             available              -
VMpool13/VLT/VM  special_small_blocks  0                      default

Commands used to create the pool:

Code:

# create mirror of 6 disks (6x Intel S3710 200GB):
zpool create -f -o ashift=12 VMpool13 raidz /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN raidz /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN
# set ZFS options
zpool set autoexpand=off VMpool13
zpool set feature@encryption=enabled VMpool13
zfs set compression=lz4 VMpool13
zfs set sync=standard VMpool13
zfs set canmount=off VMpool13
zfs set atime=off VMpool13
# set quota to 80% so pool can't get full/slow:
zfs set quota=576G VMpool13
# create encrypted dataset (256bit aes):
zfs create -o encryption=on -o keyformat=passphrase VMpool13/VLT
zfs set canmount=off VMpool13/VLT
# create dataset for encrypted VMs
zfs create -o encryption=on VMpool13/VLT/VM
zfs set canmount=on VMpool13/VLT/VM
# add dataset to PVE:
pvesm add zfspool VMpool13_VLT_VM -pool VMpool13/VLT/VM --content images,rootdir --sparse 1 --mountpoint /VMpool13/VLT/VM
# disable ARC caching:
zfs set primarycache=metadata VMpool13
# don't forget to set block size from 8K to 32K using WebUI

Commands to create test zvol:

Code:

zfs create -o volblocksize=32k -V 32G VMpool13/VLT/VM/testvol
fdisk /dev/VMpool13/VLT/VM/testvol
g
n
enter
enter
enter
w
mkfs.ext4 -b 4096 /dev/zd0p1
mkdir /mnt/test
mount -t ext4 -o noatime,nodiratime /dev/zd0p1 /mnt/test

Dunuin · Aug 21, 2021

Here are the results of a 6 disk striped mirror 8K vs 16K volblocksize (run 6 vs run 7). Theoretically 12K volblocksize should be optimal but ZFS doesn't allow this.

Write Performance (MiB/s):

	8K bs (guest)	16K bs (guest)	8K bs (host)	16K bs (host)
sync 4K random (T1)	1,25	1,15
sync 16K random (T2)	5,52	5,66
sync 32K random (T8)	9,91	9,78
4x sync 32K sequential (TX7)	19,40	18,90	20,10	26,30
sync 4M sequential (T3)	202,00	211,00
async 4K random (T4)	120,00	109,00
async 16K random (T5)	430,00	446,00
async 16K random 50% compressible (T7)	469,00	472,00
4x 16K async random (TX9)			413,00	506,00
async 32K random (T9)	691,00	760,00
4x async 32K sequential (TX6)	508,00	546,00	663,00	720,00
4x 128k async sequential (TX8)			562,00	648,00
async 4M sequential (T6)	528,00	636,00

Read Performance (MiB/s):

	8K bs (guest)	16K bs (guest)	8K bs (host)	16K bs (host)
sync 4K random (T1)	8,18	7,30
sync 16K random (T2)	30,30	27,00
sync 32K random (T8)	58,20	46,70
4x sync 32K sequential (TX7)	271,00	259,00	342,00	342,00
sync 4M sequential (T3)	765,00	994,00
async 4K random (T4)	272,00	271,00
async 16K random (T5)	790,00	963,00
async 16K random 50% compressible (T7)	1.039,00	1.028,00
4x 16K async random (TX9)			206,00	207,00
async 32K random (T9)	1.255,00	928,00
4x async 32K sequential (TX6)	271,00	259,00	303,00	341,00
4x 128k async sequential (TX8)			717,00	831,00
async 4M sequential (T6)	1.593,00	1.581,00

Total Write Amplification from fio to NAND (factor):

	8K bs (guest)	16K bs (guest)	8K bs (host)	16K bs (host)
sync 4K random (T1)	62,44	59,00
sync 16K random (T2)	18,75	18,06
sync 32K random (T8)	15,03	16,50
4x sync 32K sequential (TX7)	12,92	13,02	13,91	13,08
sync 4M sequential (T3)	4,83	4,82
async 4K random (T4)	6,53	7,53
async 16K random (T5)	7,31	5,25
async 16K random 50% compressible (T7)	4,66	4,91
4x 16K async random (TX9)			3,21	2,64
async 32K random (T9)	4,19	6,88
4x async 32K sequential (TX6)	2,76	2,78	2,46	2,49
4x 128k async sequential (TX8)			2,25	2,35
async 4M sequential (T6)	3,27	2,68

Most reliable numbers for performance should be TX6, TX7, TX8 and TX9. Especially if you look at the numbers from the fio test ran dicrectly on the host the 16K volblocksize was performing better.

Here are the fio commands for TX6-9.

TX6:

Code:

fio --directory=/mnt/test/ --name=async_seq_32K --rw=write --bs=32K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=3G
fio --directory=/mnt/test/ --name=async_seq_32K --rw=read --bs=32K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=3G

TX7:

Code:

fio --directory=/mnt/test/ --name=sync_seq_32K --rw=write --bs=32K --direct=1 --sync=1 --numjobs=4 --ioengine=psync --iodepth=1 --refill_buffers --size=512M
fio --directory=/mnt/test/ --name=sync_seq_32K --rw=read --bs=32K --direct=1 --sync=1 --numjobs=4 --ioengine=psync --iodepth=1 --refill_buffers --size=512M

TX8:

Code:

fio --directory=/mnt/test/ --name=async_seq_128K --rw=write --bs=128K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=7G
fio --directory=/mnt/test/ --name=async_seq_128K --rw=read --bs=128K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=7G

TX9:

Code:

fio --directory=/mnt/test/ --name=async_rand_16K --rw=randwrite --bs=16K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=7G
fio --directory=/mnt/test/ --name=async_rand_16K --rw=randread --bs=16K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=7G

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1277KiB/s][w=319 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=771: Sat Aug 21 04:09:52 2021
  write: IOPS=317, BW=1269KiB/s (1299kB/s)(1024MiB/826513msec); 0 zone resets
    clat (usec): min=774, max=55561, avg=3137.97, stdev=1124.87
     lat (usec): min=776, max=55562, avg=3140.18, stdev=1124.88
    clat percentiles (usec):
     |  1.00th=[ 1565],  5.00th=[ 1893], 10.00th=[ 2180], 20.00th=[ 2507],
     | 30.00th=[ 2704], 40.00th=[ 2868], 50.00th=[ 2999], 60.00th=[ 3195],
     | 70.00th=[ 3425], 80.00th=[ 3654], 90.00th=[ 3949], 95.00th=[ 4555],
     | 99.00th=[ 6128], 99.50th=[ 6652], 99.90th=[16909], 99.95th=[21365],
     | 99.99th=[28967]
   bw (  KiB/s): min=  960, max= 1912, per=100.00%, avg=1268.61, stdev=136.54, samples=1653
   iops        : min=  240, max=  478, avg=317.15, stdev=34.13, samples=1653
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=6.82%, 4=84.50%, 10=8.42%, 20=0.18%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.72%, sys=4.48%, ctx=648709, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1269KiB/s (1299kB/s), 1269KiB/s-1269KiB/s (1299kB/s-1299kB/s), io=1024MiB (1074MB), run=826513-826513msec

Disk stats (read/write):
  sda: ios=0/798070, merge=0/548159, ticks=0/781820, in_queue=836320, util=99.85%
0 min passed...Guest Writes/Reads (Changes): 5253 MiB / 17 MiB ( 5253 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 5254 MiB / 17 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5254 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5254 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5254 MiB / 17 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5254 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5255 MiB / 17 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5256 MiB / 17 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5256 MiB / 17 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=6162KiB/s][r=1540 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=980: Sat Aug 21 04:27:40 2021
  read: IOPS=1570, BW=6281KiB/s (6432kB/s)(1024MiB/166949msec)
    clat (usec): min=231, max=5774, avg=625.40, stdev=96.64
     lat (usec): min=233, max=5776, avg=627.41, stdev=96.72
    clat percentiles (usec):
     |  1.00th=[  437],  5.00th=[  478], 10.00th=[  506], 20.00th=[  545],
     | 30.00th=[  578], 40.00th=[  603], 50.00th=[  635], 60.00th=[  660],
     | 70.00th=[  676], 80.00th=[  701], 90.00th=[  725], 95.00th=[  750],
     | 99.00th=[  775], 99.50th=[  791], 99.90th=[  947], 99.95th=[ 1352],
     | 99.99th=[ 3195]
   bw (  KiB/s): min= 5528, max= 7808, per=100.00%, avg=6281.01, stdev=468.36, samples=333
   iops        : min= 1382, max= 1952, avg=1570.25, stdev=117.09, samples=333
  lat (usec)   : 250=0.01%, 500=8.73%, 750=86.87%, 1000=4.32%
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.01%
  cpu          : usr=3.03%, sys=8.93%, ctx=262289, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6281KiB/s (6432kB/s), 6281KiB/s-6281KiB/s (6432kB/s-6432kB/s), io=1024MiB (1074MB), run=166949-166949msec

Disk stats (read/write):
  sda: ios=262069/51, merge=0/44, ticks=153131/59, in_queue=166960, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 5256 MiB / 1041 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5256 MiB / 1041 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5256 MiB / 1041 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5256 MiB / 1041 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5256 MiB / 1041 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5257 MiB / 1041 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5258 MiB / 1041 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5258 MiB / 1041 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5258 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5258 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Script ran 2796 seconds or 46 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3168 MiB / 3680 MiB / 3152 MiB / 288 MiB / 286 MiB ( 3168 MiB / 3680 MiB / 3152 MiB / 288 MiB / 286 MiB )
2 min passed...6208 MiB / 6880 MiB / 6212 MiB / 672 MiB / 725 MiB ( 3040 MiB / 3200 MiB / 3060 MiB / 384 MiB / 439 MiB )
3 min passed...9184 MiB / 10016 MiB / 9181 MiB / 1248 MiB / 1260 MiB ( 2976 MiB / 3136 MiB / 2969 MiB / 576 MiB / 535 MiB )
4 min passed...12160 MiB / 13184 MiB / 12176 MiB / 1856 MiB / 1888 MiB ( 2976 MiB / 3168 MiB / 2995 MiB / 608 MiB / 628 MiB )
5 min passed...15104 MiB / 16512 MiB / 15075 MiB / 2560 MiB / 2586 MiB ( 2944 MiB / 3328 MiB / 2899 MiB / 704 MiB / 698 MiB )
6 min passed...18016 MiB / 19776 MiB / 18006 MiB / 3328 MiB / 3340 MiB ( 2912 MiB / 3264 MiB / 2931 MiB / 768 MiB / 754 MiB )
7 min passed...20960 MiB / 22848 MiB / 20948 MiB / 4128 MiB / 4144 MiB ( 2944 MiB / 3072 MiB / 2942 MiB / 800 MiB / 804 MiB )
8 min passed...23968 MiB / 26112 MiB / 23960 MiB / 5024 MiB / 5024 MiB ( 3008 MiB / 3264 MiB / 3012 MiB / 896 MiB / 880 MiB )
9 min passed...26944 MiB / 29184 MiB / 26943 MiB / 5920 MiB / 5936 MiB ( 2976 MiB / 3072 MiB / 2983 MiB / 896 MiB / 912 MiB )
10 min passed...29920 MiB / 32576 MiB / 29939 MiB / 6848 MiB / 6887 MiB ( 2976 MiB / 3392 MiB / 2996 MiB / 928 MiB / 951 MiB )
11 min passed...32832 MiB / 36096 MiB / 32840 MiB / 7904 MiB / 7863 MiB ( 2912 MiB / 3520 MiB / 2901 MiB / 1056 MiB / 976 MiB )
12 min passed...35872 MiB / 39264 MiB / 35835 MiB / 8896 MiB / 8847 MiB ( 3040 MiB / 3168 MiB / 2995 MiB / 992 MiB / 984 MiB )
13 min passed...38784 MiB / 42720 MiB / 38797 MiB / 9824 MiB / 9784 MiB ( 2912 MiB / 3456 MiB / 2962 MiB / 928 MiB / 937 MiB )
14 min passed...40960 MiB / 45024 MiB / 40951 MiB / 10368 MiB / 10333 MiB ( 2176 MiB / 2304 MiB / 2154 MiB / 544 MiB / 549 MiB )
15 min passed...40960 MiB / 45184 MiB / 40959 MiB / 10368 MiB / 10333 MiB ( 0 MiB / 160 MiB / 8 MiB / 0 MiB / 0 MiB )
16 min passed...40992 MiB / 45504 MiB / 40969 MiB / 10368 MiB / 10333 MiB ( 32 MiB / 320 MiB / 10 MiB / 0 MiB / 0 MiB )
17 min passed...40992 MiB / 45600 MiB / 40979 MiB / 10368 MiB / 10333 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...41024 MiB / 45824 MiB / 40989 MiB / 10368 MiB / 10334 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 1 MiB )
19 min passed...41056 MiB / 45952 MiB / 40998 MiB / 10368 MiB / 10334 MiB ( 32 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
20 min passed...41056 MiB / 46080 MiB / 41008 MiB / 10368 MiB / 10334 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...41056 MiB / 46144 MiB / 41017 MiB / 10368 MiB / 10334 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...41056 MiB / 46176 MiB / 41027 MiB / 10368 MiB / 10334 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
23 min passed...41056 MiB / 46368 MiB / 41036 MiB / 10368 MiB / 10334 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
24 min passed...41056 MiB / 46432 MiB / 41045 MiB / 10368 MiB / 10335 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 1 MiB )
25 min passed...41056 MiB / 46592 MiB / 41054 MiB / 10368 MiB / 10335 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...41056 MiB / 46848 MiB / 41064 MiB / 10368 MiB / 10335 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...41056 MiB / 47104 MiB / 41072 MiB / 10368 MiB / 10335 MiB ( 0 MiB / 256 MiB / 8 MiB / 0 MiB / 0 MiB )
28 min passed...41056 MiB / 47328 MiB / 41080 MiB / 10368 MiB / 10335 MiB ( 0 MiB / 224 MiB / 8 MiB / 0 MiB / 0 MiB )
29 min passed...41088 MiB / 47424 MiB / 41089 MiB / 11616 MiB / 11676 MiB ( 32 MiB / 96 MiB / 9 MiB / 1248 MiB / 1341 MiB )
30 min passed...41152 MiB / 47424 MiB / 41099 MiB / 14688 MiB / 14693 MiB ( 64 MiB / 0 MiB / 10 MiB / 3072 MiB / 3017 MiB )
31 min passed...41152 MiB / 47456 MiB / 41107 MiB / 17632 MiB / 17628 MiB ( 0 MiB / 32 MiB / 8 MiB / 2944 MiB / 2935 MiB )
32 min passed...41152 MiB / 47648 MiB / 41116 MiB / 18560 MiB / 18527 MiB ( 0 MiB / 192 MiB / 9 MiB / 928 MiB / 899 MiB )
33 min passed...41152 MiB / 47776 MiB / 41123 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 128 MiB / 7 MiB / 0 MiB / 1 MiB )
34 min passed...41152 MiB / 47776 MiB / 41131 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
35 min passed...41152 MiB / 48064 MiB / 41140 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
36 min passed...41184 MiB / 48224 MiB / 41149 MiB / 18560 MiB / 18528 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
37 min passed...41184 MiB / 48480 MiB / 41159 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
38 min passed...41184 MiB / 48576 MiB / 41168 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 96 MiB / 9 MiB / 0 MiB / 0 MiB )
39 min passed...41184 MiB / 48608 MiB / 41176 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
40 min passed...41184 MiB / 48864 MiB / 41185 MiB / 18560 MiB / 18528 MiB ( 0 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
41 min passed...41216 MiB / 48960 MiB / 41193 MiB / 18560 MiB / 18529 MiB ( 32 MiB / 96 MiB / 8 MiB / 0 MiB / 1 MiB )
42 min passed...41216 MiB / 49152 MiB / 41203 MiB / 18560 MiB / 18529 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
43 min passed...41248 MiB / 49312 MiB / 41212 MiB / 18560 MiB / 18529 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
44 min passed...41248 MiB / 49312 MiB / 41221 MiB / 18560 MiB / 18529 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
45 min passed...41248 MiB / 49568 MiB / 41231 MiB / 18560 MiB / 18529 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
46 min passed...41248 MiB / 49792 MiB / 41242 MiB / 18560 MiB / 18529 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=4884KiB/s][w=305 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1234: Sat Aug 21 05:25:26 2021
  write: IOPS=340, BW=5446KiB/s (5576kB/s)(1024MiB/192552msec); 0 zone resets
    clat (usec): min=1119, max=36825, avg=2918.14, stdev=1083.39
     lat (usec): min=1121, max=36827, avg=2920.37, stdev=1083.42
    clat percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 1745], 10.00th=[ 1893], 20.00th=[ 2147],
     | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2835], 60.00th=[ 2999],
     | 70.00th=[ 3163], 80.00th=[ 3490], 90.00th=[ 3884], 95.00th=[ 4178],
     | 99.00th=[ 5997], 99.50th=[ 6521], 99.90th=[15533], 99.95th=[18482],
     | 99.99th=[28181]
   bw (  KiB/s): min= 4352, max= 7104, per=100.00%, avg=5445.39, stdev=499.24, samples=385
   iops        : min=  272, max=  444, avg=340.33, stdev=31.21, samples=385
  lat (msec)   : 2=14.38%, 4=78.30%, 10=7.07%, 20=0.22%, 50=0.04%
  cpu          : usr=0.93%, sys=4.75%, ctx=157776, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5446KiB/s (5576kB/s), 5446KiB/s-5446KiB/s (5576kB/s-5576kB/s), io=1024MiB (1074MB), run=192552-192552msec

Disk stats (read/write):
  sda: ios=0/196963, merge=0/135087, ticks=0/176807, in_queue=192100, util=99.80%
0 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 2066 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=23.9MiB/s][r=1529 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1437: Sat Aug 21 05:41:09 2021
  read: IOPS=1542, BW=24.1MiB/s (25.3MB/s)(1024MiB/42477msec)
    clat (usec): min=245, max=3578, avg=636.74, stdev=91.33
     lat (usec): min=247, max=3580, avg=638.75, stdev=91.41
    clat percentiles (usec):
     |  1.00th=[  457],  5.00th=[  498], 10.00th=[  523], 20.00th=[  562],
     | 30.00th=[  586], 40.00th=[  611], 50.00th=[  644], 60.00th=[  668],
     | 70.00th=[  693], 80.00th=[  717], 90.00th=[  742], 95.00th=[  758],
     | 99.00th=[  783], 99.50th=[  799], 99.90th=[  914], 99.95th=[ 1020],
     | 99.99th=[ 3261]
   bw (  KiB/s): min=21696, max=29472, per=100.00%, avg=24692.68, stdev=1777.48, samples=84
   iops        : min= 1356, max= 1842, avg=1543.27, stdev=111.08, samples=84
  lat (usec)   : 250=0.01%, 500=5.63%, 750=86.84%, 1000=7.48%
  lat (msec)   : 2=0.03%, 4=0.02%
  cpu          : usr=2.93%, sys=9.29%, ctx=65539, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=24.1MiB/s (25.3MB/s), 24.1MiB/s-24.1MiB/s (25.3MB/s-25.3MB/s), io=1024MiB (1074MB), run=42477-42477msec

Disk stats (read/write):
  sda: ios=65419/11, merge=0/10, ticks=38819/17, in_queue=42424, util=99.86%
0 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2068 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2069 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2070 MiB / 1024 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2070 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 2037 seconds or 33 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
1 min passed...3360 MiB / 3808 MiB / 3360 MiB / 416 MiB / 440 MiB ( 3360 MiB / 3808 MiB / 3358 MiB / 416 MiB / 440 MiB )
2 min passed...6656 MiB / 7904 MiB / 6665 MiB / 1024 MiB / 1090 MiB ( 3296 MiB / 4096 MiB / 3305 MiB / 608 MiB / 650 MiB )
3 min passed...9920 MiB / 11360 MiB / 9910 MiB / 1792 MiB / 1816 MiB ( 3264 MiB / 3456 MiB / 3245 MiB / 768 MiB / 726 MiB )
4 min passed...10688 MiB / 12224 MiB / 10663 MiB / 1952 MiB / 1960 MiB ( 768 MiB / 864 MiB / 753 MiB / 160 MiB / 144 MiB )
5 min passed...10688 MiB / 12448 MiB / 10672 MiB / 1952 MiB / 1960 MiB ( 0 MiB / 224 MiB / 9 MiB / 0 MiB / 0 MiB )
6 min passed...10688 MiB / 12864 MiB / 10683 MiB / 1952 MiB / 1960 MiB ( 0 MiB / 416 MiB / 11 MiB / 0 MiB / 0 MiB )
7 min passed...10688 MiB / 13280 MiB / 10691 MiB / 1952 MiB / 1961 MiB ( 0 MiB / 416 MiB / 8 MiB / 0 MiB / 1 MiB )
8 min passed...10688 MiB / 13568 MiB / 10701 MiB / 1952 MiB / 1961 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
9 min passed...10720 MiB / 13696 MiB / 10712 MiB / 1952 MiB / 1961 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...10720 MiB / 14016 MiB / 10721 MiB / 1952 MiB / 1961 MiB ( 0 MiB / 320 MiB / 9 MiB / 0 MiB / 0 MiB )
11 min passed...10752 MiB / 14432 MiB / 10730 MiB / 1952 MiB / 1961 MiB ( 32 MiB / 416 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...10752 MiB / 14880 MiB / 10738 MiB / 1952 MiB / 1961 MiB ( 0 MiB / 448 MiB / 8 MiB / 0 MiB / 0 MiB )
13 min passed...10752 MiB / 15136 MiB / 10748 MiB / 1952 MiB / 1961 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
14 min passed...10752 MiB / 15264 MiB / 10759 MiB / 1952 MiB / 1962 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 1 MiB )
15 min passed...10784 MiB / 15680 MiB / 10768 MiB / 1952 MiB / 1962 MiB ( 32 MiB / 416 MiB / 9 MiB / 0 MiB / 0 MiB )
16 min passed...10784 MiB / 16128 MiB / 10780 MiB / 1952 MiB / 1962 MiB ( 0 MiB / 448 MiB / 12 MiB / 0 MiB / 0 MiB )
17 min passed...10816 MiB / 16576 MiB / 10787 MiB / 1952 MiB / 1962 MiB ( 32 MiB / 448 MiB / 7 MiB / 0 MiB / 0 MiB )
18 min passed...10816 MiB / 16608 MiB / 10796 MiB / 1952 MiB / 1962 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...10816 MiB / 16608 MiB / 10804 MiB / 3968 MiB / 4010 MiB ( 0 MiB / 0 MiB / 8 MiB / 2016 MiB / 2048 MiB )
20 min passed...10848 MiB / 16992 MiB / 10814 MiB / 3968 MiB / 4010 MiB ( 32 MiB / 384 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...10848 MiB / 17344 MiB / 10822 MiB / 3968 MiB / 4010 MiB ( 0 MiB / 352 MiB / 8 MiB / 0 MiB / 0 MiB )
22 min passed...10848 MiB / 17344 MiB / 10829 MiB / 3968 MiB / 4011 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 1 MiB )
23 min passed...10848 MiB / 17504 MiB / 10838 MiB / 3968 MiB / 4011 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
24 min passed...10848 MiB / 17504 MiB / 10847 MiB / 3968 MiB / 4011 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
25 min passed...10848 MiB / 17696 MiB / 10856 MiB / 3968 MiB / 4011 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...10880 MiB / 17888 MiB / 10866 MiB / 3968 MiB / 4011 MiB ( 32 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...10880 MiB / 18048 MiB / 10874 MiB / 3968 MiB / 4011 MiB ( 0 MiB / 160 MiB / 8 MiB / 0 MiB / 0 MiB )
28 min passed...10912 MiB / 18336 MiB / 10883 MiB / 3968 MiB / 4011 MiB ( 32 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
29 min passed...10944 MiB / 18560 MiB / 10891 MiB / 3968 MiB / 4011 MiB ( 32 MiB / 224 MiB / 8 MiB / 0 MiB / 0 MiB )
30 min passed...10944 MiB / 18944 MiB / 10901 MiB / 3968 MiB / 4012 MiB ( 0 MiB / 384 MiB / 10 MiB / 0 MiB / 1 MiB )
31 min passed...10944 MiB / 19168 MiB / 10907 MiB / 3968 MiB / 4012 MiB ( 0 MiB / 224 MiB / 6 MiB / 0 MiB / 0 MiB )
32 min passed...10944 MiB / 19520 MiB / 10917 MiB / 3968 MiB / 4012 MiB ( 0 MiB / 352 MiB / 10 MiB / 0 MiB / 0 MiB )
33 min passed...10944 MiB / 19872 MiB / 10926 MiB / 3968 MiB / 4012 MiB ( 0 MiB / 352 MiB / 9 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=204MiB/s][w=51 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1670: Sat Aug 21 06:04:27 2021
  write: IOPS=61, BW=246MiB/s (258MB/s)(10.0GiB/41598msec); 0 zone resets
    clat (usec): min=8735, max=38525, avg=14538.92, stdev=5439.54
     lat (usec): min=8739, max=38528, avg=14542.61, stdev=5439.92
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[10814], 10.00th=[11076], 20.00th=[11338],
     | 30.00th=[11600], 40.00th=[11731], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[20055], 90.00th=[24511], 95.00th=[26608],
     | 99.00th=[30802], 99.50th=[33162], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[38536]
   bw (  KiB/s): min=147456, max=319488, per=99.92%, avg=251864.54, stdev=60670.41, samples=83
   iops        : min=   36, max=   78, avg=61.47, stdev=14.79, samples=83
  lat (msec)   : 10=0.70%, 20=79.10%, 50=20.20%
  cpu          : usr=10.76%, sys=2.38%, ctx=5635, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=10.0GiB (10.7GB), run=41598-41598msec

Disk stats (read/write):
  sda: ios=0/15288, merge=0/5263, ticks=0/46392, in_queue=51332, util=98.40%
0 min passed...Guest Writes/Reads (Changes): 10280 MiB / 0 MiB ( 10280 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=885MiB/s][r=221 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1925: Sat Aug 21 06:19:40 2021
  read: IOPS=214, BW=858MiB/s (900MB/s)(10.0GiB/11931msec)
    clat (usec): min=3749, max=11206, avg=4648.52, stdev=946.63
     lat (usec): min=3751, max=11208, avg=4650.44, stdev=946.66
    clat percentiles (usec):
     |  1.00th=[ 3949],  5.00th=[ 4047], 10.00th=[ 4080], 20.00th=[ 4146],
     | 30.00th=[ 4228], 40.00th=[ 4293], 50.00th=[ 4359], 60.00th=[ 4424],
     | 70.00th=[ 4555], 80.00th=[ 4817], 90.00th=[ 5604], 95.00th=[ 6718],
     | 99.00th=[ 8979], 99.50th=[ 9372], 99.90th=[10945], 99.95th=[11207],
     | 99.99th=[11207]
   bw (  KiB/s): min=819200, max=925696, per=99.85%, avg=877534.30, stdev=29916.07, samples=23
   iops        : min=  200, max=  226, avg=214.22, stdev= 7.29, samples=23
  lat (msec)   : 4=2.93%, 10=96.68%, 20=0.39%
  cpu          : usr=0.49%, sys=5.95%, ctx=2562, majf=0, minf=523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=858MiB/s (900MB/s), 858MiB/s-858MiB/s (900MB/s-900MB/s), io=10.0GiB (10.7GB), run=11931-11931msec

Disk stats (read/write):
  sda: ios=10212/2, merge=0/2, ticks=39662/3, in_queue=40624, util=99.59%
0 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10282 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10283 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10241 MiB ( 1 MiB / 1 MiB )
7 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10241 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10241 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10241 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10241 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10241 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10241 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10241 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10241 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10241 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10286 MiB / 10241 MiB ( 0 MiB / 0 MiB )
Script ran 1855 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...31712 MiB / 34112 MiB / 31776 MiB / 32 MiB / 13 MiB ( 31712 MiB / 34112 MiB / 31776 MiB / 32 MiB / 13 MiB )
2 min passed...31744 MiB / 34304 MiB / 31788 MiB / 32 MiB / 13 MiB ( 32 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
3 min passed...31808 MiB / 34304 MiB / 31801 MiB / 32 MiB / 13 MiB ( 64 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
4 min passed...31840 MiB / 34560 MiB / 31812 MiB / 32 MiB / 13 MiB ( 32 MiB / 256 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...31840 MiB / 34592 MiB / 31821 MiB / 32 MiB / 13 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
6 min passed...31840 MiB / 34944 MiB / 31832 MiB / 32 MiB / 13 MiB ( 0 MiB / 352 MiB / 11 MiB / 0 MiB / 0 MiB )
7 min passed...31840 MiB / 35136 MiB / 31843 MiB / 32 MiB / 13 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
8 min passed...31840 MiB / 35168 MiB / 31855 MiB / 32 MiB / 14 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 1 MiB )
9 min passed...31872 MiB / 35488 MiB / 31867 MiB / 32 MiB / 14 MiB ( 32 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
10 min passed...31872 MiB / 35616 MiB / 31877 MiB / 32 MiB / 14 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...31904 MiB / 36288 MiB / 31887 MiB / 32 MiB / 14 MiB ( 32 MiB / 672 MiB / 10 MiB / 0 MiB / 0 MiB )
12 min passed...31904 MiB / 36480 MiB / 31901 MiB / 32 MiB / 14 MiB ( 0 MiB / 192 MiB / 14 MiB / 0 MiB / 0 MiB )
13 min passed...31904 MiB / 36512 MiB / 31913 MiB / 32 MiB / 14 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
14 min passed...31904 MiB / 36960 MiB / 31923 MiB / 32 MiB / 14 MiB ( 0 MiB / 448 MiB / 10 MiB / 0 MiB / 0 MiB )
15 min passed...31904 MiB / 37184 MiB / 31933 MiB / 32 MiB / 14 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
16 min passed...31904 MiB / 37824 MiB / 31944 MiB / 10432 MiB / 10425 MiB ( 0 MiB / 640 MiB / 11 MiB / 10400 MiB / 10411 MiB )
17 min passed...31904 MiB / 38048 MiB / 31955 MiB / 10432 MiB / 10426 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 1 MiB )
18 min passed...31904 MiB / 38176 MiB / 31964 MiB / 10432 MiB / 10426 MiB ( 0 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...31936 MiB / 38656 MiB / 31976 MiB / 10432 MiB / 10426 MiB ( 32 MiB / 480 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...31936 MiB / 38880 MiB / 31987 MiB / 10432 MiB / 10426 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...32000 MiB / 39168 MiB / 31996 MiB / 10432 MiB / 10426 MiB ( 64 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...32032 MiB / 39392 MiB / 32010 MiB / 10432 MiB / 10428 MiB ( 32 MiB / 224 MiB / 14 MiB / 0 MiB / 2 MiB )
23 min passed...32032 MiB / 39552 MiB / 32024 MiB / 10432 MiB / 10428 MiB ( 0 MiB / 160 MiB / 14 MiB / 0 MiB / 0 MiB )
24 min passed...32032 MiB / 39680 MiB / 32041 MiB / 10432 MiB / 10429 MiB ( 0 MiB / 128 MiB / 17 MiB / 0 MiB / 1 MiB )
25 min passed...32064 MiB / 39744 MiB / 32057 MiB / 10432 MiB / 10429 MiB ( 32 MiB / 64 MiB / 16 MiB / 0 MiB / 0 MiB )
26 min passed...32064 MiB / 39872 MiB / 32071 MiB / 10432 MiB / 10429 MiB ( 0 MiB / 128 MiB / 14 MiB / 0 MiB / 0 MiB )
27 min passed...32096 MiB / 39968 MiB / 32088 MiB / 10432 MiB / 10429 MiB ( 32 MiB / 96 MiB / 17 MiB / 0 MiB / 0 MiB )
28 min passed...32096 MiB / 40064 MiB / 32104 MiB / 10432 MiB / 10430 MiB ( 0 MiB / 96 MiB / 16 MiB / 0 MiB / 1 MiB )
29 min passed...32096 MiB / 40128 MiB / 32119 MiB / 10432 MiB / 10430 MiB ( 0 MiB / 64 MiB / 15 MiB / 0 MiB / 0 MiB )
30 min passed...32096 MiB / 40192 MiB / 32137 MiB / 10432 MiB / 10430 MiB ( 0 MiB / 64 MiB / 18 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=106MiB/s][w=27.2k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2334: Sat Aug 21 08:34:07 2021
  write: IOPS=25.4k, BW=99.2MiB/s (104MB/s)(1024MiB/10319msec); 0 zone resets
    slat (usec): min=6, max=10385, avg=23.28, stdev=52.31
    clat (usec): min=392, max=30316, avg=2488.59, stdev=2093.67
     lat (usec): min=526, max=30328, avg=2513.26, stdev=2096.59
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1401], 10.00th=[ 1565], 20.00th=[ 1663],
     | 30.00th=[ 1713], 40.00th=[ 1762], 50.00th=[ 1827], 60.00th=[ 1975],
     | 70.00th=[ 2245], 80.00th=[ 2442], 90.00th=[ 3425], 95.00th=[ 6718],
     | 99.00th=[12649], 99.50th=[14222], 99.90th=[17433], 99.95th=[18744],
     | 99.99th=[23725]
   bw (  KiB/s): min=32400, max=139736, per=99.34%, avg=100944.20, stdev=34344.23, samples=20
   iops        : min= 8100, max=34934, avg=25236.05, stdev=8586.06, samples=20
  lat (usec)   : 500=0.01%, 750=0.16%, 1000=0.83%
  lat (msec)   : 2=59.99%, 4=30.47%, 10=5.91%, 20=2.61%, 50=0.03%
  cpu          : usr=16.44%, sys=47.91%, ctx=198251, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=1024MiB (1074MB), run=10319-10319msec

Disk stats (read/write):
  sda: ios=0/256868, merge=0/1294, ticks=0/232369, in_queue=235148, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 0 MiB ( 1029 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1036 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1037 MiB / 0 MiB ( 1 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=133MiB/s][r=34.2k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2614: Sat Aug 21 08:49:16 2021
  read: IOPS=34.7k, BW=136MiB/s (142MB/s)(1024MiB/7553msec)
    slat (usec): min=4, max=640, avg=14.90, stdev=19.53
    clat (usec): min=395, max=7918, avg=1820.59, stdev=488.44
     lat (usec): min=407, max=7927, avg=1837.39, stdev=491.40
    clat percentiles (usec):
     |  1.00th=[  922],  5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1352],
     | 30.00th=[ 1582], 40.00th=[ 1811], 50.00th=[ 1909], 60.00th=[ 1975],
     | 70.00th=[ 2040], 80.00th=[ 2114], 90.00th=[ 2212], 95.00th=[ 2474],
     | 99.00th=[ 3392], 99.50th=[ 4113], 99.90th=[ 5276], 99.95th=[ 5669],
     | 99.99th=[ 6521]
   bw (  KiB/s): min=123616, max=160624, per=99.91%, avg=138705.07, stdev=10468.58, samples=15
   iops        : min=30904, max=40156, avg=34676.40, stdev=2617.09, samples=15
  lat (usec)   : 500=0.01%, 750=0.14%, 1000=2.15%
  lat (msec)   : 2=61.90%, 4=35.24%, 10=0.56%
  cpu          : usr=36.67%, sys=58.13%, ctx=9145, majf=0, minf=71
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=1024MiB (1074MB), run=7553-7553msec

Disk stats (read/write):
  sda: ios=256805/2, merge=0/0, ticks=200630/4, in_queue=200748, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1039 MiB / 1024 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1039 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1039 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1039 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1039 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1040 MiB / 1024 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1040 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1820 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
1 min passed...4832 MiB / 5600 MiB / 4855 MiB / 2816 MiB / 2802 MiB ( 4832 MiB / 5600 MiB / 4853 MiB / 2816 MiB / 2802 MiB )
2 min passed...4896 MiB / 5760 MiB / 4872 MiB / 2816 MiB / 2802 MiB ( 64 MiB / 160 MiB / 17 MiB / 0 MiB / 0 MiB )
3 min passed...4896 MiB / 6080 MiB / 4889 MiB / 2816 MiB / 2802 MiB ( 0 MiB / 320 MiB / 17 MiB / 0 MiB / 0 MiB )
4 min passed...4928 MiB / 6720 MiB / 4906 MiB / 2816 MiB / 2803 MiB ( 32 MiB / 640 MiB / 17 MiB / 0 MiB / 1 MiB )
5 min passed...4928 MiB / 7232 MiB / 4924 MiB / 2816 MiB / 2803 MiB ( 0 MiB / 512 MiB / 18 MiB / 0 MiB / 0 MiB )
6 min passed...4928 MiB / 7680 MiB / 4943 MiB / 2816 MiB / 2804 MiB ( 0 MiB / 448 MiB / 19 MiB / 0 MiB / 1 MiB )
7 min passed...4960 MiB / 7840 MiB / 4957 MiB / 2816 MiB / 2804 MiB ( 32 MiB / 160 MiB / 14 MiB / 0 MiB / 0 MiB )
8 min passed...4992 MiB / 8064 MiB / 4973 MiB / 2816 MiB / 2804 MiB ( 32 MiB / 224 MiB / 16 MiB / 0 MiB / 0 MiB )
9 min passed...4992 MiB / 8768 MiB / 4988 MiB / 2816 MiB / 2805 MiB ( 0 MiB / 704 MiB / 15 MiB / 0 MiB / 1 MiB )
10 min passed...4992 MiB / 9248 MiB / 5005 MiB / 2816 MiB / 2805 MiB ( 0 MiB / 480 MiB / 17 MiB / 0 MiB / 0 MiB )
11 min passed...4992 MiB / 9312 MiB / 5024 MiB / 2816 MiB / 2805 MiB ( 0 MiB / 64 MiB / 19 MiB / 0 MiB / 0 MiB )
12 min passed...5056 MiB / 9376 MiB / 5040 MiB / 2816 MiB / 2806 MiB ( 64 MiB / 64 MiB / 16 MiB / 0 MiB / 1 MiB )
13 min passed...5056 MiB / 9728 MiB / 5057 MiB / 2816 MiB / 2806 MiB ( 0 MiB / 352 MiB / 17 MiB / 0 MiB / 0 MiB )
14 min passed...5088 MiB / 10336 MiB / 5075 MiB / 2816 MiB / 2807 MiB ( 32 MiB / 608 MiB / 18 MiB / 0 MiB / 1 MiB )
15 min passed...5088 MiB / 10688 MiB / 5092 MiB / 2816 MiB / 2807 MiB ( 0 MiB / 352 MiB / 17 MiB / 0 MiB / 0 MiB )
16 min passed...5120 MiB / 10944 MiB / 5110 MiB / 11008 MiB / 10953 MiB ( 32 MiB / 256 MiB / 18 MiB / 8192 MiB / 8146 MiB )
17 min passed...5120 MiB / 11040 MiB / 5127 MiB / 11008 MiB / 10953 MiB ( 0 MiB / 96 MiB / 17 MiB / 0 MiB / 0 MiB )
18 min passed...5152 MiB / 11360 MiB / 5143 MiB / 11008 MiB / 10953 MiB ( 32 MiB / 320 MiB / 16 MiB / 0 MiB / 0 MiB )
19 min passed...5184 MiB / 11872 MiB / 5160 MiB / 11008 MiB / 10954 MiB ( 32 MiB / 512 MiB / 17 MiB / 0 MiB / 1 MiB )
20 min passed...5184 MiB / 12192 MiB / 5177 MiB / 11008 MiB / 10954 MiB ( 0 MiB / 320 MiB / 17 MiB / 0 MiB / 0 MiB )
21 min passed...5184 MiB / 12416 MiB / 5195 MiB / 11008 MiB / 10955 MiB ( 0 MiB / 224 MiB / 18 MiB / 0 MiB / 1 MiB )
22 min passed...5184 MiB / 12544 MiB / 5214 MiB / 11008 MiB / 10955 MiB ( 0 MiB / 128 MiB / 19 MiB / 0 MiB / 0 MiB )
23 min passed...5248 MiB / 12768 MiB / 5229 MiB / 11008 MiB / 10955 MiB ( 64 MiB / 224 MiB / 15 MiB / 0 MiB / 0 MiB )
24 min passed...5248 MiB / 13216 MiB / 5245 MiB / 11008 MiB / 10956 MiB ( 0 MiB / 448 MiB / 16 MiB / 0 MiB / 1 MiB )
25 min passed...5248 MiB / 13696 MiB / 5261 MiB / 11008 MiB / 10956 MiB ( 0 MiB / 480 MiB / 16 MiB / 0 MiB / 0 MiB )
26 min passed...5280 MiB / 13856 MiB / 5277 MiB / 11008 MiB / 10957 MiB ( 32 MiB / 160 MiB / 16 MiB / 0 MiB / 1 MiB )
27 min passed...5280 MiB / 14048 MiB / 5295 MiB / 11008 MiB / 10957 MiB ( 0 MiB / 192 MiB / 18 MiB / 0 MiB / 0 MiB )
28 min passed...5312 MiB / 14400 MiB / 5310 MiB / 11008 MiB / 10957 MiB ( 32 MiB / 352 MiB / 15 MiB / 0 MiB / 0 MiB )
29 min passed...5376 MiB / 15008 MiB / 5326 MiB / 11008 MiB / 10958 MiB ( 64 MiB / 608 MiB / 16 MiB / 0 MiB / 1 MiB )
30 min passed...5376 MiB / 15488 MiB / 5343 MiB / 11008 MiB / 10958 MiB ( 0 MiB / 480 MiB / 17 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=3135: Sat Aug 21 18:02:18 2021
  write: IOPS=25.2k, BW=394MiB/s (414MB/s)(1024MiB/2596msec); 0 zone resets
    slat (usec): min=7, max=2344, avg=21.55, stdev=23.78
    clat (usec): min=45, max=17073, avg=2499.46, stdev=1604.23
     lat (usec): min=65, max=17088, avg=2522.39, stdev=1606.18
    clat percentiles (usec):
     |  1.00th=[ 1139],  5.00th=[ 1663], 10.00th=[ 1844], 20.00th=[ 1926],
     | 30.00th=[ 1958], 40.00th=[ 2008], 50.00th=[ 2057], 60.00th=[ 2147],
     | 70.00th=[ 2311], 80.00th=[ 2507], 90.00th=[ 2868], 95.00th=[ 5866],
     | 99.00th=[10552], 99.50th=[11338], 99.90th=[13042], 99.95th=[13698],
     | 99.99th=[15270]
   bw (  KiB/s): min=283936, max=472928, per=99.23%, avg=400819.20, stdev=87113.74, samples=5
   iops        : min=17746, max=29558, avg=25051.20, stdev=5444.61, samples=5
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.04%, 750=0.03%
  lat (usec)   : 1000=0.39%
  lat (msec)   : 2=38.55%, 4=54.95%, 10=4.68%, 20=1.34%
  cpu          : usr=22.81%, sys=49.71%, ctx=49273, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=394MiB/s (414MB/s), 394MiB/s-394MiB/s (414MB/s-414MB/s), io=1024MiB (1074MB), run=2596-2596msec

Disk stats (read/write):
  sda: ios=0/59521, merge=0/0, ticks=0/43314, in_queue=45508, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1028 MiB / 0 MiB ( 1 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=3375: Sat Aug 21 18:17:20 2021
  read: IOPS=34.2k, BW=535MiB/s (561MB/s)(1024MiB/1915msec)
    slat (usec): min=4, max=488, avg=15.79, stdev=19.15
    clat (usec): min=389, max=5169, avg=1845.12, stdev=494.58
     lat (usec): min=421, max=5187, avg=1862.85, stdev=498.89
    clat percentiles (usec):
     |  1.00th=[  914],  5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 1319],
     | 30.00th=[ 1516], 40.00th=[ 1827], 50.00th=[ 1975], 60.00th=[ 2057],
     | 70.00th=[ 2114], 80.00th=[ 2180], 90.00th=[ 2343], 95.00th=[ 2671],
     | 99.00th=[ 2933], 99.50th=[ 3326], 99.90th=[ 4015], 99.95th=[ 4293],
     | 99.99th=[ 4817]
   bw (  KiB/s): min=462624, max=651584, per=100.00%, avg=557482.67, stdev=94482.28, samples=3
   iops        : min=28914, max=40724, avg=34842.67, stdev=5905.14, samples=3
  lat (usec)   : 500=0.01%, 750=0.10%, 1000=2.53%
  lat (msec)   : 2=50.68%, 4=46.58%, 10=0.10%
  cpu          : usr=37.25%, sys=57.63%, ctx=1983, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=1024MiB (1074MB), run=1915-1915msec

Disk stats (read/write):
  sda: ios=64942/0, merge=0/0, ticks=51921/0, in_queue=52616, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1806 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2656 MiB / 2880 MiB / 2614 MiB / 512 MiB / 472 MiB ( 2656 MiB / 2880 MiB / 2614 MiB / 512 MiB / 472 MiB )
2 min passed...2656 MiB / 3360 MiB / 2624 MiB / 512 MiB / 472 MiB ( 0 MiB / 480 MiB / 10 MiB / 0 MiB / 0 MiB )
3 min passed...2656 MiB / 3936 MiB / 2635 MiB / 512 MiB / 472 MiB ( 0 MiB / 576 MiB / 11 MiB / 0 MiB / 0 MiB )
4 min passed...2656 MiB / 4384 MiB / 2646 MiB / 512 MiB / 472 MiB ( 0 MiB / 448 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...2656 MiB / 4544 MiB / 2656 MiB / 512 MiB / 472 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...2656 MiB / 4800 MiB / 2667 MiB / 512 MiB / 473 MiB ( 0 MiB / 256 MiB / 11 MiB / 0 MiB / 1 MiB )
7 min passed...2656 MiB / 5184 MiB / 2676 MiB / 512 MiB / 473 MiB ( 0 MiB / 384 MiB / 9 MiB / 0 MiB / 0 MiB )
8 min passed...2656 MiB / 5440 MiB / 2685 MiB / 512 MiB / 473 MiB ( 0 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
9 min passed...2656 MiB / 5888 MiB / 2694 MiB / 512 MiB / 473 MiB ( 0 MiB / 448 MiB / 9 MiB / 0 MiB / 0 MiB )
10 min passed...2688 MiB / 6016 MiB / 2704 MiB / 512 MiB / 473 MiB ( 32 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...2720 MiB / 6368 MiB / 2714 MiB / 512 MiB / 473 MiB ( 32 MiB / 352 MiB / 10 MiB / 0 MiB / 0 MiB )
12 min passed...2752 MiB / 6528 MiB / 2722 MiB / 512 MiB / 473 MiB ( 32 MiB / 160 MiB / 8 MiB / 0 MiB / 0 MiB )
13 min passed...2752 MiB / 6880 MiB / 2734 MiB / 512 MiB / 473 MiB ( 0 MiB / 352 MiB / 12 MiB / 0 MiB / 0 MiB )
14 min passed...2784 MiB / 7360 MiB / 2744 MiB / 512 MiB / 474 MiB ( 32 MiB / 480 MiB / 10 MiB / 0 MiB / 1 MiB )
15 min passed...2784 MiB / 7520 MiB / 2755 MiB / 2624 MiB / 2514 MiB ( 0 MiB / 160 MiB / 11 MiB / 2112 MiB / 2040 MiB )
16 min passed...2784 MiB / 7744 MiB / 2765 MiB / 2624 MiB / 2514 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
17 min passed...2784 MiB / 8096 MiB / 2775 MiB / 2624 MiB / 2515 MiB ( 0 MiB / 352 MiB / 10 MiB / 0 MiB / 1 MiB )
18 min passed...2816 MiB / 8320 MiB / 2786 MiB / 2624 MiB / 2515 MiB ( 32 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
19 min passed...2816 MiB / 8736 MiB / 2793 MiB / 2624 MiB / 2515 MiB ( 0 MiB / 416 MiB / 7 MiB / 0 MiB / 0 MiB )
20 min passed...2848 MiB / 9088 MiB / 2804 MiB / 2624 MiB / 2516 MiB ( 32 MiB / 352 MiB / 11 MiB / 0 MiB / 1 MiB )
21 min passed...2848 MiB / 9216 MiB / 2814 MiB / 2624 MiB / 2516 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...2848 MiB / 9472 MiB / 2823 MiB / 2624 MiB / 2516 MiB ( 0 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
23 min passed...2848 MiB / 9568 MiB / 2832 MiB / 2624 MiB / 2517 MiB ( 0 MiB / 96 MiB / 9 MiB / 0 MiB / 1 MiB )
24 min passed...2848 MiB / 9920 MiB / 2841 MiB / 2624 MiB / 2517 MiB ( 0 MiB / 352 MiB / 9 MiB / 0 MiB / 0 MiB )
25 min passed...2848 MiB / 10304 MiB / 2850 MiB / 2624 MiB / 2517 MiB ( 0 MiB / 384 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...2880 MiB / 10368 MiB / 2861 MiB / 2624 MiB / 2517 MiB ( 32 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
27 min passed...2880 MiB / 10656 MiB / 2871 MiB / 2624 MiB / 2518 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 1 MiB )
28 min passed...2880 MiB / 10688 MiB / 2879 MiB / 2624 MiB / 2518 MiB ( 0 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
29 min passed...2912 MiB / 10944 MiB / 2891 MiB / 2624 MiB / 2518 MiB ( 32 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
30 min passed...2944 MiB / 11168 MiB / 2901 MiB / 2624 MiB / 2519 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 6: async 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=773MiB/s][w=193 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3650: Sat Aug 21 20:00:46 2021
  write: IOPS=258, BW=1035MiB/s (1085MB/s)(10.0GiB/9893msec); 0 zone resets
    slat (usec): min=93, max=1409.3k, avg=1289.48, stdev=38378.21
    clat (usec): min=1088, max=2830.9k, avg=243963.97, stdev=451330.66
     lat (usec): min=1289, max=2831.0k, avg=245256.28, stdev=452618.73
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   65], 10.00th=[   71], 20.00th=[   78],
     | 30.00th=[   82], 40.00th=[   85], 50.00th=[   87], 60.00th=[   89],
     | 70.00th=[   93], 80.00th=[  117], 90.00th=[  380], 95.00th=[ 1485],
     | 99.00th=[ 1569], 99.50th=[ 2735], 99.90th=[ 2836], 99.95th=[ 2836],
     | 99.99th=[ 2836]
   bw (  MiB/s): min=   63, max= 2728, per=100.00%, avg=1372.45, stdev=1068.51, samples=13
   iops        : min=   15, max=  682, avg=343.00, stdev=267.14, samples=13
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.16%, 20=0.31%, 50=0.90%
  lat (msec)   : 100=76.13%, 250=8.44%, 500=4.02%
  cpu          : usr=28.71%, sys=5.57%, ctx=2467, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64


Run status group 0 (all jobs):
  WRITE: bw=1035MiB/s (1085MB/s), 1035MiB/s-1035MiB/s (1085MB/s-1085MB/s), io=10.0GiB (10.7GB), run=9893-9893msec


Disk stats (read/write):
  sda: ios=0/9902, merge=0/94, ticks=0/1424383, in_queue=486480, util=50.35%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 10241 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10243 MiB / 0 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10243 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10243 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1461MiB/s][r=365 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3857: Sat Aug 21 20:15:54 2021
  read: IOPS=367, BW=1471MiB/s (1542MB/s)(10.0GiB/6962msec)
    slat (usec): min=84, max=2551, avg=184.12, stdev=139.18
    clat (msec): min=34, max=304, avg=173.10, stdev=59.71
     lat (msec): min=34, max=304, avg=173.29, stdev=59.68
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   63], 10.00th=[   77], 20.00th=[  111],
     | 30.00th=[  150], 40.00th=[  174], 50.00th=[  190], 60.00th=[  201],
     | 70.00th=[  209], 80.00th=[  226], 90.00th=[  236], 95.00th=[  249],
     | 99.00th=[  279], 99.50th=[  296], 99.90th=[  300], 99.95th=[  300],
     | 99.99th=[  305]
   bw (  MiB/s): min= 1200, max= 1616, per=99.69%, avg=1466.23, stdev=129.37, samples=13
   iops        : min=  300, max=  404, avg=366.54, stdev=32.33, samples=13
  lat (msec)   : 50=1.99%, 100=15.74%, 250=77.54%, 500=4.73%
  cpu          : usr=0.45%, sys=7.05%, ctx=2473, majf=0, minf=649
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64


Run status group 0 (all jobs):
   READ: bw=1471MiB/s (1542MB/s), 1471MiB/s-1471MiB/s (1542MB/s-1542MB/s), io=10.0GiB (10.7GB), run=6962-6962msec


Disk stats (read/write):
  sda: ios=10134/0, merge=0/0, ticks=1295367/0, in_queue=1309628, util=99.99%
0 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1818 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...15744 MiB / 16704 MiB / 15686 MiB / 192 MiB / 205 MiB ( 15744 MiB / 16704 MiB / 15686 MiB / 192 MiB / 205 MiB )
2 min passed...15744 MiB / 16960 MiB / 15699 MiB / 192 MiB / 205 MiB ( 0 MiB / 256 MiB / 13 MiB / 0 MiB / 0 MiB )
3 min passed...15744 MiB / 17120 MiB / 15712 MiB / 192 MiB / 206 MiB ( 0 MiB / 160 MiB / 13 MiB / 0 MiB / 1 MiB )
4 min passed...15744 MiB / 17280 MiB / 15724 MiB / 192 MiB / 206 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...15744 MiB / 17504 MiB / 15738 MiB / 192 MiB / 206 MiB ( 0 MiB / 224 MiB / 14 MiB / 0 MiB / 0 MiB )
6 min passed...15744 MiB / 17504 MiB / 15751 MiB / 192 MiB / 207 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 1 MiB )
7 min passed...15744 MiB / 17632 MiB / 15762 MiB / 192 MiB / 207 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
8 min passed...15744 MiB / 17824 MiB / 15774 MiB / 192 MiB / 207 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
9 min passed...15808 MiB / 17888 MiB / 15784 MiB / 192 MiB / 208 MiB ( 64 MiB / 64 MiB / 10 MiB / 0 MiB / 1 MiB )
10 min passed...15808 MiB / 18112 MiB / 15797 MiB / 192 MiB / 208 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 0 MiB )
11 min passed...15840 MiB / 18240 MiB / 15806 MiB / 192 MiB / 208 MiB ( 32 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...15840 MiB / 18368 MiB / 15817 MiB / 192 MiB / 208 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
13 min passed...15872 MiB / 18688 MiB / 15828 MiB / 192 MiB / 209 MiB ( 32 MiB / 320 MiB / 11 MiB / 0 MiB / 1 MiB )
14 min passed...15904 MiB / 18976 MiB / 15838 MiB / 192 MiB / 209 MiB ( 32 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
15 min passed...15936 MiB / 19168 MiB / 15849 MiB / 192 MiB / 209 MiB ( 32 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...15936 MiB / 19168 MiB / 15860 MiB / 10496 MiB / 10563 MiB ( 0 MiB / 0 MiB / 11 MiB / 10304 MiB / 10354 MiB )
17 min passed...15936 MiB / 19424 MiB / 15874 MiB / 10496 MiB / 10564 MiB ( 0 MiB / 256 MiB / 14 MiB / 0 MiB / 1 MiB )
18 min passed...15936 MiB / 19616 MiB / 15884 MiB / 10496 MiB / 10564 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...15936 MiB / 19776 MiB / 15893 MiB / 10496 MiB / 10564 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
20 min passed...15936 MiB / 20032 MiB / 15905 MiB / 10496 MiB / 10565 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 1 MiB )
21 min passed...15936 MiB / 20064 MiB / 15914 MiB / 10496 MiB / 10565 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...15936 MiB / 20160 MiB / 15927 MiB / 10496 MiB / 10565 MiB ( 0 MiB / 96 MiB / 13 MiB / 0 MiB / 0 MiB )
23 min passed...15936 MiB / 20320 MiB / 15936 MiB / 10496 MiB / 10565 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
24 min passed...15968 MiB / 20608 MiB / 15949 MiB / 10496 MiB / 10566 MiB ( 32 MiB / 288 MiB / 13 MiB / 0 MiB / 1 MiB )
25 min passed...15968 MiB / 20896 MiB / 15963 MiB / 10496 MiB / 10566 MiB ( 0 MiB / 288 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...15968 MiB / 21120 MiB / 15976 MiB / 10496 MiB / 10566 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 0 MiB )
27 min passed...15968 MiB / 21248 MiB / 15989 MiB / 10496 MiB / 10566 MiB ( 0 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
28 min passed...16000 MiB / 21504 MiB / 16000 MiB / 10496 MiB / 10567 MiB ( 32 MiB / 256 MiB / 11 MiB / 0 MiB / 1 MiB )
29 min passed...16032 MiB / 21792 MiB / 16011 MiB / 10496 MiB / 10567 MiB ( 32 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...16064 MiB / 21984 MiB / 16020 MiB / 10496 MiB / 10567 MiB ( 32 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][-.-%][w=348MiB/s][w=22.3k IOPS][eta 00m:00s]
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=4093: Sat Aug 21 20:38:08 2021
  write: IOPS=25.5k, BW=398MiB/s (418MB/s)(1024MiB/2570msec); 0 zone resets
    slat (usec): min=6, max=5224, avg=22.87, stdev=44.31
    clat (usec): min=60, max=26118, avg=2474.05, stdev=1589.22
     lat (usec): min=81, max=26141, avg=2498.30, stdev=1591.18
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1663], 10.00th=[ 1795], 20.00th=[ 1942],
     | 30.00th=[ 2008], 40.00th=[ 2057], 50.00th=[ 2114], 60.00th=[ 2180],
     | 70.00th=[ 2245], 80.00th=[ 2376], 90.00th=[ 2802], 95.00th=[ 4490],
     | 99.00th=[10945], 99.50th=[12256], 99.90th=[15664], 99.95th=[16909],
     | 99.99th=[20579]
   bw (  KiB/s): min=208128, max=469792, per=99.32%, avg=405248.00, stdev=111116.15, samples=5
   iops        : min=13008, max=29362, avg=25328.00, stdev=6944.76, samples=5
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.27%
  lat (msec)   : 2=27.28%, 4=66.88%, 10=3.99%, 20=1.49%, 50=0.02%
  cpu          : usr=21.14%, sys=49.82%, ctx=47588, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=398MiB/s (418MB/s), 398MiB/s-398MiB/s (418MB/s-418MB/s), io=1024MiB (1074MB), run=2570-2570msec

Disk stats (read/write):
  sda: ios=0/59716, merge=0/0, ticks=0/37613, in_queue=38212, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=4328: Sat Aug 21 20:53:10 2021
  read: IOPS=48.2k, BW=753MiB/s (790MB/s)(1024MiB/1359msec)
    slat (usec): min=4, max=670, avg=10.65, stdev=12.64
    clat (usec): min=338, max=14701, avg=1309.26, stdev=612.01
     lat (usec): min=344, max=14717, avg=1321.27, stdev=613.83
    clat percentiles (usec):
     |  1.00th=[  644],  5.00th=[  766], 10.00th=[  832], 20.00th=[  914],
     | 30.00th=[  979], 40.00th=[ 1037], 50.00th=[ 1106], 60.00th=[ 1188],
     | 70.00th=[ 1336], 80.00th=[ 1680], 90.00th=[ 2057], 95.00th=[ 2474],
     | 99.00th=[ 3654], 99.50th=[ 4424], 99.90th=[ 5800], 99.95th=[ 6259],
     | 99.99th=[ 7046]
   bw (  KiB/s): min=805408, max=845440, per=100.00%, avg=825424.00, stdev=28306.90, samples=2
   iops        : min=50338, max=52840, avg=51589.00, stdev=1769.18, samples=2
  lat (usec)   : 500=0.05%, 750=4.08%, 1000=29.77%
  lat (msec)   : 2=54.90%, 4=10.49%, 10=0.71%, 20=0.01%
  cpu          : usr=37.41%, sys=54.93%, ctx=2073, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=753MiB/s (790MB/s), 753MiB/s-753MiB/s (790MB/s-790MB/s), io=1024MiB (1074MB), run=1359-1359msec

Disk stats (read/write):
  sda: ios=59211/0, merge=0/0, ticks=52238/0, in_queue=52964, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2016 MiB / 2496 MiB / 2012 MiB / 448 MiB / 429 MiB ( 2016 MiB / 2496 MiB / 2012 MiB / 448 MiB / 429 MiB )
2 min passed...2080 MiB / 2688 MiB / 2026 MiB / 448 MiB / 430 MiB ( 64 MiB / 192 MiB / 14 MiB / 0 MiB / 1 MiB )
3 min passed...2080 MiB / 2688 MiB / 2036 MiB / 448 MiB / 430 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
4 min passed...2112 MiB / 3168 MiB / 2049 MiB / 448 MiB / 430 MiB ( 32 MiB / 480 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...2112 MiB / 3424 MiB / 2059 MiB / 448 MiB / 430 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...2112 MiB / 3680 MiB / 2071 MiB / 448 MiB / 430 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
7 min passed...2112 MiB / 3808 MiB / 2083 MiB / 448 MiB / 431 MiB ( 0 MiB / 128 MiB / 12 MiB / 0 MiB / 1 MiB )
8 min passed...2112 MiB / 3968 MiB / 2096 MiB / 448 MiB / 431 MiB ( 0 MiB / 160 MiB / 13 MiB / 0 MiB / 0 MiB )
9 min passed...2112 MiB / 4160 MiB / 2110 MiB / 448 MiB / 431 MiB ( 0 MiB / 192 MiB / 14 MiB / 0 MiB / 0 MiB )
10 min passed...2112 MiB / 4224 MiB / 2120 MiB / 448 MiB / 431 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...2112 MiB / 4384 MiB / 2132 MiB / 448 MiB / 432 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 1 MiB )
12 min passed...2112 MiB / 4800 MiB / 2144 MiB / 448 MiB / 432 MiB ( 0 MiB / 416 MiB / 12 MiB / 0 MiB / 0 MiB )
13 min passed...2144 MiB / 4960 MiB / 2153 MiB / 448 MiB / 432 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...2144 MiB / 4960 MiB / 2165 MiB / 448 MiB / 432 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...2144 MiB / 5472 MiB / 2175 MiB / 672 MiB / 722 MiB ( 0 MiB / 512 MiB / 10 MiB / 224 MiB / 290 MiB )
16 min passed...2144 MiB / 5536 MiB / 2186 MiB / 1792 MiB / 1708 MiB ( 0 MiB / 64 MiB / 11 MiB / 1120 MiB / 986 MiB )
17 min passed...2176 MiB / 5728 MiB / 2197 MiB / 1792 MiB / 1708 MiB ( 32 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
18 min passed...2240 MiB / 5856 MiB / 2211 MiB / 1792 MiB / 1709 MiB ( 64 MiB / 128 MiB / 14 MiB / 0 MiB / 1 MiB )
19 min passed...2240 MiB / 5920 MiB / 2226 MiB / 1792 MiB / 1709 MiB ( 0 MiB / 64 MiB / 15 MiB / 0 MiB / 0 MiB )
20 min passed...2304 MiB / 6080 MiB / 2239 MiB / 1792 MiB / 1709 MiB ( 64 MiB / 160 MiB / 13 MiB / 0 MiB / 0 MiB )
21 min passed...2304 MiB / 6208 MiB / 2251 MiB / 1792 MiB / 1709 MiB ( 0 MiB / 128 MiB / 12 MiB / 0 MiB / 0 MiB )
22 min passed...2304 MiB / 6400 MiB / 2262 MiB / 1792 MiB / 1709 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...2304 MiB / 6592 MiB / 2274 MiB / 1792 MiB / 1710 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 1 MiB )
24 min passed...2304 MiB / 6592 MiB / 2288 MiB / 1792 MiB / 1710 MiB ( 0 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
25 min passed...2304 MiB / 6656 MiB / 2298 MiB / 1792 MiB / 1710 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
26 min passed...2304 MiB / 6912 MiB / 2309 MiB / 1792 MiB / 1710 MiB ( 0 MiB / 256 MiB / 11 MiB / 0 MiB / 0 MiB )
27 min passed...2304 MiB / 7104 MiB / 2320 MiB / 1792 MiB / 1710 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
28 min passed...2304 MiB / 7200 MiB / 2331 MiB / 1792 MiB / 1711 MiB ( 0 MiB / 96 MiB / 11 MiB / 0 MiB / 1 MiB )
29 min passed...2304 MiB / 7232 MiB / 2341 MiB / 1792 MiB / 1711 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...2304 MiB / 7232 MiB / 2353 MiB / 1792 MiB / 1711 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 21, 2021

Round 2 - Run 8 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=11.8MiB/s][w=376 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4568: Sat Aug 21 21:24:58 2021
  write: IOPS=375, BW=11.7MiB/s (12.3MB/s)(1024MiB/87290msec); 0 zone resets
    clat (usec): min=1189, max=27105, avg=2636.84, stdev=853.80
     lat (usec): min=1191, max=27108, avg=2639.12, stdev=853.81
    clat percentiles (usec):
     |  1.00th=[ 1663],  5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2114],
     | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2704],
     | 70.00th=[ 2900], 80.00th=[ 3064], 90.00th=[ 3228], 95.00th=[ 3392],
     | 99.00th=[ 5211], 99.50th=[ 5866], 99.90th=[13173], 99.95th=[15795],
     | 99.99th=[23200]
   bw (  KiB/s): min= 9984, max=16064, per=100.00%, avg=12019.28, stdev=805.30, samples=174
   iops        : min=  312, max=  502, avg=375.60, stdev=25.16, samples=174
  lat (msec)   : 2=11.10%, 4=86.22%, 10=2.47%, 20=0.18%, 50=0.03%
  cpu          : usr=1.59%, sys=5.45%, ctx=66524, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=1024MiB (1074MB), run=87290-87290msec

Disk stats (read/write):
  sda: ios=0/98272, merge=0/65950, ticks=0/77548, in_queue=87040, util=99.81%
0 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 1539 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1539 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=46.0MiB/s][r=1503 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4771: Sat Aug 21 21:40:21 2021
  read: IOPS=1496, BW=46.8MiB/s (49.0MB/s)(1024MiB/21902msec)
    clat (usec): min=382, max=5544, avg=656.94, stdev=177.82
     lat (usec): min=383, max=5546, avg=658.94, stdev=177.86
    clat percentiles (usec):
     |  1.00th=[  461],  5.00th=[  502], 10.00th=[  529], 20.00th=[  562],
     | 30.00th=[  594], 40.00th=[  627], 50.00th=[  652], 60.00th=[  676],
     | 70.00th=[  701], 80.00th=[  725], 90.00th=[  758], 95.00th=[  775],
     | 99.00th=[  881], 99.50th=[ 1631], 99.90th=[ 3228], 99.95th=[ 3392],
     | 99.99th=[ 4359]
   bw (  KiB/s): min=41728, max=55552, per=100.00%, avg=47887.74, stdev=4099.35, samples=43
   iops        : min= 1304, max= 1736, avg=1496.49, stdev=128.11, samples=43
  lat (usec)   : 500=4.78%, 750=83.23%, 1000=11.17%
  lat (msec)   : 2=0.40%, 4=0.41%, 10=0.02%
  cpu          : usr=2.89%, sys=9.34%, ctx=32783, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=46.8MiB/s (49.0MB/s), 46.8MiB/s-46.8MiB/s (49.0MB/s-49.0MB/s), io=1024MiB (1074MB), run=21902-21902msec

Disk stats (read/write):
  sda: ios=32761/4, merge=0/4, ticks=20033/6, in_queue=21944, util=99.82%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1024 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1543 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1911 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5376 MiB / 5696 MiB / 5406 MiB / 128 MiB / 152 MiB ( 5376 MiB / 5696 MiB / 5406 MiB / 128 MiB / 152 MiB )
2 min passed...8544 MiB / 9216 MiB / 8546 MiB / 192 MiB / 234 MiB ( 3168 MiB / 3520 MiB / 3140 MiB / 64 MiB / 82 MiB )
3 min passed...8576 MiB / 9504 MiB / 8558 MiB / 192 MiB / 234 MiB ( 32 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
4 min passed...8608 MiB / 9824 MiB / 8569 MiB / 192 MiB / 234 MiB ( 32 MiB / 320 MiB / 11 MiB / 0 MiB / 0 MiB )
5 min passed...8608 MiB / 9856 MiB / 8580 MiB / 192 MiB / 234 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
6 min passed...8608 MiB / 9856 MiB / 8591 MiB / 192 MiB / 235 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 1 MiB )
7 min passed...8608 MiB / 10048 MiB / 8600 MiB / 192 MiB / 235 MiB ( 0 MiB / 192 MiB / 9 MiB / 0 MiB / 0 MiB )
8 min passed...8608 MiB / 10240 MiB / 8610 MiB / 192 MiB / 235 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
9 min passed...8608 MiB / 10560 MiB / 8620 MiB / 192 MiB / 235 MiB ( 0 MiB / 320 MiB / 10 MiB / 0 MiB / 0 MiB )
10 min passed...8608 MiB / 10816 MiB / 8629 MiB / 192 MiB / 235 MiB ( 0 MiB / 256 MiB / 9 MiB / 0 MiB / 0 MiB )
11 min passed...8608 MiB / 10816 MiB / 8638 MiB / 192 MiB / 235 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...8640 MiB / 10816 MiB / 8648 MiB / 192 MiB / 236 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 1 MiB )
13 min passed...8640 MiB / 11072 MiB / 8658 MiB / 192 MiB / 236 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
14 min passed...8640 MiB / 11328 MiB / 8670 MiB / 192 MiB / 236 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...8640 MiB / 11456 MiB / 8677 MiB / 192 MiB / 237 MiB ( 0 MiB / 128 MiB / 7 MiB / 0 MiB / 1 MiB )
16 min passed...8640 MiB / 11456 MiB / 8687 MiB / 192 MiB / 237 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
17 min passed...8640 MiB / 11648 MiB / 8696 MiB / 1216 MiB / 1261 MiB ( 0 MiB / 192 MiB / 9 MiB / 1024 MiB / 1024 MiB )
18 min passed...8672 MiB / 11936 MiB / 8705 MiB / 1216 MiB / 1262 MiB ( 32 MiB / 288 MiB / 9 MiB / 0 MiB / 1 MiB )
19 min passed...8736 MiB / 12256 MiB / 8715 MiB / 1216 MiB / 1262 MiB ( 64 MiB / 320 MiB / 10 MiB / 0 MiB / 0 MiB )
20 min passed...8768 MiB / 12448 MiB / 8723 MiB / 1216 MiB / 1262 MiB ( 32 MiB / 192 MiB / 8 MiB / 0 MiB / 0 MiB )
21 min passed...8800 MiB / 12480 MiB / 8734 MiB / 1216 MiB / 1263 MiB ( 32 MiB / 32 MiB / 11 MiB / 0 MiB / 1 MiB )
22 min passed...8800 MiB / 12512 MiB / 8744 MiB / 1216 MiB / 1263 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
23 min passed...8800 MiB / 12640 MiB / 8754 MiB / 1216 MiB / 1263 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...8800 MiB / 12928 MiB / 8764 MiB / 1216 MiB / 1264 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 1 MiB )
25 min passed...8800 MiB / 13312 MiB / 8774 MiB / 1216 MiB / 1264 MiB ( 0 MiB / 384 MiB / 10 MiB / 0 MiB / 0 MiB )
26 min passed...8800 MiB / 13312 MiB / 8784 MiB / 1216 MiB / 1264 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...8800 MiB / 13312 MiB / 8793 MiB / 1216 MiB / 1265 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 1 MiB )
28 min passed...8800 MiB / 13440 MiB / 8806 MiB / 1216 MiB / 1265 MiB ( 0 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
29 min passed...8800 MiB / 13728 MiB / 8816 MiB / 1216 MiB / 1265 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...8800 MiB / 14016 MiB / 8828 MiB / 1216 MiB / 1265 MiB ( 0 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
31 min passed...8800 MiB / 14144 MiB / 8839 MiB / 1216 MiB / 1266 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 22, 2021

Round 2 - Run 8 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=5003: Sat Aug 21 21:56:04 2021
  write: IOPS=23.4k, BW=730MiB/s (765MB/s)(1024MiB/1403msec); 0 zone resets
    slat (usec): min=8, max=1938, avg=22.15, stdev=18.41
    clat (usec): min=85, max=11034, avg=2695.14, stdev=516.84
     lat (usec): min=111, max=11092, avg=2718.66, stdev=520.39
    clat percentiles (usec):
     |  1.00th=[ 2057],  5.00th=[ 2343], 10.00th=[ 2376], 20.00th=[ 2442],
     | 30.00th=[ 2474], 40.00th=[ 2507], 50.00th=[ 2540], 60.00th=[ 2573],
     | 70.00th=[ 2638], 80.00th=[ 2802], 90.00th=[ 3294], 95.00th=[ 3687],
     | 99.00th=[ 5014], 99.50th=[ 5342], 99.90th=[ 6063], 99.95th=[ 6783],
     | 99.99th=[ 8160]
   bw (  KiB/s): min=781888, max=795584, per=100.00%, avg=788736.00, stdev=9684.53, samples=2
   iops        : min=24434, max=24862, avg=24648.00, stdev=302.64, samples=2
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.44%, 4=95.79%, 10=3.69%, 20=0.01%
  cpu          : usr=29.32%, sys=51.71%, ctx=28640, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=730MiB/s (765MB/s), 730MiB/s-730MiB/s (765MB/s-765MB/s), io=1024MiB (1074MB), run=1403-1403msec

Disk stats (read/write):
  sda: ios=0/32569, merge=0/0, ticks=0/5509, in_queue=5872, util=99.93%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=5219: Sat Aug 21 22:11:06 2021
  read: IOPS=32.7k, BW=1023MiB/s (1073MB/s)(1024MiB/1001msec)
    slat (usec): min=4, max=894, avg=16.56, stdev=20.34
    clat (usec): min=331, max=5946, avg=1925.87, stdev=432.05
     lat (usec): min=371, max=5953, avg=1944.28, stdev=436.17
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1172], 10.00th=[ 1303], 20.00th=[ 1516],
     | 30.00th=[ 1729], 40.00th=[ 1942], 50.00th=[ 2040], 60.00th=[ 2114],
     | 70.00th=[ 2147], 80.00th=[ 2212], 90.00th=[ 2311], 95.00th=[ 2474],
     | 99.00th=[ 3032], 99.50th=[ 3195], 99.90th=[ 4359], 99.95th=[ 4752],
     | 99.99th=[ 5604]
   bw (  MiB/s): min= 1137, max= 1137, per=100.00%, avg=1137.81, stdev= 0.00, samples=1
   iops        : min=36410, max=36410, avg=36410.00, stdev= 0.00, samples=1
  lat (usec)   : 500=0.02%, 750=0.07%, 1000=0.97%
  lat (msec)   : 2=44.37%, 4=54.40%, 10=0.17%
  cpu          : usr=34.20%, sys=58.60%, ctx=946, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1023MiB/s (1073MB/s), 1023MiB/s-1023MiB/s (1073MB/s-1073MB/s), io=1024MiB (1074MB), run=1001-1001msec

Disk stats (read/write):
  sda: ios=29697/3, merge=0/0, ticks=26558/5, in_queue=26448, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1803 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1920 MiB / 1984 MiB / 1886 MiB / 0 MiB / 1 MiB ( 1920 MiB / 1984 MiB / 1886 MiB / 0 MiB / 1 MiB )
2 min passed...1920 MiB / 2304 MiB / 1898 MiB / 0 MiB / 1 MiB ( 0 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
3 min passed...1920 MiB / 2464 MiB / 1908 MiB / 0 MiB / 2 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 1 MiB )
4 min passed...1952 MiB / 2464 MiB / 1917 MiB / 0 MiB / 2 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
5 min passed...1952 MiB / 2496 MiB / 1930 MiB / 0 MiB / 2 MiB ( 0 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
6 min passed...1952 MiB / 2496 MiB / 1940 MiB / 0 MiB / 2 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
7 min passed...1952 MiB / 2688 MiB / 1951 MiB / 0 MiB / 3 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 1 MiB )
8 min passed...1984 MiB / 2848 MiB / 1960 MiB / 0 MiB / 3 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
9 min passed...2048 MiB / 3040 MiB / 1968 MiB / 0 MiB / 3 MiB ( 64 MiB / 192 MiB / 8 MiB / 0 MiB / 0 MiB )
10 min passed...2048 MiB / 3040 MiB / 1980 MiB / 0 MiB / 4 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
11 min passed...2080 MiB / 3040 MiB / 1989 MiB / 0 MiB / 4 MiB ( 32 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...2112 MiB / 3136 MiB / 1999 MiB / 0 MiB / 4 MiB ( 32 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...2112 MiB / 3360 MiB / 2008 MiB / 0 MiB / 4 MiB ( 0 MiB / 224 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...2112 MiB / 3936 MiB / 2017 MiB / 0 MiB / 5 MiB ( 0 MiB / 576 MiB / 9 MiB / 0 MiB / 1 MiB )
15 min passed...2112 MiB / 3936 MiB / 2025 MiB / 0 MiB / 5 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
16 min passed...2112 MiB / 3968 MiB / 2038 MiB / 1056 MiB / 1030 MiB ( 0 MiB / 32 MiB / 13 MiB / 1056 MiB / 1025 MiB )
17 min passed...2112 MiB / 4000 MiB / 2050 MiB / 1056 MiB / 1030 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
18 min passed...2112 MiB / 4320 MiB / 2059 MiB / 1056 MiB / 1030 MiB ( 0 MiB / 320 MiB / 9 MiB / 0 MiB / 0 MiB )
19 min passed...2112 MiB / 4640 MiB / 2072 MiB / 1056 MiB / 1030 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
20 min passed...2112 MiB / 4800 MiB / 2081 MiB / 1056 MiB / 1031 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 1 MiB )
21 min passed...2144 MiB / 4928 MiB / 2091 MiB / 1056 MiB / 1031 MiB ( 32 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...2144 MiB / 4928 MiB / 2101 MiB / 1056 MiB / 1031 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
23 min passed...2144 MiB / 5152 MiB / 2112 MiB / 1056 MiB / 1032 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 1 MiB )
24 min passed...2176 MiB / 5280 MiB / 2122 MiB / 1056 MiB / 1032 MiB ( 32 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
25 min passed...2176 MiB / 5312 MiB / 2130 MiB / 1056 MiB / 1032 MiB ( 0 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
26 min passed...2208 MiB / 5376 MiB / 2142 MiB / 1056 MiB / 1032 MiB ( 32 MiB / 64 MiB / 12 MiB / 0 MiB / 0 MiB )
27 min passed...2208 MiB / 5376 MiB / 2150 MiB / 1056 MiB / 1032 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
28 min passed...2208 MiB / 5376 MiB / 2162 MiB / 1056 MiB / 1033 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
29 min passed...2208 MiB / 5600 MiB / 2172 MiB / 1056 MiB / 1033 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...2208 MiB / 5952 MiB / 2184 MiB / 1056 MiB / 1033 MiB ( 0 MiB / 352 MiB / 12 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 22, 2021

Round 2 - Run 8 - Test X6: 4x async sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=3): [W(2),f(1),W(1)][100.0%][w=721MiB/s][w=23.1k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5585: Sun Aug 22 00:23:27 2021
  write: IOPS=5187, BW=162MiB/s (170MB/s)(3072MiB/18951msec); 0 zone resets
    slat (usec): min=9, max=29738, avg=47.81, stdev=118.51
    clat (usec): min=2, max=47685, avg=118.31, stdev=322.58
     lat (usec): min=49, max=53515, avg=168.88, stdev=349.46
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   36], 20.00th=[   64],
     | 30.00th=[   76], 40.00th=[   87], 50.00th=[   98], 60.00th=[  113],
     | 70.00th=[  129], 80.00th=[  153], 90.00th=[  194], 95.00th=[  237],
     | 99.00th=[  429], 99.50th=[  652], 99.90th=[ 1795], 99.95th=[ 2933],
     | 99.99th=[12649]
   bw (  KiB/s): min=97792, max=200047, per=24.97%, avg=165128.19, stdev=22519.74, samples=37
   iops        : min= 3056, max= 6251, avg=5160.14, stdev=703.74, samples=37
  lat (usec)   : 4=7.25%, 10=1.72%, 20=0.30%, 50=2.94%, 100=39.45%
  lat (usec)   : 250=44.14%, 500=3.43%, 750=0.37%, 1000=0.15%
  lat (msec)   : 2=0.16%, 4=0.05%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=10.37%, sys=16.65%, ctx=138923, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5586: Sun Aug 22 00:23:27 2021
  write: IOPS=5166, BW=161MiB/s (169MB/s)(3072MiB/19026msec); 0 zone resets
    slat (usec): min=10, max=58681, avg=47.55, stdev=198.16
    clat (usec): min=2, max=112837, avg=119.39, stdev=485.02
     lat (usec): min=50, max=112872, avg=169.70, stdev=530.85
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   38], 20.00th=[   64],
     | 30.00th=[   76], 40.00th=[   86], 50.00th=[   98], 60.00th=[  112],
     | 70.00th=[  129], 80.00th=[  153], 90.00th=[  192], 95.00th=[  237],
     | 99.00th=[  429], 99.50th=[  644], 99.90th=[ 1942], 99.95th=[ 3130],
     | 99.99th=[15926]
   bw (  KiB/s): min=92480, max=199728, per=24.97%, avg=165119.74, stdev=25454.70, samples=38
   iops        : min= 2890, max= 6241, avg=5159.89, stdev=795.44, samples=38
  lat (usec)   : 4=6.99%, 10=1.80%, 20=0.34%, 50=2.94%, 100=39.90%
  lat (usec)   : 250=43.93%, 500=3.34%, 750=0.36%, 1000=0.14%
  lat (msec)   : 2=0.16%, 4=0.06%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=9.77%, sys=17.18%, ctx=138611, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5587: Sun Aug 22 00:23:27 2021
  write: IOPS=5210, BW=163MiB/s (171MB/s)(3072MiB/18866msec); 0 zone resets
    slat (usec): min=9, max=22649, avg=46.87, stdev=97.42
    clat (usec): min=2, max=48844, avg=118.23, stdev=304.87
     lat (usec): min=47, max=48897, avg=167.93, stdev=321.21
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   41], 20.00th=[   64],
     | 30.00th=[   76], 40.00th=[   87], 50.00th=[   98], 60.00th=[  113],
     | 70.00th=[  129], 80.00th=[  153], 90.00th=[  192], 95.00th=[  239],
     | 99.00th=[  433], 99.50th=[  660], 99.90th=[ 1778], 99.95th=[ 2638],
     | 99.99th=[10159]
   bw (  KiB/s): min=101248, max=199744, per=25.09%, avg=165937.14, stdev=22100.74, samples=37
   iops        : min= 3164, max= 6242, avg=5185.49, stdev=690.63, samples=37
  lat (usec)   : 4=7.05%, 10=1.57%, 20=0.29%, 50=2.87%, 100=39.96%
  lat (usec)   : 250=43.97%, 500=3.54%, 750=0.35%, 1000=0.16%
  lat (msec)   : 2=0.17%, 4=0.05%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=9.78%, sys=17.39%, ctx=138170, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5588: Sun Aug 22 00:23:27 2021
  write: IOPS=5191, BW=162MiB/s (170MB/s)(3072MiB/18934msec); 0 zone resets
    slat (usec): min=9, max=9250, avg=47.02, stdev=69.72
    clat (usec): min=2, max=113013, avg=119.10, stdev=463.20
     lat (usec): min=47, max=113135, avg=168.79, stdev=467.69
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[   41], 20.00th=[   64],
     | 30.00th=[   76], 40.00th=[   86], 50.00th=[   97], 60.00th=[  112],
     | 70.00th=[  128], 80.00th=[  151], 90.00th=[  192], 95.00th=[  235],
     | 99.00th=[  437], 99.50th=[  693], 99.90th=[ 1942], 99.95th=[ 3359],
     | 99.99th=[12256]
   bw (  KiB/s): min=101952, max=202048, per=25.00%, avg=165339.54, stdev=24229.28, samples=37
   iops        : min= 3186, max= 6314, avg=5166.76, stdev=757.15, samples=37
  lat (usec)   : 4=7.00%, 10=1.61%, 20=0.29%, 50=2.89%, 100=40.56%
  lat (usec)   : 250=43.63%, 500=3.23%, 750=0.36%, 1000=0.18%
  lat (msec)   : 2=0.17%, 4=0.06%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=9.18%, sys=17.71%, ctx=138098, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=646MiB/s (677MB/s), 161MiB/s-163MiB/s (169MB/s-171MB/s), io=12.0GiB (12.9GB), run=18866-19026msec

Disk stats (read/write):
  sda: ios=0/391862, merge=0/227, ticks=0/45665, in_queue=51588, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 0 MiB ( 12289 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12291 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 22, 2021

Round 2 - Run 8 - Test X6: 4x async sequential read/write 32K

Guest (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=264MiB/s][r=8455 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5793: Sun Aug 22 00:39:15 2021
  read: IOPS=2074, BW=64.8MiB/s (67.0MB/s)(3072MiB/47394msec)
    slat (usec): min=8, max=349, avg=31.54, stdev= 9.33
    clat (usec): min=88, max=5195, avg=441.03, stdev=110.93
     lat (usec): min=105, max=5223, avg=474.51, stdev=112.62
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  359], 10.00th=[  383], 20.00th=[  400],
     | 30.00th=[  416], 40.00th=[  429], 50.00th=[  437], 60.00th=[  449],
     | 70.00th=[  465], 80.00th=[  482], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  619], 99.50th=[  676], 99.90th=[ 2147], 99.95th=[ 2737],
     | 99.99th=[ 3425]
   bw (  KiB/s): min=60800, max=111393, per=25.01%, avg=66390.05, stdev=6843.43, samples=94
   iops        : min= 1900, max= 3481, avg=2074.66, stdev=213.86, samples=94
  lat (usec)   : 100=0.01%, 250=2.60%, 500=84.45%, 750=12.63%, 1000=0.12%
  lat (msec)   : 2=0.08%, 4=0.10%, 10=0.01%
  cpu          : usr=4.19%, sys=9.51%, ctx=98397, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5794: Sun Aug 22 00:39:15 2021
  read: IOPS=2078, BW=64.0MiB/s (68.1MB/s)(3072MiB/47290msec)
    slat (usec): min=9, max=250, avg=31.30, stdev= 9.24
    clat (usec): min=90, max=5270, avg=440.30, stdev=106.96
     lat (usec): min=107, max=5303, avg=473.52, stdev=108.35
    clat percentiles (usec):
     |  1.00th=[  163],  5.00th=[  355], 10.00th=[  379], 20.00th=[  400],
     | 30.00th=[  412], 40.00th=[  424], 50.00th=[  437], 60.00th=[  449],
     | 70.00th=[  465], 80.00th=[  482], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  611], 99.50th=[  668], 99.90th=[ 2147], 99.95th=[ 2638],
     | 99.99th=[ 3163]
   bw (  KiB/s): min=61248, max=108391, per=25.06%, avg=66527.68, stdev=6731.37, samples=94
   iops        : min= 1914, max= 3387, avg=2078.98, stdev=210.34, samples=94
  lat (usec)   : 100=0.01%, 250=2.71%, 500=84.48%, 750=12.52%, 1000=0.10%
  lat (msec)   : 2=0.07%, 4=0.10%, 10=0.01%
  cpu          : usr=4.03%, sys=9.47%, ctx=98382, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5795: Sun Aug 22 00:39:15 2021
  read: IOPS=2080, BW=65.0MiB/s (68.2MB/s)(3072MiB/47244msec)
    slat (usec): min=7, max=516, avg=31.44, stdev= 9.46
    clat (usec): min=5, max=6778, avg=439.65, stdev=114.27
     lat (usec): min=96, max=6812, avg=473.01, stdev=115.87
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  355], 10.00th=[  379], 20.00th=[  400],
     | 30.00th=[  412], 40.00th=[  424], 50.00th=[  437], 60.00th=[  449],
     | 70.00th=[  461], 80.00th=[  482], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  611], 99.50th=[  668], 99.90th=[ 2245], 99.95th=[ 2835],
     | 99.99th=[ 3359]
   bw (  KiB/s): min=60032, max=102208, per=25.08%, avg=66586.44, stdev=6461.81, samples=94
   iops        : min= 1876, max= 3194, avg=2080.80, stdev=201.93, samples=94
  lat (usec)   : 10=0.01%, 100=0.05%, 250=2.96%, 500=84.23%, 750=12.46%
  lat (usec)   : 1000=0.11%
  lat (msec)   : 2=0.08%, 4=0.11%, 10=0.01%
  cpu          : usr=4.11%, sys=9.42%, ctx=98422, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5796: Sun Aug 22 00:39:15 2021
  read: IOPS=2078, BW=64.9MiB/s (68.1MB/s)(3072MiB/47298msec)
    slat (usec): min=9, max=863, avg=31.42, stdev= 9.66
    clat (usec): min=6, max=6184, avg=440.23, stdev=117.69
     lat (usec): min=103, max=6215, avg=473.58, stdev=119.15
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  355], 10.00th=[  379], 20.00th=[  400],
     | 30.00th=[  416], 40.00th=[  424], 50.00th=[  437], 60.00th=[  449],
     | 70.00th=[  465], 80.00th=[  482], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  619], 99.50th=[  668], 99.90th=[ 2474], 99.95th=[ 2966],
     | 99.99th=[ 3326]
   bw (  KiB/s): min=60288, max=117397, per=25.05%, avg=66516.49, stdev=7029.10, samples=94
   iops        : min= 1884, max= 3668, avg=2078.61, stdev=219.61, samples=94
  lat (usec)   : 10=0.01%, 50=0.01%, 100=0.03%, 250=3.06%, 500=84.06%
  lat (usec)   : 750=12.52%, 1000=0.12%
  lat (msec)   : 2=0.07%, 4=0.13%, 10=0.01%
  cpu          : usr=3.99%, sys=9.56%, ctx=98409, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=259MiB/s (272MB/s), 64.8MiB/s-65.0MiB/s (67.0MB/s-68.2MB/s), io=12.0GiB (12.9GB), run=47244-47394msec

Disk stats (read/write):
  sda: ios=391951/14, merge=0/10, ticks=171630/14, in_queue=183028, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12291 MiB / 12288 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12291 MiB / 12288 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12288 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 12293 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Script ran 1868 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
1 min passed...23136 MiB / 24896 MiB / 23194 MiB / 32 MiB / 51 MiB ( 23136 MiB / 24896 MiB / 23193 MiB / 32 MiB / 51 MiB )
2 min passed...23168 MiB / 25280 MiB / 23206 MiB / 32 MiB / 51 MiB ( 32 MiB / 384 MiB / 12 MiB / 0 MiB / 0 MiB )
3 min passed...23168 MiB / 25440 MiB / 23218 MiB / 32 MiB / 51 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
4 min passed...23264 MiB / 25600 MiB / 23230 MiB / 32 MiB / 51 MiB ( 96 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...23264 MiB / 25856 MiB / 23242 MiB / 32 MiB / 51 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
6 min passed...23264 MiB / 26080 MiB / 23252 MiB / 32 MiB / 51 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
7 min passed...23264 MiB / 26624 MiB / 23263 MiB / 32 MiB / 51 MiB ( 0 MiB / 544 MiB / 11 MiB / 0 MiB / 0 MiB )
8 min passed...23264 MiB / 26656 MiB / 23274 MiB / 32 MiB / 51 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
9 min passed...23296 MiB / 26816 MiB / 23283 MiB / 32 MiB / 52 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 1 MiB )
10 min passed...23328 MiB / 26944 MiB / 23294 MiB / 32 MiB / 52 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
11 min passed...23328 MiB / 26944 MiB / 23303 MiB / 32 MiB / 52 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...23328 MiB / 27456 MiB / 23315 MiB / 32 MiB / 52 MiB ( 0 MiB / 512 MiB / 12 MiB / 0 MiB / 0 MiB )
13 min passed...23328 MiB / 27520 MiB / 23326 MiB / 32 MiB / 52 MiB ( 0 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
14 min passed...23328 MiB / 27744 MiB / 23336 MiB / 32 MiB / 52 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
15 min passed...23328 MiB / 27872 MiB / 23345 MiB / 32 MiB / 52 MiB ( 0 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
16 min passed...23328 MiB / 28000 MiB / 23357 MiB / 11840 MiB / 11873 MiB ( 0 MiB / 128 MiB / 12 MiB / 11808 MiB / 11821 MiB )
17 min passed...23328 MiB / 28352 MiB / 23365 MiB / 12032 MiB / 11988 MiB ( 0 MiB / 352 MiB / 8 MiB / 192 MiB / 115 MiB )
18 min passed...23360 MiB / 28640 MiB / 23376 MiB / 12032 MiB / 11988 MiB ( 32 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
19 min passed...23392 MiB / 28768 MiB / 23384 MiB / 12032 MiB / 11988 MiB ( 32 MiB / 128 MiB / 8 MiB / 0 MiB / 0 MiB )
20 min passed...23424 MiB / 28832 MiB / 23395 MiB / 12032 MiB / 11988 MiB ( 32 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...23424 MiB / 29088 MiB / 23405 MiB / 12032 MiB / 11988 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...23424 MiB / 29248 MiB / 23415 MiB / 12032 MiB / 11989 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 1 MiB )
23 min passed...23424 MiB / 29248 MiB / 23427 MiB / 12032 MiB / 11989 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...23424 MiB / 29312 MiB / 23437 MiB / 12032 MiB / 11989 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
25 min passed...23424 MiB / 29376 MiB / 23445 MiB / 12032 MiB / 11989 MiB ( 0 MiB / 64 MiB / 8 MiB / 0 MiB / 0 MiB )
26 min passed...23424 MiB / 29504 MiB / 23455 MiB / 12032 MiB / 11989 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...23456 MiB / 29568 MiB / 23465 MiB / 12032 MiB / 11989 MiB ( 32 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...23456 MiB / 30144 MiB / 23474 MiB / 12032 MiB / 11990 MiB ( 0 MiB / 576 MiB / 9 MiB / 0 MiB / 1 MiB )
29 min passed...23456 MiB / 30176 MiB / 23484 MiB / 12032 MiB / 11990 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
30 min passed...23456 MiB / 30368 MiB / 23494 MiB / 12032 MiB / 11990 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
31 min passed...23456 MiB / 30496 MiB / 23507 MiB / 12032 MiB / 11990 MiB ( 0 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 22, 2021

Round 2 - Run 8 - Test X7: 4x sync sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=19.8MiB/s][w=634 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6028: Sun Aug 22 00:58:37 2021
  write: IOPS=181, BW=5805KiB/s (5945kB/s)(512MiB/90313msec); 0 zone resets
    clat (usec): min=2913, max=43716, avg=5485.91, stdev=1669.77
     lat (usec): min=2915, max=43719, avg=5488.30, stdev=1669.77
    clat percentiles (usec):
     |  1.00th=[ 4113],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5407],
     | 70.00th=[ 5604], 80.00th=[ 5800], 90.00th=[ 6259], 95.00th=[ 7177],
     | 99.00th=[ 9110], 99.50th=[17957], 99.90th=[26870], 99.95th=[31589],
     | 99.99th=[43254]
   bw (  KiB/s): min= 4416, max= 6592, per=25.00%, avg=5804.63, stdev=453.94, samples=180
   iops        : min=  138, max=  206, avg=181.35, stdev=14.18, samples=180
  lat (msec)   : 4=0.42%, 10=98.68%, 20=0.52%, 50=0.38%
  cpu          : usr=0.69%, sys=2.38%, ctx=49263, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6029: Sun Aug 22 00:58:37 2021
  write: IOPS=181, BW=5805KiB/s (5944kB/s)(512MiB/90317msec); 0 zone resets
    clat (usec): min=3311, max=50356, avg=5485.66, stdev=1660.44
     lat (usec): min=3313, max=50358, avg=5488.06, stdev=1660.46
    clat percentiles (usec):
     |  1.00th=[ 4113],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5407],
     | 70.00th=[ 5604], 80.00th=[ 5800], 90.00th=[ 6259], 95.00th=[ 7177],
     | 99.00th=[ 8848], 99.50th=[18220], 99.90th=[25297], 99.95th=[28443],
     | 99.99th=[43779]
   bw (  KiB/s): min= 4416, max= 6592, per=25.00%, avg=5804.69, stdev=450.13, samples=180
   iops        : min=  138, max=  206, avg=181.36, stdev=14.06, samples=180
  lat (msec)   : 4=0.37%, 10=98.73%, 20=0.51%, 50=0.38%, 100=0.01%
  cpu          : usr=0.73%, sys=2.50%, ctx=49806, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6030: Sun Aug 22 00:58:37 2021
  write: IOPS=181, BW=5805KiB/s (5944kB/s)(512MiB/90320msec); 0 zone resets
    clat (usec): min=2970, max=50333, avg=5486.20, stdev=1666.35
     lat (usec): min=2971, max=50336, avg=5488.60, stdev=1666.33
    clat percentiles (usec):
     |  1.00th=[ 4113],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5407],
     | 70.00th=[ 5604], 80.00th=[ 5800], 90.00th=[ 6259], 95.00th=[ 7111],
     | 99.00th=[ 8979], 99.50th=[17957], 99.90th=[26608], 99.95th=[29492],
     | 99.99th=[43254]
   bw (  KiB/s): min= 4416, max= 6592, per=25.00%, avg=5804.29, stdev=451.75, samples=180
   iops        : min=  138, max=  206, avg=181.34, stdev=14.13, samples=180
  lat (msec)   : 4=0.41%, 10=98.67%, 20=0.54%, 50=0.37%, 100=0.01%
  cpu          : usr=0.68%, sys=2.39%, ctx=52313, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6031: Sun Aug 22 00:58:37 2021
  write: IOPS=181, BW=5805KiB/s (5944kB/s)(512MiB/90318msec); 0 zone resets
    clat (usec): min=2913, max=41057, avg=5485.94, stdev=1646.98
     lat (usec): min=2914, max=41060, avg=5488.36, stdev=1646.94
    clat percentiles (usec):
     |  1.00th=[ 4146],  5.00th=[ 4424], 10.00th=[ 4555], 20.00th=[ 4752],
     | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5407],
     | 70.00th=[ 5604], 80.00th=[ 5800], 90.00th=[ 6259], 95.00th=[ 7177],
     | 99.00th=[ 9241], 99.50th=[17957], 99.90th=[25035], 99.95th=[30540],
     | 99.99th=[40109]
   bw (  KiB/s): min= 4416, max= 6592, per=25.00%, avg=5805.25, stdev=450.03, samples=180
   iops        : min=  138, max=  206, avg=181.40, stdev=14.08, samples=180
  lat (msec)   : 4=0.38%, 10=98.68%, 20=0.54%, 50=0.39%
  cpu          : usr=0.66%, sys=2.45%, ctx=49274, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=22.7MiB/s (23.8MB/s), 5805KiB/s-5805KiB/s (5944kB/s-5945kB/s), io=2048MiB (2147MB), run=90313-90320msec

Disk stats (read/write):
  sda: ios=0/130920, merge=0/120868, ticks=0/100074, in_queue=106428, util=99.61%
0 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 2778 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2778 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2779 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2780 MiB / 0 MiB ( 1 MiB / 0 MiB )

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member