how to best benchmark SSDs?

Dunuin · Aug 17, 2021

Round 2 - Run 6 - Test X6: 4x async sequential read/write 32K

Guest (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=266MiB/s][r=8503 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5132: Mon Aug 16 23:46:11 2021
  read: IOPS=2075, BW=64.9MiB/s (68.0MB/s)(3072MiB/47354msec)
    slat (usec): min=8, max=394, avg=31.50, stdev= 9.24
    clat (usec): min=60, max=9063, avg=440.82, stdev=182.50
     lat (usec): min=111, max=9094, avg=474.16, stdev=183.20
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  355], 10.00th=[  371], 20.00th=[  388],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  449], 80.00th=[  469], 90.00th=[  498], 95.00th=[  523],
     | 99.00th=[  742], 99.50th=[ 1811], 99.90th=[ 3032], 99.95th=[ 3163],
     | 99.99th=[ 3884]
   bw (  KiB/s): min=61888, max=103936, per=25.00%, avg=66436.62, stdev=4296.59, samples=94
   iops        : min= 1934, max= 3248, avg=2076.12, stdev=134.28, samples=94
  lat (usec)   : 100=0.01%, 250=0.89%, 500=90.04%, 750=8.07%, 1000=0.26%
  lat (msec)   : 2=0.26%, 4=0.46%, 10=0.01%
  cpu          : usr=3.83%, sys=9.54%, ctx=98415, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5133: Mon Aug 16 23:46:11 2021
  read: IOPS=2078, BW=64.0MiB/s (68.1MB/s)(3072MiB/47293msec)
    slat (usec): min=8, max=314, avg=31.65, stdev= 8.94
    clat (usec): min=100, max=7621, avg=440.03, stdev=180.85
     lat (usec): min=114, max=7653, avg=473.51, stdev=181.57
    clat percentiles (usec):
     |  1.00th=[  200],  5.00th=[  355], 10.00th=[  371], 20.00th=[  388],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  449], 80.00th=[  469], 90.00th=[  494], 95.00th=[  523],
     | 99.00th=[  709], 99.50th=[ 1860], 99.90th=[ 3032], 99.95th=[ 3130],
     | 99.99th=[ 3523]
   bw (  KiB/s): min=61888, max=109888, per=25.03%, avg=66514.13, stdev=4760.68, samples=94
   iops        : min= 1934, max= 3434, avg=2078.53, stdev=148.77, samples=94
  lat (usec)   : 250=1.14%, 500=90.10%, 750=7.82%, 1000=0.23%
  lat (msec)   : 2=0.23%, 4=0.46%, 10=0.01%
  cpu          : usr=3.85%, sys=9.67%, ctx=98361, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5134: Mon Aug 16 23:46:11 2021
  read: IOPS=2083, BW=65.1MiB/s (68.3MB/s)(3072MiB/47173msec)
    slat (usec): min=9, max=238, avg=31.55, stdev= 9.08
    clat (usec): min=92, max=6539, avg=438.94, stdev=176.92
     lat (usec): min=106, max=6564, avg=472.32, stdev=177.69
    clat percentiles (usec):
     |  1.00th=[  198],  5.00th=[  355], 10.00th=[  371], 20.00th=[  388],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  449], 80.00th=[  469], 90.00th=[  494], 95.00th=[  523],
     | 99.00th=[  693], 99.50th=[ 1631], 99.90th=[ 3032], 99.95th=[ 3130],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=63488, max=106496, per=25.10%, avg=66685.23, stdev=4587.12, samples=94
   iops        : min= 1984, max= 3328, avg=2083.89, stdev=143.36, samples=94
  lat (usec)   : 100=0.01%, 250=1.39%, 500=89.71%, 750=7.96%, 1000=0.23%
  lat (msec)   : 2=0.24%, 4=0.45%, 10=0.01%
  cpu          : usr=3.97%, sys=9.44%, ctx=98367, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5135: Mon Aug 16 23:46:11 2021
  read: IOPS=2081, BW=65.1MiB/s (68.2MB/s)(3072MiB/47224msec)
    slat (usec): min=8, max=314, avg=31.65, stdev= 9.25
    clat (usec): min=20, max=6603, avg=439.33, stdev=183.24
     lat (usec): min=115, max=6662, avg=472.82, stdev=183.93
    clat percentiles (usec):
     |  1.00th=[  192],  5.00th=[  351], 10.00th=[  367], 20.00th=[  388],
     | 30.00th=[  400], 40.00th=[  412], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  449], 80.00th=[  469], 90.00th=[  494], 95.00th=[  523],
     | 99.00th=[  734], 99.50th=[ 1926], 99.90th=[ 3032], 99.95th=[ 3130],
     | 99.99th=[ 3392]
   bw (  KiB/s): min=62016, max=107264, per=25.06%, avg=66601.44, stdev=4709.12, samples=94
   iops        : min= 1938, max= 3352, avg=2081.30, stdev=147.19, samples=94
  lat (usec)   : 50=0.01%, 100=0.01%, 250=1.66%, 500=89.41%, 750=7.95%
  lat (usec)   : 1000=0.24%
  lat (msec)   : 2=0.25%, 4=0.48%, 10=0.01%
  cpu          : usr=3.85%, sys=9.54%, ctx=98455, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=259MiB/s (272MB/s), 64.9MiB/s-65.1MiB/s (68.0MB/s-68.3MB/s), io=12.0GiB (12.9GB), run=47173-47354msec

Disk stats (read/write):
  sda: ios=392300/7, merge=0/2, ticks=171310/7, in_queue=180228, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12289 MiB / 12288 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12292 MiB / 12289 MiB ( 3 MiB / 1 MiB )
Final...Guest Writes/Reads (Changes): 12292 MiB / 12289 MiB ( 0 MiB / 0 MiB )
Script ran 1872 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...29056 MiB / 31136 MiB / 29071 MiB / 128 MiB / 126 MiB ( 29056 MiB / 31136 MiB / 29071 MiB / 128 MiB / 126 MiB )
2 min passed...29056 MiB / 31136 MiB / 29071 MiB / 128 MiB / 126 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...29056 MiB / 31264 MiB / 29071 MiB / 128 MiB / 126 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...29056 MiB / 31552 MiB / 29071 MiB / 128 MiB / 126 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...29056 MiB / 31808 MiB / 29071 MiB / 128 MiB / 126 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...29056 MiB / 32064 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
7 min passed...29056 MiB / 32064 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...29056 MiB / 32064 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...29056 MiB / 32256 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...29056 MiB / 32448 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...29056 MiB / 32832 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...29056 MiB / 33344 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...29056 MiB / 33440 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...29056 MiB / 33696 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...29056 MiB / 34144 MiB / 29074 MiB / 128 MiB / 126 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...29056 MiB / 34528 MiB / 29080 MiB / 10848 MiB / 10893 MiB ( 0 MiB / 384 MiB / 6 MiB / 10720 MiB / 10767 MiB )
17 min passed...29056 MiB / 34880 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 352 MiB / 0 MiB / 1408 MiB / 1354 MiB )
18 min passed...29056 MiB / 35136 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...29056 MiB / 35136 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...29056 MiB / 35808 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...29056 MiB / 36032 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...29056 MiB / 36512 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...29056 MiB / 36800 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...29056 MiB / 36960 MiB / 29080 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...29056 MiB / 37376 MiB / 29083 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 416 MiB / 3 MiB / 0 MiB / 0 MiB )
26 min passed...29056 MiB / 37728 MiB / 29084 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 352 MiB / 1 MiB / 0 MiB / 0 MiB )
27 min passed...29056 MiB / 38112 MiB / 29084 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...29056 MiB / 38272 MiB / 29084 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...29088 MiB / 38560 MiB / 29087 MiB / 12256 MiB / 12247 MiB ( 32 MiB / 288 MiB / 3 MiB / 0 MiB / 0 MiB )
30 min passed...29088 MiB / 38752 MiB / 29088 MiB / 12256 MiB / 12247 MiB ( 0 MiB / 192 MiB / 1 MiB / 0 MiB / 0 MiB )
31 min passed...29088 MiB / 39520 MiB / 29126 MiB / 12256 MiB / 12248 MiB ( 0 MiB / 768 MiB / 38 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 17, 2021

Round 2 - Run 6 - Test X7: 4x sync sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=17.0MiB/s][w=545 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5381: Tue Aug 17 00:12:29 2021
  write: IOPS=150, BW=4826KiB/s (4942kB/s)(512MiB/108639msec); 0 zone resets
    clat (usec): min=4542, max=56143, avg=6605.68, stdev=2087.73
     lat (usec): min=4544, max=56145, avg=6607.95, stdev=2087.78
    clat percentiles (usec):
     |  1.00th=[ 5211],  5.00th=[ 5538], 10.00th=[ 5669], 20.00th=[ 5866],
     | 30.00th=[ 5997], 40.00th=[ 6128], 50.00th=[ 6259], 60.00th=[ 6390],
     | 70.00th=[ 6521], 80.00th=[ 6783], 90.00th=[ 7701], 95.00th=[ 8586],
     | 99.00th=[12256], 99.50th=[21890], 99.90th=[34341], 99.95th=[43254],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3520, max= 5440, per=25.00%, avg=4825.95, stdev=391.34, samples=217
   iops        : min=  110, max=  170, avg=150.79, stdev=12.25, samples=217
  lat (msec)   : 10=98.63%, 20=0.64%, 50=0.71%, 100=0.02%
  cpu          : usr=0.52%, sys=1.71%, ctx=49300, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5382: Tue Aug 17 00:12:29 2021
  write: IOPS=150, BW=4826KiB/s (4942kB/s)(512MiB/108637msec); 0 zone resets
    clat (usec): min=4433, max=56065, avg=6605.17, stdev=2087.77
     lat (usec): min=4435, max=56067, avg=6607.49, stdev=2087.75
    clat percentiles (usec):
     |  1.00th=[ 5211],  5.00th=[ 5473], 10.00th=[ 5669], 20.00th=[ 5866],
     | 30.00th=[ 5997], 40.00th=[ 6128], 50.00th=[ 6259], 60.00th=[ 6390],
     | 70.00th=[ 6521], 80.00th=[ 6783], 90.00th=[ 7701], 95.00th=[ 8586],
     | 99.00th=[12256], 99.50th=[21890], 99.90th=[34341], 99.95th=[43254],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3520, max= 5440, per=25.00%, avg=4825.07, stdev=390.82, samples=217
   iops        : min=  110, max=  170, avg=150.71, stdev=12.20, samples=217
  lat (msec)   : 10=98.64%, 20=0.63%, 50=0.71%, 100=0.02%
  cpu          : usr=0.49%, sys=1.71%, ctx=56493, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5383: Tue Aug 17 00:12:29 2021
  write: IOPS=150, BW=4826KiB/s (4942kB/s)(512MiB/108634msec); 0 zone resets
    clat (usec): min=4167, max=52829, avg=6603.58, stdev=2080.05
     lat (usec): min=4169, max=52832, avg=6605.98, stdev=2080.08
    clat percentiles (usec):
     |  1.00th=[ 5211],  5.00th=[ 5538], 10.00th=[ 5669], 20.00th=[ 5866],
     | 30.00th=[ 5997], 40.00th=[ 6128], 50.00th=[ 6259], 60.00th=[ 6390],
     | 70.00th=[ 6521], 80.00th=[ 6783], 90.00th=[ 7635], 95.00th=[ 8586],
     | 99.00th=[12649], 99.50th=[22152], 99.90th=[33817], 99.95th=[46924],
     | 99.99th=[52691]
   bw (  KiB/s): min= 3520, max= 5440, per=25.00%, avg=4825.17, stdev=391.37, samples=217
   iops        : min=  110, max=  170, avg=150.71, stdev=12.22, samples=217
  lat (msec)   : 10=98.77%, 20=0.51%, 50=0.70%, 100=0.02%
  cpu          : usr=0.66%, sys=1.96%, ctx=50094, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5384: Tue Aug 17 00:12:29 2021
  write: IOPS=150, BW=4826KiB/s (4942kB/s)(512MiB/108637msec); 0 zone resets
    clat (usec): min=4491, max=55955, avg=6604.96, stdev=2092.06
     lat (usec): min=4494, max=55957, avg=6607.34, stdev=2092.05
    clat percentiles (usec):
     |  1.00th=[ 5211],  5.00th=[ 5538], 10.00th=[ 5669], 20.00th=[ 5866],
     | 30.00th=[ 5997], 40.00th=[ 6128], 50.00th=[ 6259], 60.00th=[ 6390],
     | 70.00th=[ 6521], 80.00th=[ 6783], 90.00th=[ 7701], 95.00th=[ 8586],
     | 99.00th=[12256], 99.50th=[21890], 99.90th=[34341], 99.95th=[44827],
     | 99.99th=[54264]
   bw (  KiB/s): min= 3520, max= 5440, per=25.00%, avg=4826.04, stdev=391.35, samples=217
   iops        : min=  110, max=  170, avg=150.80, stdev=12.25, samples=217
  lat (msec)   : 10=98.65%, 20=0.62%, 50=0.70%, 100=0.02%
  cpu          : usr=0.52%, sys=1.69%, ctx=56432, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=18.9MiB/s (19.8MB/s), 4826KiB/s-4826KiB/s (4942kB/s-4942kB/s), io=2048MiB (2147MB), run=108634-108639msec

Disk stats (read/write):
  sda: ios=0/130855, merge=0/114480, ticks=0/115484, in_queue=121388, util=99.64%
0 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 2752 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2752 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2753 MiB / 0 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2753 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2753 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 17, 2021

Round 2 - Run 6 - Test X7: 4x sync sequential read/write 32K

Guest (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][87.5%][r=257MiB/s][r=8230 IOPS][eta 00m:01s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5591: Tue Aug 17 00:27:38 2021
  read: IOPS=2075, BW=64.9MiB/s (68.0MB/s)(512MiB/7893msec)
    clat (usec): min=301, max=3919, avg=472.53, stdev=169.70
     lat (usec): min=303, max=3922, avg=474.34, stdev=169.72
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  388], 10.00th=[  404], 20.00th=[  420],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  469],
     | 70.00th=[  482], 80.00th=[  498], 90.00th=[  529], 95.00th=[  553],
     | 99.00th=[  693], 99.50th=[ 1663], 99.90th=[ 3032], 99.95th=[ 3097],
     | 99.99th=[ 3589]
   bw (  KiB/s): min=63936, max=72384, per=25.01%, avg=66444.80, stdev=1944.71, samples=15
   iops        : min= 1998, max= 2262, avg=2076.40, stdev=60.77, samples=15
  lat (usec)   : 500=80.52%, 750=18.55%, 1000=0.24%
  lat (msec)   : 2=0.23%, 4=0.45%
  cpu          : usr=2.48%, sys=9.64%, ctx=16390, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5592: Tue Aug 17 00:27:38 2021
  read: IOPS=2081, BW=65.1MiB/s (68.2MB/s)(512MiB/7870msec)
    clat (usec): min=268, max=5308, avg=471.21, stdev=163.29
     lat (usec): min=269, max=5310, avg=473.00, stdev=163.32
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  388], 10.00th=[  400], 20.00th=[  420],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  453], 60.00th=[  469],
     | 70.00th=[  482], 80.00th=[  502], 90.00th=[  529], 95.00th=[  553],
     | 99.00th=[  676], 99.50th=[ 1221], 99.90th=[ 2966], 99.95th=[ 3064],
     | 99.99th=[ 4621]
   bw (  KiB/s): min=64640, max=70976, per=25.05%, avg=66568.53, stdev=1675.72, samples=15
   iops        : min= 2020, max= 2218, avg=2080.27, stdev=52.37, samples=15
  lat (usec)   : 500=80.11%, 750=19.03%, 1000=0.20%
  lat (msec)   : 2=0.26%, 4=0.38%, 10=0.01%
  cpu          : usr=2.69%, sys=9.63%, ctx=16393, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5593: Tue Aug 17 00:27:38 2021
  read: IOPS=2096, BW=65.5MiB/s (68.7MB/s)(512MiB/7814msec)
    clat (usec): min=280, max=7544, avg=467.81, stdev=141.42
     lat (usec): min=281, max=7546, avg=469.62, stdev=141.46
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  388], 10.00th=[  400], 20.00th=[  420],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  453], 60.00th=[  465],
     | 70.00th=[  482], 80.00th=[  498], 90.00th=[  529], 95.00th=[  553],
     | 99.00th=[  644], 99.50th=[ 1012], 99.90th=[ 2737], 99.95th=[ 2999],
     | 99.99th=[ 3130]
   bw (  KiB/s): min=65792, max=73344, per=25.25%, avg=67097.27, stdev=1849.50, samples=15
   iops        : min= 2056, max= 2292, avg=2096.73, stdev=57.83, samples=15
  lat (usec)   : 500=80.85%, 750=18.40%, 1000=0.23%
  lat (msec)   : 2=0.26%, 4=0.26%, 10=0.01%
  cpu          : usr=3.02%, sys=9.47%, ctx=16390, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5594: Tue Aug 17 00:27:38 2021
  read: IOPS=2076, BW=64.9MiB/s (68.0MB/s)(512MiB/7890msec)
    clat (usec): min=295, max=5610, avg=472.41, stdev=180.72
     lat (usec): min=297, max=5635, avg=474.21, stdev=180.78
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  388], 10.00th=[  400], 20.00th=[  416],
     | 30.00th=[  429], 40.00th=[  441], 50.00th=[  453], 60.00th=[  465],
     | 70.00th=[  482], 80.00th=[  498], 90.00th=[  529], 95.00th=[  553],
     | 99.00th=[  717], 99.50th=[ 2073], 99.90th=[ 2999], 99.95th=[ 3130],
     | 99.99th=[ 5145]
   bw (  KiB/s): min=63744, max=71744, per=25.00%, avg=66423.07, stdev=1775.65, samples=15
   iops        : min= 1992, max= 2242, avg=2075.67, stdev=55.48, samples=15
  lat (usec)   : 500=81.32%, 750=17.73%, 1000=0.26%
  lat (msec)   : 2=0.18%, 4=0.50%, 10=0.01%
  cpu          : usr=2.52%, sys=9.42%, ctx=16392, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=259MiB/s (272MB/s), 64.9MiB/s-65.5MiB/s (68.0MB/s-68.7MB/s), io=2048MiB (2147MB), run=7814-7893msec

Disk stats (read/write):
  sda: ios=63691/0, merge=0/0, ticks=27771/0, in_queue=29776, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2753 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Script ran 1918 seconds or 31 minutes.

host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...11168 MiB / 12416 MiB / 11213 MiB / 128 MiB / 138 MiB ( 11168 MiB / 12416 MiB / 11213 MiB / 128 MiB / 138 MiB )
2 min passed...20960 MiB / 23360 MiB / 20943 MiB / 256 MiB / 256 MiB ( 9792 MiB / 10944 MiB / 9730 MiB / 128 MiB / 118 MiB )
3 min passed...20960 MiB / 23552 MiB / 20943 MiB / 256 MiB / 256 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...20960 MiB / 23552 MiB / 20943 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...20960 MiB / 23648 MiB / 20947 MiB / 256 MiB / 256 MiB ( 0 MiB / 96 MiB / 4 MiB / 0 MiB / 0 MiB )
6 min passed...20960 MiB / 24160 MiB / 20947 MiB / 256 MiB / 256 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...20960 MiB / 24608 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 448 MiB / 4 MiB / 0 MiB / 0 MiB )
8 min passed...20960 MiB / 24928 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...20960 MiB / 24928 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...20960 MiB / 24960 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...20960 MiB / 25344 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...20960 MiB / 25696 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...20960 MiB / 26176 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...20960 MiB / 26176 MiB / 20951 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...20960 MiB / 26176 MiB / 20955 MiB / 256 MiB / 256 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
16 min passed...20960 MiB / 26656 MiB / 20955 MiB / 256 MiB / 256 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...20960 MiB / 27136 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 480 MiB / 0 MiB / 2048 MiB / 2048 MiB )
18 min passed...20960 MiB / 27584 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...20960 MiB / 27680 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...20960 MiB / 27680 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...20960 MiB / 27936 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...20960 MiB / 28544 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...20960 MiB / 28992 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...20960 MiB / 29280 MiB / 20955 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...20960 MiB / 29376 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 4 MiB / 0 MiB / 0 MiB )
26 min passed...20960 MiB / 29472 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...20960 MiB / 29792 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...20960 MiB / 30144 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...20960 MiB / 30336 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...20960 MiB / 30464 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...20960 MiB / 30560 MiB / 20959 MiB / 2304 MiB / 2304 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )

chrcoluk · Aug 17, 2021

Dunuin said:
I also tested "logbias=throughput" vs "logbias=latency". I ran the fio tests (Test 1 + 3 + TX7) directly on the host to a ext4 ontop of a zvol (8k volblocksize) ontop of a 4 disk striped mirror (ashift=12).

Write Performance:

logbias=throughput logbias=latency
sync 4k random 1,45 MiB/s 1,47 MiB/s
sync 4M sequential 184,00 MiB/s 179,00 MiB/s
4x sync 32K sequential 18,80 MiB/s 19,10 MiB/s

Total Write Amplification:

logbias=throughput logbias=latency
sync 4k random 54,78 x 56,72 x
sync 4M sequential 4,52 x 4,70 x
4x sync 32K sequential 12,03 x 12,36 x

So there is neither a big performance difference nor a write amplification difference.

Shame, there was reports, so I thought was worth mentioning.

Dunuin · Aug 17, 2021

Special Test 26: like Round 2 - Run 6 - Test X6 but fio run on host on 16K zvol

Fio test on host (part 1):

Code:

root@Hypervisor:~# bash /root/scripts/bench/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 3 (f=1): [_(1),f(2),W(1)][100.0%][w=1195MiB/s][w=38.2k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18646: Tue Aug 17 01:43:22 2021
  write: IOPS=5843, BW=183MiB/s (191MB/s)(3072MiB/16823msec); 0 zone resets
    slat (usec): min=6, max=55872, avg=22.09, stdev=211.47
    clat (nsec): min=1176, max=1313.9M, avg=137361.59, stdev=7965896.26
     lat (usec): min=44, max=1313.0k, avg=159.73, stdev=7968.63
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   43], 10.00th=[   46], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   57], 50.00th=[   60], 60.00th=[   64],
     | 70.00th=[   70], 80.00th=[   81], 90.00th=[  106], 95.00th=[  135],
     | 99.00th=[  251], 99.50th=[  343], 99.90th=[ 2089], 99.95th=[ 9896],
     | 99.99th=[28705]
   bw (  KiB/s): min=  832, max=384576, per=30.52%, avg=225061.93, stdev=116705.58, samples=27
   iops        : min=   26, max=12018, avg=7033.15, stdev=3647.04, samples=27
  lat (usec)   : 2=0.53%, 4=1.48%, 10=0.02%, 20=0.05%, 50=18.54%
  lat (usec)   : 100=67.56%, 250=10.80%, 500=0.77%, 750=0.09%, 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.03%, 10=0.03%, 20=0.03%, 50=0.02%
  lat (msec)   : 1000=0.01%
  cpu          : usr=8.36%, sys=10.14%, ctx=102157, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18647: Tue Aug 17 01:43:22 2021
  write: IOPS=5785, BW=181MiB/s (190MB/s)(3072MiB/16991msec); 0 zone resets
    slat (usec): min=6, max=44066, avg=22.69, stdev=216.90
    clat (nsec): min=1147, max=1313.0M, avg=138221.86, stdev=7974198.63
     lat (usec): min=43, max=1313.0k, avg=161.19, stdev=7977.26
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   43], 10.00th=[   47], 20.00th=[   50],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   70], 80.00th=[   81], 90.00th=[  105], 95.00th=[  133],
     | 99.00th=[  247], 99.50th=[  326], 99.90th=[ 1418], 99.95th=[ 8455],
     | 99.99th=[36963]
   bw (  KiB/s): min=  640, max=380160, per=29.97%, avg=220996.19, stdev=116574.78, samples=27
   iops        : min=   20, max=11880, avg=6906.11, stdev=3642.99, samples=27
  lat (usec)   : 2=0.64%, 4=1.44%, 10=0.02%, 20=0.06%, 50=17.41%
  lat (usec)   : 100=68.86%, 250=10.60%, 500=0.76%, 750=0.08%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.02%, 10=0.04%, 20=0.02%, 50=0.02%
  lat (msec)   : 100=0.01%, 1000=0.01%
  cpu          : usr=8.29%, sys=10.07%, ctx=102184, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18648: Tue Aug 17 01:43:22 2021
  write: IOPS=5806, BW=181MiB/s (190MB/s)(3072MiB/16929msec); 0 zone resets
    slat (usec): min=7, max=35490, avg=21.87, stdev=157.56
    clat (nsec): min=1143, max=1313.9M, avg=138748.51, stdev=7973735.91
     lat (usec): min=43, max=1313.0k, avg=160.91, stdev=7975.22
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   43], 10.00th=[   47], 20.00th=[   50],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   61], 60.00th=[   65],
     | 70.00th=[   70], 80.00th=[   81], 90.00th=[  106], 95.00th=[  135],
     | 99.00th=[  245], 99.50th=[  322], 99.90th=[ 1778], 99.95th=[ 8848],
     | 99.99th=[45351]
   bw (  KiB/s): min=  896, max=383872, per=30.18%, avg=222534.96, stdev=114659.76, samples=27
   iops        : min=   28, max=11996, avg=6954.11, stdev=3583.20, samples=27
  lat (usec)   : 2=0.65%, 4=1.31%, 10=0.02%, 20=0.05%, 50=17.60%
  lat (usec)   : 100=68.57%, 250=10.84%, 500=0.74%, 750=0.06%, 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.02%, 10=0.03%, 20=0.03%, 50=0.02%
  lat (msec)   : 100=0.01%, 1000=0.01%
  cpu          : usr=7.90%, sys=10.51%, ctx=102240, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18649: Tue Aug 17 01:43:22 2021
  write: IOPS=5760, BW=180MiB/s (189MB/s)(3072MiB/17065msec); 0 zone resets
    slat (usec): min=7, max=58999, avg=22.99, stdev=276.74
    clat (nsec): min=1145, max=1313.9M, avg=138556.61, stdev=7975102.99
     lat (usec): min=44, max=1313.9k, avg=161.81, stdev=7979.93
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   43], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   61], 60.00th=[   65],
     | 70.00th=[   70], 80.00th=[   81], 90.00th=[  105], 95.00th=[  135],
     | 99.00th=[  247], 99.50th=[  326], 99.90th=[ 2245], 99.95th=[ 7242],
     | 99.99th=[38011]
   bw (  KiB/s): min=  832, max=377792, per=30.30%, avg=223391.18, stdev=117789.27, samples=28
   iops        : min=   26, max=11806, avg=6980.93, stdev=3680.98, samples=28
  lat (usec)   : 2=0.64%, 4=1.41%, 10=0.02%, 20=0.06%, 50=17.36%
  lat (usec)   : 100=68.94%, 250=10.59%, 500=0.76%, 750=0.08%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.03%, 10=0.03%, 20=0.02%, 50=0.02%
  lat (msec)   : 100=0.01%, 1000=0.01%
  cpu          : usr=7.85%, sys=10.49%, ctx=102202, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=720MiB/s (755MB/s), 180MiB/s-183MiB/s (189MB/s-191MB/s), io=12.0GiB (12.9GB), run=16823-17065msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 17, 2021

Special Test 26: like Round 2 - Run 6 - Test X6 but fio run on host on 16K zvol

Fio test on host (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 2 (f=2): [R(2),_(2)][100.0%][r=313MiB/s][r=10.0k IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=13128: Tue Aug 17 01:58:59 2021
  read: IOPS=2733, BW=85.4MiB/s (89.6MB/s)(3072MiB/35960msec)
    slat (usec): min=5, max=144, avg=14.80, stdev= 3.98
    clat (usec): min=125, max=5753, avg=347.90, stdev=118.01
     lat (usec): min=133, max=5767, avg=363.06, stdev=118.20
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  306], 20.00th=[  314],
     | 30.00th=[  322], 40.00th=[  330], 50.00th=[  338], 60.00th=[  347],
     | 70.00th=[  355], 80.00th=[  367], 90.00th=[  388], 95.00th=[  412],
     | 99.00th=[  469], 99.50th=[  611], 99.90th=[ 2704], 99.95th=[ 2900],
     | 99.99th=[ 3228]
   bw (  KiB/s): min=83968, max=92544, per=25.09%, avg=87494.58, stdev=1689.55, samples=71
   iops        : min= 2624, max= 2892, avg=2734.20, stdev=52.80, samples=71
  lat (usec)   : 250=0.42%, 500=98.91%, 750=0.26%, 1000=0.13%
  lat (msec)   : 2=0.10%, 4=0.19%, 10=0.01%
  cpu          : usr=2.35%, sys=5.69%, ctx=98308, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=13129: Tue Aug 17 01:58:59 2021
  read: IOPS=2724, BW=85.1MiB/s (89.3MB/s)(3072MiB/36086msec)
    slat (usec): min=5, max=154, avg=14.82, stdev= 3.94
    clat (usec): min=122, max=6622, avg=349.15, stdev=130.50
     lat (usec): min=140, max=6649, avg=364.32, stdev=130.73
    clat percentiles (usec):
     |  1.00th=[  281],  5.00th=[  297], 10.00th=[  306], 20.00th=[  314],
     | 30.00th=[  322], 40.00th=[  330], 50.00th=[  338], 60.00th=[  347],
     | 70.00th=[  355], 80.00th=[  367], 90.00th=[  388], 95.00th=[  408],
     | 99.00th=[  469], 99.50th=[  635], 99.90th=[ 2737], 99.95th=[ 2966],
     | 99.99th=[ 3425]
   bw (  KiB/s): min=83584, max=91200, per=25.00%, avg=87175.42, stdev=1756.36, samples=71
   iops        : min= 2612, max= 2850, avg=2724.23, stdev=54.88, samples=71
  lat (usec)   : 250=0.26%, 500=99.07%, 750=0.23%, 1000=0.10%
  lat (msec)   : 2=0.11%, 4=0.23%, 10=0.01%
  cpu          : usr=2.19%, sys=5.83%, ctx=98309, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=13130: Tue Aug 17 01:58:59 2021
  read: IOPS=2745, BW=85.8MiB/s (89.0MB/s)(3072MiB/35807msec)
    slat (usec): min=5, max=126, avg=14.71, stdev= 3.90
    clat (usec): min=67, max=6191, avg=346.43, stdev=117.69
     lat (usec): min=77, max=6205, avg=361.49, stdev=117.96
    clat percentiles (usec):
     |  1.00th=[  269],  5.00th=[  293], 10.00th=[  306], 20.00th=[  314],
     | 30.00th=[  322], 40.00th=[  330], 50.00th=[  338], 60.00th=[  347],
     | 70.00th=[  355], 80.00th=[  367], 90.00th=[  388], 95.00th=[  408],
     | 99.00th=[  461], 99.50th=[  537], 99.90th=[ 2671], 99.95th=[ 2868],
     | 99.99th=[ 3130]
   bw (  KiB/s): min=84416, max=104640, per=25.20%, avg=87877.52, stdev=2720.46, samples=71
   iops        : min= 2638, max= 3270, avg=2746.11, stdev=85.04, samples=71
  lat (usec)   : 100=0.10%, 250=0.72%, 500=98.59%, 750=0.22%, 1000=0.10%
  lat (msec)   : 2=0.08%, 4=0.19%, 10=0.01%
  cpu          : usr=2.42%, sys=5.62%, ctx=98310, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=13131: Tue Aug 17 01:58:59 2021
  read: IOPS=2755, BW=86.1MiB/s (90.3MB/s)(3072MiB/35682msec)
    slat (usec): min=5, max=101, avg=14.77, stdev= 3.87
    clat (usec): min=81, max=7681, avg=345.10, stdev=112.22
     lat (usec): min=89, max=7695, avg=360.23, stdev=112.44
    clat percentiles (usec):
     |  1.00th=[  258],  5.00th=[  293], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  322], 40.00th=[  330], 50.00th=[  338], 60.00th=[  347],
     | 70.00th=[  355], 80.00th=[  367], 90.00th=[  388], 95.00th=[  408],
     | 99.00th=[  461], 99.50th=[  529], 99.90th=[ 2573], 99.95th=[ 2802],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=83840, max=97344, per=25.28%, avg=88163.11, stdev=2473.62, samples=71
   iops        : min= 2620, max= 3042, avg=2755.04, stdev=77.31, samples=71
  lat (usec)   : 100=0.05%, 250=0.91%, 500=98.45%, 750=0.24%, 1000=0.11%
  lat (msec)   : 2=0.08%, 4=0.15%, 10=0.01%
  cpu          : usr=2.39%, sys=5.69%, ctx=98312, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=341MiB/s (357MB/s), 85.1MiB/s-86.1MiB/s (89.3MB/s-90.3MB/s), io=12.0GiB (12.9GB), run=35682-36086msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1857 seconds or 30 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...25056 MiB / 26848 MiB / 25092 MiB / 128 MiB / 117 MiB ( 25056 MiB / 26848 MiB / 25092 MiB / 128 MiB / 117 MiB )
2 min passed...25056 MiB / 27136 MiB / 25092 MiB / 128 MiB / 117 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...25056 MiB / 27808 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 672 MiB / 6 MiB / 0 MiB / 0 MiB )
4 min passed...25056 MiB / 28256 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...25056 MiB / 28288 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...25056 MiB / 28416 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...25056 MiB / 28672 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...25056 MiB / 29120 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...25056 MiB / 29600 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...25056 MiB / 29632 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...25056 MiB / 29760 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...25056 MiB / 29952 MiB / 25098 MiB / 128 MiB / 117 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...25088 MiB / 30208 MiB / 25103 MiB / 128 MiB / 117 MiB ( 32 MiB / 256 MiB / 5 MiB / 0 MiB / 0 MiB )
14 min passed...25088 MiB / 30560 MiB / 25103 MiB / 128 MiB / 117 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...25088 MiB / 30592 MiB / 25103 MiB / 128 MiB / 117 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...25088 MiB / 30816 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 224 MiB / 0 MiB / 12224 MiB / 12238 MiB )
17 min passed...25088 MiB / 30912 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...25088 MiB / 30976 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...25088 MiB / 31424 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...25088 MiB / 31488 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...25088 MiB / 31648 MiB / 25103 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...25088 MiB / 31840 MiB / 25105 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
23 min passed...25088 MiB / 32064 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 224 MiB / 5 MiB / 0 MiB / 0 MiB )
24 min passed...25088 MiB / 32544 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...25088 MiB / 32672 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...25088 MiB / 32832 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...25088 MiB / 32992 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...25088 MiB / 33248 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...25088 MiB / 33696 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...25088 MiB / 33920 MiB / 25110 MiB / 12352 MiB / 12355 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 17, 2021

Special Test 27: like Round 2 - Run 6 - Test X7 but fio run on host on 16K zvol

Fio test on host (part 1):

Code:

root@Hypervisor:~# bash /root/scripts/bench/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 3 (f=3): [W(1),_(1),W(2)][100.0%][w=25.7MiB/s][w=822 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=2565: Tue Aug 17 02:36:36 2021
  write: IOPS=210, BW=6730KiB/s (6891kB/s)(512MiB/77905msec); 0 zone resets
    clat (usec): min=2186, max=55101, avg=4735.36, stdev=2201.07
     lat (usec): min=2187, max=55102, avg=4735.99, stdev=2201.08
    clat percentiles (usec):
     |  1.00th=[ 2900],  5.00th=[ 3097], 10.00th=[ 3228], 20.00th=[ 3687],
     | 30.00th=[ 4080], 40.00th=[ 4293], 50.00th=[ 4424], 60.00th=[ 4555],
     | 70.00th=[ 4752], 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 7111],
     | 99.00th=[12387], 99.50th=[20841], 99.90th=[28443], 99.95th=[42206],
     | 99.99th=[50594]
   bw (  KiB/s): min= 3712, max=10176, per=25.01%, avg=6730.56, stdev=1359.16, samples=155
   iops        : min=  116, max=  318, avg=210.32, stdev=42.48, samples=155
  lat (msec)   : 4=26.33%, 10=72.50%, 20=0.65%, 50=0.51%, 100=0.01%
  cpu          : usr=0.74%, sys=2.37%, ctx=64312, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=2566: Tue Aug 17 02:36:36 2021
  write: IOPS=210, BW=6732KiB/s (6894kB/s)(512MiB/77877msec); 0 zone resets
    clat (usec): min=1968, max=60141, avg=4733.56, stdev=2169.50
     lat (usec): min=1969, max=60142, avg=4734.20, stdev=2169.51
    clat percentiles (usec):
     |  1.00th=[ 2900],  5.00th=[ 3097], 10.00th=[ 3228], 20.00th=[ 3654],
     | 30.00th=[ 4080], 40.00th=[ 4293], 50.00th=[ 4424], 60.00th=[ 4555],
     | 70.00th=[ 4752], 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 7111],
     | 99.00th=[12256], 99.50th=[20579], 99.90th=[27395], 99.95th=[37487],
     | 99.99th=[50594]
   bw (  KiB/s): min= 3776, max=10176, per=25.02%, avg=6732.87, stdev=1355.78, samples=155
   iops        : min=  118, max=  318, avg=210.39, stdev=42.37, samples=155
  lat (msec)   : 2=0.01%, 4=26.38%, 10=72.47%, 20=0.60%, 50=0.52%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.90%, sys=2.28%, ctx=64300, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=2567: Tue Aug 17 02:36:36 2021
  write: IOPS=210, BW=6729KiB/s (6890kB/s)(512MiB/77919msec); 0 zone resets
    clat (usec): min=2589, max=50644, avg=4736.12, stdev=2188.05
     lat (usec): min=2589, max=50644, avg=4736.77, stdev=2188.06
    clat percentiles (usec):
     |  1.00th=[ 2900],  5.00th=[ 3097], 10.00th=[ 3228], 20.00th=[ 3687],
     | 30.00th=[ 4080], 40.00th=[ 4293], 50.00th=[ 4424], 60.00th=[ 4555],
     | 70.00th=[ 4752], 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 7111],
     | 99.00th=[12387], 99.50th=[20841], 99.90th=[27919], 99.95th=[40633],
     | 99.99th=[50594]
   bw (  KiB/s): min= 3776, max=10176, per=25.00%, avg=6729.15, stdev=1361.61, samples=155
   iops        : min=  118, max=  318, avg=210.27, stdev=42.55, samples=155
  lat (msec)   : 4=26.07%, 10=72.71%, 20=0.66%, 50=0.54%, 100=0.01%
  cpu          : usr=0.82%, sys=2.38%, ctx=64314, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=2568: Tue Aug 17 02:36:36 2021
  write: IOPS=210, BW=6728KiB/s (6890kB/s)(512MiB/77921msec); 0 zone resets
    clat (usec): min=2224, max=50823, avg=4736.10, stdev=2181.12
     lat (usec): min=2225, max=50825, avg=4736.78, stdev=2181.14
    clat percentiles (usec):
     |  1.00th=[ 2900],  5.00th=[ 3097], 10.00th=[ 3228], 20.00th=[ 3687],
     | 30.00th=[ 4080], 40.00th=[ 4228], 50.00th=[ 4424], 60.00th=[ 4555],
     | 70.00th=[ 4752], 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 7111],
     | 99.00th=[12387], 99.50th=[20579], 99.90th=[28181], 99.95th=[33162],
     | 99.99th=[50594]
   bw (  KiB/s): min= 3712, max=10176, per=25.00%, avg=6728.74, stdev=1359.74, samples=155
   iops        : min=  116, max=  318, avg=210.26, stdev=42.50, samples=155
  lat (msec)   : 4=26.12%, 10=72.70%, 20=0.65%, 50=0.51%, 100=0.02%
  cpu          : usr=0.79%, sys=2.38%, ctx=64285, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=26.3MiB/s (27.6MB/s), 6728KiB/s-6732KiB/s (6890kB/s-6894kB/s), io=2048MiB (2147MB), run=77877-77921msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 17, 2021

Special Test 27: like Round 2 - Run 6 - Test X7 but fio run on host on 16K zvol

Fio test on host (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=342MiB/s][r=10.0k IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=14789: Tue Aug 17 02:51:43 2021
  read: IOPS=2735, BW=85.5MiB/s (89.6MB/s)(512MiB/5990msec)
    clat (usec): min=215, max=3300, avg=363.54, stdev=162.01
     lat (usec): min=216, max=3300, avg=363.76, stdev=162.01
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  306], 10.00th=[  314], 20.00th=[  322],
     | 30.00th=[  330], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  400], 95.00th=[  424],
     | 99.00th=[  619], 99.50th=[ 1516], 99.90th=[ 2868], 99.95th=[ 2933],
     | 99.99th=[ 3130]
   bw (  KiB/s): min=85248, max=90048, per=24.96%, avg=87400.73, stdev=1640.77, samples=11
   iops        : min= 2664, max= 2814, avg=2731.27, stdev=51.27, samples=11
  lat (usec)   : 250=0.12%, 500=98.66%, 750=0.40%, 1000=0.20%
  lat (msec)   : 2=0.18%, 4=0.43%
  cpu          : usr=1.37%, sys=4.76%, ctx=16387, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=14790: Tue Aug 17 02:51:43 2021
  read: IOPS=2743, BW=85.7MiB/s (89.9MB/s)(512MiB/5973msec)
    clat (usec): min=229, max=3270, avg=362.46, stdev=157.47
     lat (usec): min=229, max=3270, avg=362.68, stdev=157.48
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  306], 10.00th=[  314], 20.00th=[  322],
     | 30.00th=[  330], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  400], 95.00th=[  420],
     | 99.00th=[  506], 99.50th=[ 1418], 99.90th=[ 2802], 99.95th=[ 2933],
     | 99.99th=[ 3195]
   bw (  KiB/s): min=85184, max=89536, per=25.09%, avg=87848.73, stdev=1192.28, samples=11
   iops        : min= 2662, max= 2798, avg=2745.27, stdev=37.26, samples=11
  lat (usec)   : 250=0.04%, 500=98.92%, 750=0.31%, 1000=0.15%
  lat (msec)   : 2=0.18%, 4=0.41%
  cpu          : usr=2.21%, sys=3.94%, ctx=16387, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=14791: Tue Aug 17 02:51:43 2021
  read: IOPS=2755, BW=86.1MiB/s (90.3MB/s)(512MiB/5946msec)
    clat (usec): min=219, max=3286, avg=360.84, stdev=146.53
     lat (usec): min=219, max=3286, avg=361.06, stdev=146.53
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  306], 10.00th=[  314], 20.00th=[  322],
     | 30.00th=[  330], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  400], 95.00th=[  424],
     | 99.00th=[  529], 99.50th=[  955], 99.90th=[ 2835], 99.95th=[ 2933],
     | 99.99th=[ 3097]
   bw (  KiB/s): min=86784, max=89728, per=25.18%, avg=88162.91, stdev=857.87, samples=11
   iops        : min= 2712, max= 2804, avg=2755.09, stdev=26.81, samples=11
  lat (usec)   : 250=0.05%, 500=98.80%, 750=0.46%, 1000=0.24%
  lat (msec)   : 2=0.13%, 4=0.31%
  cpu          : usr=1.78%, sys=4.42%, ctx=16388, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=14792: Tue Aug 17 02:51:43 2021
  read: IOPS=2744, BW=85.8MiB/s (89.9MB/s)(512MiB/5969msec)
    clat (usec): min=236, max=3357, avg=362.26, stdev=151.86
     lat (usec): min=236, max=3357, avg=362.49, stdev=151.86
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  306], 10.00th=[  314], 20.00th=[  322],
     | 30.00th=[  330], 40.00th=[  338], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  400], 95.00th=[  420],
     | 99.00th=[  523], 99.50th=[ 1057], 99.90th=[ 2835], 99.95th=[ 2933],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=85440, max=90304, per=25.12%, avg=87930.18, stdev=1609.56, samples=11
   iops        : min= 2670, max= 2822, avg=2747.82, stdev=50.30, samples=11
  lat (usec)   : 250=0.03%, 500=98.90%, 750=0.38%, 1000=0.17%
  lat (msec)   : 2=0.16%, 4=0.35%
  cpu          : usr=1.47%, sys=4.66%, ctx=16387, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=342MiB/s (359MB/s), 85.5MiB/s-86.1MiB/s (89.6MB/s-90.3MB/s), io=2048MiB (2147MB), run=5946-5990msec

Disk stats (read/write):
  zd32: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1889 seconds or 31 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
1 min passed...14656 MiB / 15552 MiB / 14757 MiB / 160 MiB / 153 MiB ( 14656 MiB / 15552 MiB / 14756 MiB / 160 MiB / 153 MiB )
2 min passed...20992 MiB / 22592 MiB / 20999 MiB / 288 MiB / 238 MiB ( 6336 MiB / 7040 MiB / 6242 MiB / 128 MiB / 85 MiB )
3 min passed...20992 MiB / 23136 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...20992 MiB / 23488 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...20992 MiB / 23552 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...20992 MiB / 23552 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...20992 MiB / 23584 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...20992 MiB / 24288 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 704 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...20992 MiB / 24800 MiB / 20999 MiB / 288 MiB / 238 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...20992 MiB / 24992 MiB / 21001 MiB / 288 MiB / 238 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
11 min passed...20992 MiB / 24992 MiB / 21005 MiB / 288 MiB / 238 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
12 min passed...20992 MiB / 25312 MiB / 21005 MiB / 288 MiB / 238 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...20992 MiB / 25888 MiB / 21005 MiB / 288 MiB / 238 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...20992 MiB / 26464 MiB / 21005 MiB / 288 MiB / 238 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...20992 MiB / 26784 MiB / 21007 MiB / 288 MiB / 238 MiB ( 0 MiB / 320 MiB / 2 MiB / 0 MiB / 0 MiB )
16 min passed...20992 MiB / 26784 MiB / 21009 MiB / 288 MiB / 238 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
17 min passed...20992 MiB / 27136 MiB / 21009 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 352 MiB / 0 MiB / 1984 MiB / 2048 MiB )
18 min passed...20992 MiB / 27648 MiB / 21009 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...20992 MiB / 28192 MiB / 21009 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...20992 MiB / 28512 MiB / 21010 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 320 MiB / 1 MiB / 0 MiB / 0 MiB )
21 min passed...20992 MiB / 28640 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 128 MiB / 5 MiB / 0 MiB / 0 MiB )
22 min passed...20992 MiB / 29024 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...20992 MiB / 29376 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...20992 MiB / 29824 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...20992 MiB / 30016 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...20992 MiB / 30208 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...20992 MiB / 30592 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...20992 MiB / 30944 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...20992 MiB / 31296 MiB / 21015 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...20992 MiB / 31680 MiB / 21016 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 384 MiB / 1 MiB / 0 MiB / 0 MiB )
31 min passed...20992 MiB / 31968 MiB / 21021 MiB / 2272 MiB / 2286 MiB ( 0 MiB / 288 MiB / 5 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Special Test 28: like Round 2 - Run 7 - Test X6 but fio run on host on 8K zvol

Fio test on host (part 1):

Code:

root@Hypervisor:~# bash /root/scripts/bench/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=4): [W(4)][94.7%][w=841MiB/s][w=26.9k IOPS][eta 00m:01s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18939: Wed Aug 18 03:27:15 2021
  write: IOPS=5335, BW=167MiB/s (175MB/s)(3072MiB/18424msec); 0 zone resets
    slat (usec): min=7, max=42049, avg=22.58, stdev=171.19
    clat (nsec): min=1120, max=1377.1M, avg=152940.70, stdev=6428423.69
     lat (usec): min=52, max=1377.2k, avg=175.85, stdev=6430.97
    clat percentiles (nsec):
     |  1.00th=[    1848],  5.00th=[   50944], 10.00th=[   55040],
     | 20.00th=[   60672], 30.00th=[   65280], 40.00th=[   70144],
     | 50.00th=[   76288], 60.00th=[   83456], 70.00th=[  102912],
     | 80.00th=[  134144], 90.00th=[  183296], 95.00th=[  216064],
     | 99.00th=[  428032], 99.50th=[  651264], 99.90th=[ 5013504],
     | 99.95th=[ 9109504], 99.99th=[25559040]
   bw (  KiB/s): min= 9920, max=324288, per=27.02%, avg=183406.85, stdev=88111.70, samples=33
   iops        : min=  310, max=10134, avg=5731.39, stdev=2753.52, samples=33
  lat (usec)   : 2=1.44%, 4=0.87%, 10=0.03%, 20=0.09%, 50=1.70%
  lat (usec)   : 100=64.85%, 250=27.75%, 500=2.52%, 750=0.31%, 1000=0.09%
  lat (msec)   : 2=0.13%, 4=0.09%, 10=0.08%, 20=0.03%, 50=0.01%
  lat (msec)   : 500=0.01%, 750=0.01%
  cpu          : usr=7.94%, sys=8.37%, ctx=106210, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18940: Wed Aug 18 03:27:15 2021
  write: IOPS=5302, BW=166MiB/s (174MB/s)(3072MiB/18540msec); 0 zone resets
    slat (usec): min=6, max=10581, avg=21.68, stdev=81.89
    clat (nsec): min=1165, max=1377.0M, avg=154013.32, stdev=6430573.76
     lat (usec): min=52, max=1377.0k, avg=176.01, stdev=6431.06
    clat percentiles (nsec):
     |  1.00th=[    1864],  5.00th=[   51456], 10.00th=[   55552],
     | 20.00th=[   60672], 30.00th=[   65280], 40.00th=[   70144],
     | 50.00th=[   76288], 60.00th=[   83456], 70.00th=[  102912],
     | 80.00th=[  134144], 90.00th=[  185344], 95.00th=[  216064],
     | 99.00th=[  428032], 99.50th=[  667648], 99.90th=[ 5341184],
     | 99.95th=[ 9371648], 99.99th=[33161216]
   bw (  KiB/s): min=10304, max=324480, per=27.14%, avg=184216.88, stdev=88322.69, samples=34
   iops        : min=  322, max=10140, avg=5756.74, stdev=2760.09, samples=34
  lat (usec)   : 2=1.46%, 4=0.85%, 10=0.03%, 20=0.09%, 50=1.58%
  lat (usec)   : 100=64.82%, 250=27.87%, 500=2.56%, 750=0.32%, 1000=0.09%
  lat (msec)   : 2=0.13%, 4=0.09%, 10=0.09%, 20=0.03%, 50=0.01%
  lat (msec)   : 500=0.01%, 750=0.01%
  cpu          : usr=7.08%, sys=9.11%, ctx=106056, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18941: Wed Aug 18 03:27:15 2021
  write: IOPS=5303, BW=166MiB/s (174MB/s)(3072MiB/18535msec); 0 zone resets
    slat (usec): min=6, max=30163, avg=22.56, stdev=160.25
    clat (nsec): min=1097, max=1377.1M, avg=154101.91, stdev=6431652.39
     lat (usec): min=51, max=1377.1k, avg=176.98, stdev=6433.64
    clat percentiles (nsec):
     |  1.00th=[    1864],  5.00th=[   51456], 10.00th=[   55552],
     | 20.00th=[   60672], 30.00th=[   66048], 40.00th=[   70144],
     | 50.00th=[   76288], 60.00th=[   83456], 70.00th=[  101888],
     | 80.00th=[  134144], 90.00th=[  183296], 95.00th=[  218112],
     | 99.00th=[  432128], 99.50th=[  684032], 99.90th=[ 4947968],
     | 99.95th=[ 9633792], 99.99th=[34865152]
   bw (  KiB/s): min=10112, max=328768, per=27.16%, avg=184339.12, stdev=88544.61, samples=34
   iops        : min=  316, max=10274, avg=5760.56, stdev=2767.04, samples=34
  lat (usec)   : 2=1.43%, 4=0.83%, 10=0.02%, 20=0.08%, 50=1.59%
  lat (usec)   : 100=65.15%, 250=27.57%, 500=2.53%, 750=0.35%, 1000=0.09%
  lat (msec)   : 2=0.14%, 4=0.09%, 10=0.08%, 20=0.03%, 50=0.02%
  lat (msec)   : 500=0.01%, 750=0.01%
  cpu          : usr=7.25%, sys=8.92%, ctx=105946, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=18942: Wed Aug 18 03:27:15 2021
  write: IOPS=5302, BW=166MiB/s (174MB/s)(3072MiB/18539msec); 0 zone resets
    slat (usec): min=7, max=11300, avg=21.73, stdev=85.33
    clat (nsec): min=1121, max=1377.4M, avg=154585.62, stdev=6433023.91
     lat (usec): min=51, max=1377.5k, avg=176.63, stdev=6433.63
    clat percentiles (nsec):
     |  1.00th=[    1848],  5.00th=[   51456], 10.00th=[   55552],
     | 20.00th=[   60672], 30.00th=[   65280], 40.00th=[   70144],
     | 50.00th=[   76288], 60.00th=[   83456], 70.00th=[  102912],
     | 80.00th=[  134144], 90.00th=[  183296], 95.00th=[  216064],
     | 99.00th=[  428032], 99.50th=[  684032], 99.90th=[ 5013504],
     | 99.95th=[10289152], 99.99th=[34340864]
   bw (  KiB/s): min= 9920, max=324416, per=27.15%, avg=184262.26, stdev=86529.32, samples=34
   iops        : min=  310, max=10138, avg=5758.15, stdev=2704.07, samples=34
  lat (usec)   : 2=1.49%, 4=0.75%, 10=0.02%, 20=0.06%, 50=1.53%
  lat (usec)   : 100=65.20%, 250=27.64%, 500=2.54%, 750=0.31%, 1000=0.09%
  lat (msec)   : 2=0.14%, 4=0.10%, 10=0.08%, 20=0.03%, 50=0.02%
  lat (msec)   : 100=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=6.79%, sys=9.40%, ctx=105871, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=663MiB/s (695MB/s), 166MiB/s-167MiB/s (174MB/s-175MB/s), io=12.0GiB (12.9GB), run=18424-18540msec

Disk stats (read/write):
  zd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 18, 2021

Special Test 28: like Round 2 - Run 7 - Test X6 but fio run on host on 8K zvol

Fio test on host (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=292MiB/s][r=9328 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=21164: Wed Aug 18 03:42:56 2021
  read: IOPS=2431, BW=75.0MiB/s (79.7MB/s)(3072MiB/40431msec)
    slat (usec): min=5, max=142, avg=13.97, stdev= 4.11
    clat (usec): min=117, max=8185, avg=394.61, stdev=314.68
     lat (usec): min=126, max=8201, avg=408.89, stdev=314.82
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  343], 60.00th=[  355],
     | 70.00th=[  367], 80.00th=[  379], 90.00th=[  408], 95.00th=[  449],
     | 99.00th=[ 2507], 99.50th=[ 2769], 99.90th=[ 3130], 99.95th=[ 3556],
     | 99.99th=[ 5735]
   bw (  KiB/s): min=70464, max=86400, per=25.05%, avg=77818.11, stdev=3050.36, samples=80
   iops        : min= 2202, max= 2700, avg=2431.81, stdev=95.33, samples=80
  lat (usec)   : 250=0.09%, 500=95.95%, 750=1.00%, 1000=0.73%
  lat (msec)   : 2=0.76%, 4=1.43%, 10=0.05%
  cpu          : usr=1.61%, sys=4.95%, ctx=98316, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=21165: Wed Aug 18 03:42:56 2021
  read: IOPS=2427, BW=75.9MiB/s (79.5MB/s)(3072MiB/40500msec)
    slat (usec): min=5, max=125, avg=13.95, stdev= 4.08
    clat (usec): min=86, max=7837, avg=395.33, stdev=316.95
     lat (usec): min=95, max=7851, avg=409.59, stdev=317.13
    clat percentiles (usec):
     |  1.00th=[  269],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  343], 60.00th=[  355],
     | 70.00th=[  367], 80.00th=[  379], 90.00th=[  408], 95.00th=[  449],
     | 99.00th=[ 2540], 99.50th=[ 2802], 99.90th=[ 3130], 99.95th=[ 3359],
     | 99.99th=[ 5538]
   bw (  KiB/s): min=70080, max=85056, per=25.01%, avg=77691.29, stdev=3528.43, samples=80
   iops        : min= 2190, max= 2658, avg=2427.84, stdev=110.26, samples=80
  lat (usec)   : 100=0.01%, 250=0.51%, 500=95.53%, 750=0.94%, 1000=0.69%
  lat (msec)   : 2=0.75%, 4=1.53%, 10=0.03%
  cpu          : usr=1.84%, sys=4.71%, ctx=98318, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=21166: Wed Aug 18 03:42:56 2021
  read: IOPS=2445, BW=76.4MiB/s (80.1MB/s)(3072MiB/40202msec)
    slat (usec): min=5, max=104, avg=13.88, stdev= 4.05
    clat (usec): min=86, max=5800, avg=392.38, stdev=309.72
     lat (usec): min=95, max=5813, avg=406.57, stdev=309.88
    clat percentiles (usec):
     |  1.00th=[  265],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  343], 60.00th=[  355],
     | 70.00th=[  367], 80.00th=[  379], 90.00th=[  408], 95.00th=[  445],
     | 99.00th=[ 2474], 99.50th=[ 2769], 99.90th=[ 3130], 99.95th=[ 3458],
     | 99.99th=[ 5473]
   bw (  KiB/s): min=70592, max=88128, per=25.19%, avg=78252.04, stdev=3481.80, samples=80
   iops        : min= 2206, max= 2754, avg=2445.40, stdev=108.81, samples=80
  lat (usec)   : 100=0.03%, 250=0.53%, 500=95.71%, 750=0.89%, 1000=0.66%
  lat (msec)   : 2=0.71%, 4=1.43%, 10=0.04%
  cpu          : usr=1.92%, sys=4.66%, ctx=98311, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=21167: Wed Aug 18 03:42:56 2021
  read: IOPS=2445, BW=76.4MiB/s (80.1MB/s)(3072MiB/40204msec)
    slat (nsec): min=5671, max=98199, avg=13905.20, stdev=4089.07
    clat (usec): min=90, max=7886, avg=392.38, stdev=309.56
     lat (usec): min=98, max=7898, avg=406.59, stdev=309.74
    clat percentiles (usec):
     |  1.00th=[  269],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  343], 60.00th=[  355],
     | 70.00th=[  367], 80.00th=[  379], 90.00th=[  408], 95.00th=[  445],
     | 99.00th=[ 2474], 99.50th=[ 2802], 99.90th=[ 3163], 99.95th=[ 3490],
     | 99.99th=[ 5604]
   bw (  KiB/s): min=69952, max=88896, per=25.19%, avg=78260.14, stdev=3647.71, samples=80
   iops        : min= 2186, max= 2778, avg=2445.62, stdev=113.99, samples=80
  lat (usec)   : 100=0.02%, 250=0.41%, 500=95.75%, 750=0.95%, 1000=0.71%
  lat (msec)   : 2=0.73%, 4=1.39%, 10=0.04%
  cpu          : usr=1.83%, sys=4.74%, ctx=98311, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=303MiB/s (318MB/s), 75.9MiB/s-76.4MiB/s (79.5MB/s-80.1MB/s), io=12.0GiB (12.9GB), run=40202-40500msec

Disk stats (read/write):
  zd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1864 seconds or 31 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...25344 MiB / 26912 MiB / 25302 MiB / 160 MiB / 117 MiB ( 25344 MiB / 26912 MiB / 25302 MiB / 160 MiB / 117 MiB )
2 min passed...25344 MiB / 26976 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...25344 MiB / 27040 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...25344 MiB / 27200 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...25344 MiB / 28160 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 960 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...25344 MiB / 28480 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...25344 MiB / 28480 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...25344 MiB / 28736 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...25344 MiB / 28800 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...25344 MiB / 29248 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...25344 MiB / 29856 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...25344 MiB / 29952 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...25344 MiB / 30080 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...25344 MiB / 30240 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...25344 MiB / 30272 MiB / 25302 MiB / 160 MiB / 117 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...25344 MiB / 31232 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 960 MiB / 0 MiB / 12224 MiB / 12246 MiB )
17 min passed...25344 MiB / 31520 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...25344 MiB / 31520 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...25344 MiB / 31584 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...25344 MiB / 31584 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...25344 MiB / 31872 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...25344 MiB / 33024 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 1152 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...25344 MiB / 33216 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...25344 MiB / 33344 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...25344 MiB / 33408 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...25344 MiB / 33408 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...25344 MiB / 34112 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 704 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...25344 MiB / 34400 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...25344 MiB / 34400 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...25344 MiB / 34688 MiB / 25302 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...25344 MiB / 34720 MiB / 25312 MiB / 12384 MiB / 12363 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Special Test 29: like Round 2 - Run 7 - Test X7 but fio run on host on 8K zvol

Fio test on host (part 1):

Code:

root@Hypervisor:~# bash /root/scripts/bench/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=21.1MiB/s][w=676 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=21638: Wed Aug 18 16:12:24 2021
  write: IOPS=161, BW=5156KiB/s (5280kB/s)(512MiB/101680msec); 0 zone resets
    clat (usec): min=2658, max=80895, avg=6185.72, stdev=2598.63
     lat (usec): min=2658, max=80896, avg=6186.38, stdev=2598.79
    clat percentiles (usec):
     |  1.00th=[ 4293],  5.00th=[ 4686], 10.00th=[ 4883], 20.00th=[ 5080],
     | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5604], 60.00th=[ 5800],
     | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 7898], 95.00th=[ 8356],
     | 99.00th=[11731], 99.50th=[24773], 99.90th=[35390], 99.95th=[46400],
     | 99.99th=[69731]
   bw (  KiB/s): min= 3392, max= 5952, per=25.03%, avg=5154.88, stdev=495.56, samples=203
   iops        : min=  106, max=  186, avg=161.08, stdev=15.47, samples=203
  lat (msec)   : 4=0.15%, 10=98.08%, 20=1.01%, 50=0.71%, 100=0.04%
  cpu          : usr=0.66%, sys=1.93%, ctx=65593, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=21639: Wed Aug 18 16:12:24 2021
  write: IOPS=161, BW=5156KiB/s (5280kB/s)(512MiB/101680msec); 0 zone resets
    clat (usec): min=3498, max=70660, avg=6186.20, stdev=2544.49
     lat (usec): min=3499, max=70660, avg=6186.83, stdev=2544.50
    clat percentiles (usec):
     |  1.00th=[ 4293],  5.00th=[ 4686], 10.00th=[ 4883], 20.00th=[ 5080],
     | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5604], 60.00th=[ 5800],
     | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 7898], 95.00th=[ 8455],
     | 99.00th=[11863], 99.50th=[25822], 99.90th=[35390], 99.95th=[44303],
     | 99.99th=[68682]
   bw (  KiB/s): min= 3392, max= 6016, per=25.03%, avg=5155.01, stdev=493.93, samples=203
   iops        : min=  106, max=  188, avg=161.07, stdev=15.43, samples=203
  lat (msec)   : 4=0.18%, 10=97.88%, 20=1.17%, 50=0.73%, 100=0.03%
  cpu          : usr=0.64%, sys=1.93%, ctx=65609, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=21640: Wed Aug 18 16:12:24 2021
  write: IOPS=160, BW=5150KiB/s (5274kB/s)(512MiB/101799msec); 0 zone resets
    clat (usec): min=2333, max=80953, avg=6193.48, stdev=2651.05
     lat (usec): min=2333, max=80954, avg=6194.13, stdev=2651.07
    clat percentiles (usec):
     |  1.00th=[ 4359],  5.00th=[ 4686], 10.00th=[ 4883], 20.00th=[ 5080],
     | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5604], 60.00th=[ 5800],
     | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 7898], 95.00th=[ 8356],
     | 99.00th=[12125], 99.50th=[25560], 99.90th=[38536], 99.95th=[47449],
     | 99.99th=[69731]
   bw (  KiB/s): min= 3392, max= 6016, per=25.03%, avg=5153.75, stdev=491.42, samples=203
   iops        : min=  106, max=  188, avg=161.03, stdev=15.35, samples=203
  lat (msec)   : 4=0.15%, 10=98.05%, 20=1.02%, 50=0.75%, 100=0.03%
  cpu          : usr=0.65%, sys=1.93%, ctx=65675, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=21641: Wed Aug 18 16:12:24 2021
  write: IOPS=160, BW=5149KiB/s (5272kB/s)(512MiB/101829msec); 0 zone resets
    clat (usec): min=3632, max=91799, avg=6195.36, stdev=2679.93
     lat (usec): min=3632, max=91800, avg=6196.04, stdev=2679.95
    clat percentiles (usec):
     |  1.00th=[ 4293],  5.00th=[ 4686], 10.00th=[ 4883], 20.00th=[ 5080],
     | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5604], 60.00th=[ 5800],
     | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 7898], 95.00th=[ 8455],
     | 99.00th=[12125], 99.50th=[25297], 99.90th=[38011], 99.95th=[47449],
     | 99.99th=[70779]
   bw (  KiB/s): min= 3392, max= 6016, per=25.02%, avg=5153.19, stdev=492.10, samples=203
   iops        : min=  106, max=  188, avg=161.01, stdev=15.39, samples=203
  lat (msec)   : 4=0.15%, 10=97.97%, 20=1.10%, 50=0.73%, 100=0.05%
  cpu          : usr=0.66%, sys=1.93%, ctx=65623, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=20.1MiB/s (21.1MB/s), 5149KiB/s-5156KiB/s (5272kB/s-5280kB/s), io=2048MiB (2147MB), run=101680-101829msec

Disk stats (read/write):
  zd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Dunuin · Aug 18, 2021

Special Test 29: like Round 2 - Run 7 - Test X7 but fio run on host on 8K zvol

Fio test on host (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=343MiB/s][r=10.0k IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11024: Wed Aug 18 16:27:31 2021
  read: IOPS=2766, BW=86.5MiB/s (90.7MB/s)(512MiB/5922msec)
    clat (usec): min=236, max=8202, avg=359.34, stdev=174.01
     lat (usec): min=237, max=8203, avg=359.55, stdev=174.02
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  343], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  396], 95.00th=[  416],
     | 99.00th=[  537], 99.50th=[ 1582], 99.90th=[ 2835], 99.95th=[ 2900],
     | 99.99th=[ 3851]
   bw (  KiB/s): min=85504, max=94336, per=25.27%, avg=88599.27, stdev=2832.57, samples=11
   iops        : min= 2672, max= 2948, avg=2768.73, stdev=88.52, samples=11
  lat (usec)   : 250=0.07%, 500=98.79%, 750=0.33%, 1000=0.16%
  lat (msec)   : 2=0.23%, 4=0.41%, 10=0.01%
  cpu          : usr=1.60%, sys=4.31%, ctx=16386, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11025: Wed Aug 18 16:27:31 2021
  read: IOPS=2740, BW=85.6MiB/s (89.8MB/s)(512MiB/5978msec)
    clat (usec): min=232, max=8247, avg=362.78, stdev=186.61
     lat (usec): min=232, max=8248, avg=363.00, stdev=186.61
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  367], 80.00th=[  379], 90.00th=[  400], 95.00th=[  420],
     | 99.00th=[  709], 99.50th=[ 1958], 99.90th=[ 2900], 99.95th=[ 3032],
     | 99.99th=[ 3195]
   bw (  KiB/s): min=83840, max=91968, per=24.97%, avg=87548.45, stdev=2293.09, samples=11
   iops        : min= 2620, max= 2874, avg=2735.82, stdev=71.60, samples=11
  lat (usec)   : 250=0.06%, 500=98.62%, 750=0.36%, 1000=0.21%
  lat (msec)   : 2=0.26%, 4=0.48%, 10=0.01%
  cpu          : usr=2.18%, sys=3.68%, ctx=16386, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11026: Wed Aug 18 16:27:31 2021
  read: IOPS=2743, BW=85.7MiB/s (89.9MB/s)(512MiB/5972msec)
    clat (usec): min=238, max=8220, avg=362.39, stdev=181.03
     lat (usec): min=238, max=8220, avg=362.60, stdev=181.03
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  293], 10.00th=[  302], 20.00th=[  318],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  379], 90.00th=[  400], 95.00th=[  420],
     | 99.00th=[  668], 99.50th=[ 1696], 99.90th=[ 2900], 99.95th=[ 2999],
     | 99.99th=[ 3195]
   bw (  KiB/s): min=82816, max=93189, per=25.03%, avg=87734.73, stdev=3082.71, samples=11
   iops        : min= 2588, max= 2912, avg=2741.64, stdev=96.36, samples=11
  lat (usec)   : 250=0.04%, 500=98.66%, 750=0.38%, 1000=0.24%
  lat (msec)   : 2=0.23%, 4=0.45%, 10=0.01%
  cpu          : usr=1.46%, sys=4.40%, ctx=16389, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=11027: Wed Aug 18 16:27:31 2021
  read: IOPS=2738, BW=85.6MiB/s (89.7MB/s)(512MiB/5982msec)
    clat (usec): min=225, max=8359, avg=363.01, stdev=192.70
     lat (usec): min=225, max=8360, avg=363.24, stdev=192.72
    clat percentiles (usec):
     |  1.00th=[  273],  5.00th=[  289], 10.00th=[  302], 20.00th=[  314],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  347], 60.00th=[  355],
     | 70.00th=[  363], 80.00th=[  375], 90.00th=[  400], 95.00th=[  416],
     | 99.00th=[  734], 99.50th=[ 2212], 99.90th=[ 2900], 99.95th=[ 3064],
     | 99.99th=[ 3785]
   bw (  KiB/s): min=84544, max=91712, per=24.99%, avg=87618.73, stdev=2320.83, samples=11
   iops        : min= 2642, max= 2866, avg=2738.00, stdev=72.53, samples=11
  lat (usec)   : 250=0.07%, 500=98.61%, 750=0.38%, 1000=0.20%
  lat (msec)   : 2=0.16%, 4=0.57%, 10=0.01%
  cpu          : usr=1.14%, sys=4.73%, ctx=16389, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=342MiB/s (359MB/s), 85.6MiB/s-86.5MiB/s (89.7MB/s-90.7MB/s), io=2048MiB (2147MB), run=5922-5982msec

Disk stats (read/write):
  zd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Script ran 1912 seconds or 31 minutes.

Iostat/SMART logging on host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...12128 MiB / 13440 MiB / 12152 MiB / 352 MiB / 350 MiB ( 12128 MiB / 13440 MiB / 12152 MiB / 352 MiB / 350 MiB )
2 min passed...21472 MiB / 23360 MiB / 21437 MiB / 640 MiB / 610 MiB ( 9344 MiB / 9920 MiB / 9285 MiB / 288 MiB / 260 MiB )
3 min passed...21472 MiB / 23360 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...21472 MiB / 24032 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...21472 MiB / 24640 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...21472 MiB / 24896 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...21472 MiB / 25280 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...21472 MiB / 25440 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...21472 MiB / 25664 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...21472 MiB / 26336 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...21472 MiB / 26624 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...21472 MiB / 27040 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...21472 MiB / 27392 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...21472 MiB / 27456 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...21472 MiB / 28032 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...21472 MiB / 28480 MiB / 21439 MiB / 640 MiB / 610 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...21472 MiB / 28832 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 352 MiB / 0 MiB / 2016 MiB / 2048 MiB )
18 min passed...21472 MiB / 29184 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...21472 MiB / 29472 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...21472 MiB / 29664 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...21472 MiB / 30304 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...21472 MiB / 30560 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...21472 MiB / 30944 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...21472 MiB / 31264 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...21472 MiB / 31488 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...21472 MiB / 31904 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...21472 MiB / 32352 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...21472 MiB / 32640 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...21472 MiB / 33152 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...21472 MiB / 33376 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...21472 MiB / 33728 MiB / 21439 MiB / 2656 MiB / 2658 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

This is the pool and VM that will be used in Round 2 Run 7:

Host is a striped mirror of 6x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=8K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 6 are bold)

It is the same pool as run 6 just that the VM uses 8K volblocksize. I also ran Test TX6 and TX7 directly on the host in a zvol with 8K volblocksize.

For detailed info see run 6.

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1229KiB/s][w=307 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=753: Wed Aug 18 17:07:53 2021
  write: IOPS=320, BW=1283KiB/s (1314kB/s)(1024MiB/817218msec); 0 zone resets
    clat (usec): min=1222, max=51938, avg=3102.41, stdev=1024.31
     lat (usec): min=1223, max=51940, avg=3104.59, stdev=1024.32
    clat percentiles (usec):
     |  1.00th=[ 2024],  5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2573],
     | 30.00th=[ 2704], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3130],
     | 70.00th=[ 3294], 80.00th=[ 3490], 90.00th=[ 3720], 95.00th=[ 3982],
     | 99.00th=[ 5866], 99.50th=[ 6390], 99.90th=[15139], 99.95th=[21103],
     | 99.99th=[35390]
   bw (  KiB/s): min=  984, max= 1752, per=100.00%, avg=1283.02, stdev=111.53, samples=1634
   iops        : min=  246, max=  438, avg=320.75, stdev=27.88, samples=1634
  lat (msec)   : 2=0.86%, 4=94.46%, 10=4.46%, 20=0.17%, 50=0.05%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.74%, sys=4.59%, ctx=618061, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1283KiB/s (1314kB/s), 1283KiB/s-1283KiB/s (1314kB/s-1314kB/s), io=1024MiB (1074MB), run=817218-817218msec

Disk stats (read/write):
  sda: ios=1/798231, merge=0/549211, ticks=0/764740, in_queue=819876, util=99.72%
0 min passed...Guest Writes/Reads (Changes): 5258 MiB / 17 MiB ( 5258 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 5259 MiB / 17 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5259 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5259 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5259 MiB / 17 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5259 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5260 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5261 MiB / 17 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5261 MiB / 17 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5261 MiB / 17 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5261 MiB / 17 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=10.3MiB/s][r=2629 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=967: Wed Aug 18 17:24:59 2021
  read: IOPS=2095, BW=8380KiB/s (8582kB/s)(1024MiB/125121msec)
    clat (usec): min=207, max=6384, avg=466.68, stdev=165.28
     lat (usec): min=209, max=6386, avg=468.59, stdev=165.35
    clat percentiles (usec):
     |  1.00th=[  281],  5.00th=[  310], 10.00th=[  330], 20.00th=[  363],
     | 30.00th=[  392], 40.00th=[  420], 50.00th=[  449], 60.00th=[  482],
     | 70.00th=[  515], 80.00th=[  562], 90.00th=[  603], 95.00th=[  635],
     | 99.00th=[  685], 99.50th=[ 1074], 99.90th=[ 2835], 99.95th=[ 3064],
     | 99.99th=[ 3392]
   bw (  KiB/s): min= 6496, max=12408, per=99.97%, avg=8377.76, stdev=1218.12, samples=250
   iops        : min= 1624, max= 3102, avg=2094.43, stdev=304.53, samples=250
  lat (usec)   : 250=0.07%, 500=65.52%, 750=33.75%, 1000=0.13%
  lat (msec)   : 2=0.25%, 4=0.28%, 10=0.01%
  cpu          : usr=3.75%, sys=10.48%, ctx=262253, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=8380KiB/s (8582kB/s), 8380KiB/s-8380KiB/s (8582kB/s-8582kB/s), io=1024MiB (1074MB), run=125121-125121msec

Disk stats (read/write):
  sda: ios=261572/33, merge=0/42, ticks=112701/36, in_queue=124860, util=99.93%
0 min passed...Guest Writes/Reads (Changes): 5261 MiB / 1041 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5261 MiB / 1041 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5262 MiB / 1041 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5263 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Script ran 2744 seconds or 45 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3584 MiB / 4256 MiB / 3642 MiB / 192 MiB / 211 MiB ( 3584 MiB / 4256 MiB / 3642 MiB / 192 MiB / 211 MiB )
2 min passed...7552 MiB / 8736 MiB / 7590 MiB / 416 MiB / 453 MiB ( 3968 MiB / 4480 MiB / 3948 MiB / 224 MiB / 242 MiB )
3 min passed...11360 MiB / 13152 MiB / 11394 MiB / 704 MiB / 730 MiB ( 3808 MiB / 4416 MiB / 3804 MiB / 288 MiB / 277 MiB )
4 min passed...15200 MiB / 17568 MiB / 15235 MiB / 960 MiB / 1034 MiB ( 3840 MiB / 4416 MiB / 3841 MiB / 256 MiB / 304 MiB )
5 min passed...19072 MiB / 22048 MiB / 19103 MiB / 1312 MiB / 1357 MiB ( 3872 MiB / 4480 MiB / 3868 MiB / 352 MiB / 323 MiB )
6 min passed...22752 MiB / 26560 MiB / 22767 MiB / 1696 MiB / 1691 MiB ( 3680 MiB / 4512 MiB / 3664 MiB / 384 MiB / 334 MiB )
7 min passed...26496 MiB / 30528 MiB / 26513 MiB / 2048 MiB / 2045 MiB ( 3744 MiB / 3968 MiB / 3746 MiB / 352 MiB / 354 MiB )
8 min passed...30176 MiB / 35072 MiB / 30228 MiB / 2400 MiB / 2400 MiB ( 3680 MiB / 4544 MiB / 3715 MiB / 352 MiB / 355 MiB )
9 min passed...33920 MiB / 39136 MiB / 33972 MiB / 2688 MiB / 2754 MiB ( 3744 MiB / 4064 MiB / 3744 MiB / 288 MiB / 354 MiB )
10 min passed...37696 MiB / 43712 MiB / 37705 MiB / 3040 MiB / 3109 MiB ( 3776 MiB / 4576 MiB / 3733 MiB / 352 MiB / 355 MiB )
11 min passed...41312 MiB / 48032 MiB / 41362 MiB / 3424 MiB / 3463 MiB ( 3616 MiB / 4320 MiB / 3657 MiB / 384 MiB / 354 MiB )
12 min passed...44960 MiB / 52160 MiB / 44995 MiB / 3808 MiB / 3830 MiB ( 3648 MiB / 4128 MiB / 3633 MiB / 384 MiB / 367 MiB )
13 min passed...48608 MiB / 56224 MiB / 48639 MiB / 4192 MiB / 4184 MiB ( 3648 MiB / 4064 MiB / 3644 MiB / 384 MiB / 354 MiB )
14 min passed...50592 MiB / 58784 MiB / 50619 MiB / 4384 MiB / 4373 MiB ( 1984 MiB / 2560 MiB / 1980 MiB / 192 MiB / 189 MiB )
15 min passed...50624 MiB / 59104 MiB / 50630 MiB / 4384 MiB / 4373 MiB ( 32 MiB / 320 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...50656 MiB / 59648 MiB / 50644 MiB / 4384 MiB / 4373 MiB ( 32 MiB / 544 MiB / 14 MiB / 0 MiB / 0 MiB )
17 min passed...50688 MiB / 60256 MiB / 50656 MiB / 4384 MiB / 4374 MiB ( 32 MiB / 608 MiB / 12 MiB / 0 MiB / 1 MiB )
18 min passed...50688 MiB / 60256 MiB / 50666 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...50688 MiB / 60736 MiB / 50677 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 480 MiB / 11 MiB / 0 MiB / 0 MiB )
20 min passed...50688 MiB / 61024 MiB / 50689 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
21 min passed...50720 MiB / 61408 MiB / 50702 MiB / 4384 MiB / 4374 MiB ( 32 MiB / 384 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...50720 MiB / 61856 MiB / 50714 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 448 MiB / 12 MiB / 0 MiB / 0 MiB )
23 min passed...50720 MiB / 62016 MiB / 50729 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 160 MiB / 15 MiB / 0 MiB / 0 MiB )
24 min passed...50720 MiB / 62144 MiB / 50740 MiB / 4384 MiB / 4374 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
25 min passed...50720 MiB / 62656 MiB / 50753 MiB / 4384 MiB / 4375 MiB ( 0 MiB / 512 MiB / 13 MiB / 0 MiB / 1 MiB )
26 min passed...50720 MiB / 63008 MiB / 50767 MiB / 4384 MiB / 4375 MiB ( 0 MiB / 352 MiB / 14 MiB / 0 MiB / 0 MiB )
27 min passed...50752 MiB / 63552 MiB / 50779 MiB / 4384 MiB / 4375 MiB ( 32 MiB / 544 MiB / 12 MiB / 0 MiB / 0 MiB )
28 min passed...50752 MiB / 63936 MiB / 50789 MiB / 4384 MiB / 4375 MiB ( 0 MiB / 384 MiB / 10 MiB / 0 MiB / 0 MiB )
29 min passed...50752 MiB / 63968 MiB / 50800 MiB / 4928 MiB / 4937 MiB ( 0 MiB / 32 MiB / 11 MiB / 544 MiB / 562 MiB )
30 min passed...50784 MiB / 64352 MiB / 50812 MiB / 5920 MiB / 5934 MiB ( 32 MiB / 384 MiB / 12 MiB / 992 MiB / 997 MiB )
31 min passed...50848 MiB / 64832 MiB / 50825 MiB / 6432 MiB / 6423 MiB ( 64 MiB / 480 MiB / 13 MiB / 512 MiB / 489 MiB )
32 min passed...50848 MiB / 65056 MiB / 50840 MiB / 6432 MiB / 6423 MiB ( 0 MiB / 224 MiB / 15 MiB / 0 MiB / 0 MiB )
33 min passed...50880 MiB / 65760 MiB / 50852 MiB / 6432 MiB / 6423 MiB ( 32 MiB / 704 MiB / 12 MiB / 0 MiB / 0 MiB )
34 min passed...50880 MiB / 65952 MiB / 50863 MiB / 6432 MiB / 6423 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
35 min passed...50880 MiB / 65952 MiB / 50875 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
36 min passed...50912 MiB / 66304 MiB / 50889 MiB / 6432 MiB / 6424 MiB ( 32 MiB / 352 MiB / 14 MiB / 0 MiB / 0 MiB )
37 min passed...50912 MiB / 66656 MiB / 50902 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 352 MiB / 13 MiB / 0 MiB / 0 MiB )
38 min passed...50912 MiB / 67072 MiB / 50915 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 416 MiB / 13 MiB / 0 MiB / 0 MiB )
39 min passed...50912 MiB / 67520 MiB / 50928 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 448 MiB / 13 MiB / 0 MiB / 0 MiB )
40 min passed...50912 MiB / 67552 MiB / 50938 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
41 min passed...50912 MiB / 67776 MiB / 50950 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 224 MiB / 12 MiB / 0 MiB / 0 MiB )
42 min passed...50912 MiB / 68288 MiB / 50962 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 512 MiB / 12 MiB / 0 MiB / 0 MiB )
43 min passed...50944 MiB / 68352 MiB / 50973 MiB / 6432 MiB / 6424 MiB ( 32 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
44 min passed...50944 MiB / 68864 MiB / 50984 MiB / 6432 MiB / 6424 MiB ( 0 MiB / 512 MiB / 11 MiB / 0 MiB / 0 MiB )
45 min passed...50976 MiB / 69056 MiB / 50993 MiB / 6432 MiB / 6425 MiB ( 32 MiB / 192 MiB / 9 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=5621KiB/s][w=351 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1232: Wed Aug 18 18:47:00 2021
  write: IOPS=353, BW=5652KiB/s (5788kB/s)(1024MiB/185527msec); 0 zone resets
    clat (usec): min=1259, max=45743, avg=2810.94, stdev=858.48
     lat (usec): min=1260, max=45745, avg=2813.10, stdev=858.48
    clat percentiles (usec):
     |  1.00th=[ 2008],  5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2474],
     | 30.00th=[ 2573], 40.00th=[ 2671], 50.00th=[ 2737], 60.00th=[ 2835],
     | 70.00th=[ 2900], 80.00th=[ 2999], 90.00th=[ 3163], 95.00th=[ 3294],
     | 99.00th=[ 5080], 99.50th=[ 5669], 99.90th=[14222], 99.95th=[19530],
     | 99.99th=[30540]
   bw (  KiB/s): min= 4576, max= 6976, per=100.00%, avg=5651.42, stdev=417.58, samples=371
   iops        : min=  286, max=  436, avg=353.20, stdev=26.10, samples=371
  lat (msec)   : 2=0.95%, 4=96.81%, 10=2.02%, 20=0.17%, 50=0.05%
  cpu          : usr=1.14%, sys=4.95%, ctx=142252, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5652KiB/s (5788kB/s), 5652KiB/s-5652KiB/s (5788kB/s-5788kB/s), io=1024MiB (1074MB), run=185527-185527msec

Disk stats (read/write):
  sda: ios=0/196943, merge=0/135063, ticks=0/167913, in_queue=185004, util=99.75%
0 min passed...Guest Writes/Reads (Changes): 2066 MiB / 0 MiB ( 2066 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2067 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2068 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2069 MiB / 0 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2069 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2069 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2069 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2070 MiB / 0 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2070 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2070 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=26.9MiB/s][r=1723 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1435: Wed Aug 18 19:02:35 2021
  read: IOPS=1939, BW=30.3MiB/s (31.8MB/s)(1024MiB/33786msec)
    clat (usec): min=225, max=4500, avg=504.83, stdev=131.01
     lat (usec): min=227, max=4502, avg=506.70, stdev=131.07
    clat percentiles (usec):
     |  1.00th=[  359],  5.00th=[  383], 10.00th=[  400], 20.00th=[  424],
     | 30.00th=[  445], 40.00th=[  465], 50.00th=[  486], 60.00th=[  510],
     | 70.00th=[  537], 80.00th=[  578], 90.00th=[  627], 95.00th=[  660],
     | 99.00th=[  709], 99.50th=[  734], 99.90th=[ 2606], 99.95th=[ 3032],
     | 99.99th=[ 3392]
   bw (  KiB/s): min=26560, max=36512, per=100.00%, avg=31072.70, stdev=3036.03, samples=67
   iops        : min= 1660, max= 2282, avg=1942.03, stdev=189.76, samples=67
  lat (usec)   : 250=0.01%, 500=56.27%, 750=43.32%, 1000=0.11%
  lat (msec)   : 2=0.14%, 4=0.16%, 10=0.01%
  cpu          : usr=3.36%, sys=10.56%, ctx=65540, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.3MiB/s (31.8MB/s), 30.3MiB/s-30.3MiB/s (31.8MB/s-31.8MB/s), io=1024MiB (1074MB), run=33786-33786msec

Disk stats (read/write):
  sda: ios=65305/25, merge=0/9, ticks=30405/25, in_queue=33768, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2070 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2071 MiB / 1024 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2071 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2071 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2071 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2072 MiB / 1024 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2072 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2072 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2072 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2073 MiB / 1024 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2073 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2073 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2073 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2074 MiB / 1024 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2074 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2074 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2074 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 2021 seconds or 33 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4224 MiB / 5088 MiB / 4217 MiB / 192 MiB / 191 MiB ( 4224 MiB / 5088 MiB / 4217 MiB / 192 MiB / 191 MiB )
2 min passed...8608 MiB / 9984 MiB / 8591 MiB / 384 MiB / 395 MiB ( 4384 MiB / 4896 MiB / 4374 MiB / 192 MiB / 204 MiB )
3 min passed...13088 MiB / 14784 MiB / 13087 MiB / 576 MiB / 572 MiB ( 4480 MiB / 4800 MiB / 4496 MiB / 192 MiB / 177 MiB )
4 min passed...13536 MiB / 15264 MiB / 13517 MiB / 576 MiB / 587 MiB ( 448 MiB / 480 MiB / 430 MiB / 0 MiB / 15 MiB )
5 min passed...13568 MiB / 15296 MiB / 13534 MiB / 576 MiB / 588 MiB ( 32 MiB / 32 MiB / 17 MiB / 0 MiB / 1 MiB )
6 min passed...13600 MiB / 15872 MiB / 13556 MiB / 576 MiB / 588 MiB ( 32 MiB / 576 MiB / 22 MiB / 0 MiB / 0 MiB )
7 min passed...13600 MiB / 16288 MiB / 13575 MiB / 576 MiB / 588 MiB ( 0 MiB / 416 MiB / 19 MiB / 0 MiB / 0 MiB )
8 min passed...13632 MiB / 16640 MiB / 13594 MiB / 576 MiB / 588 MiB ( 32 MiB / 352 MiB / 19 MiB / 0 MiB / 0 MiB )
9 min passed...13632 MiB / 16800 MiB / 13614 MiB / 576 MiB / 589 MiB ( 0 MiB / 160 MiB / 20 MiB / 0 MiB / 1 MiB )
10 min passed...13664 MiB / 16800 MiB / 13633 MiB / 576 MiB / 589 MiB ( 32 MiB / 0 MiB / 19 MiB / 0 MiB / 0 MiB )
11 min passed...13664 MiB / 17120 MiB / 13650 MiB / 576 MiB / 590 MiB ( 0 MiB / 320 MiB / 17 MiB / 0 MiB / 1 MiB )
12 min passed...13696 MiB / 17728 MiB / 13672 MiB / 576 MiB / 590 MiB ( 32 MiB / 608 MiB / 22 MiB / 0 MiB / 0 MiB )
13 min passed...13696 MiB / 17856 MiB / 13694 MiB / 576 MiB / 590 MiB ( 0 MiB / 128 MiB / 22 MiB / 0 MiB / 0 MiB )
14 min passed...13760 MiB / 18304 MiB / 13715 MiB / 576 MiB / 591 MiB ( 64 MiB / 448 MiB / 21 MiB / 0 MiB / 1 MiB )
15 min passed...13792 MiB / 18336 MiB / 13734 MiB / 576 MiB / 591 MiB ( 32 MiB / 32 MiB / 19 MiB / 0 MiB / 0 MiB )
16 min passed...13792 MiB / 18432 MiB / 13752 MiB / 576 MiB / 591 MiB ( 0 MiB / 96 MiB / 18 MiB / 0 MiB / 0 MiB )
17 min passed...13792 MiB / 19200 MiB / 13772 MiB / 576 MiB / 592 MiB ( 0 MiB / 768 MiB / 20 MiB / 0 MiB / 1 MiB )
18 min passed...13792 MiB / 19712 MiB / 13794 MiB / 608 MiB / 646 MiB ( 0 MiB / 512 MiB / 22 MiB / 32 MiB / 54 MiB )
19 min passed...13856 MiB / 19904 MiB / 13815 MiB / 1600 MiB / 1616 MiB ( 64 MiB / 192 MiB / 21 MiB / 992 MiB / 970 MiB )
20 min passed...13856 MiB / 19968 MiB / 13834 MiB / 1600 MiB / 1617 MiB ( 0 MiB / 64 MiB / 19 MiB / 0 MiB / 1 MiB )
21 min passed...13888 MiB / 19968 MiB / 13853 MiB / 1600 MiB / 1617 MiB ( 32 MiB / 0 MiB / 19 MiB / 0 MiB / 0 MiB )
22 min passed...13888 MiB / 20384 MiB / 13871 MiB / 1600 MiB / 1617 MiB ( 0 MiB / 416 MiB / 18 MiB / 0 MiB / 0 MiB )
23 min passed...13920 MiB / 20896 MiB / 13892 MiB / 1600 MiB / 1618 MiB ( 32 MiB / 512 MiB / 21 MiB / 0 MiB / 1 MiB )
24 min passed...13920 MiB / 20960 MiB / 13911 MiB / 1600 MiB / 1618 MiB ( 0 MiB / 64 MiB / 19 MiB / 0 MiB / 0 MiB )
25 min passed...13984 MiB / 21504 MiB / 13930 MiB / 1600 MiB / 1618 MiB ( 64 MiB / 544 MiB / 19 MiB / 0 MiB / 0 MiB )
26 min passed...13984 MiB / 21600 MiB / 13950 MiB / 1600 MiB / 1619 MiB ( 0 MiB / 96 MiB / 20 MiB / 0 MiB / 1 MiB )
27 min passed...13984 MiB / 21824 MiB / 13967 MiB / 1600 MiB / 1619 MiB ( 0 MiB / 224 MiB / 17 MiB / 0 MiB / 0 MiB )
28 min passed...13984 MiB / 22304 MiB / 13987 MiB / 1600 MiB / 1619 MiB ( 0 MiB / 480 MiB / 20 MiB / 0 MiB / 0 MiB )
29 min passed...14016 MiB / 22496 MiB / 14006 MiB / 1600 MiB / 1619 MiB ( 32 MiB / 192 MiB / 19 MiB / 0 MiB / 0 MiB )
30 min passed...14048 MiB / 22848 MiB / 14024 MiB / 1600 MiB / 1619 MiB ( 32 MiB / 352 MiB / 18 MiB / 0 MiB / 0 MiB )
31 min passed...14080 MiB / 23008 MiB / 14042 MiB / 1600 MiB / 1620 MiB ( 32 MiB / 160 MiB / 18 MiB / 0 MiB / 1 MiB )
32 min passed...14080 MiB / 23072 MiB / 14064 MiB / 1600 MiB / 1620 MiB ( 0 MiB / 64 MiB / 22 MiB / 0 MiB / 0 MiB )
33 min passed...14112 MiB / 23456 MiB / 14081 MiB / 1600 MiB / 1620 MiB ( 32 MiB / 384 MiB / 17 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=208MiB/s][w=52 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1680: Wed Aug 18 19:45:16 2021
  write: IOPS=50, BW=202MiB/s (212MB/s)(10.0GiB/50688msec); 0 zone resets
    clat (usec): min=11630, max=66669, avg=18143.58, stdev=6965.55
     lat (usec): min=11633, max=66672, avg=18147.04, stdev=6965.65
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13304], 10.00th=[13698], 20.00th=[13960],
     | 30.00th=[14222], 40.00th=[14353], 50.00th=[14484], 60.00th=[14746],
     | 70.00th=[16188], 80.00th=[25560], 90.00th=[28705], 95.00th=[30802],
     | 99.00th=[40633], 99.50th=[48497], 99.90th=[61080], 99.95th=[63177],
     | 99.99th=[66847]
   bw (  KiB/s): min=122880, max=270336, per=99.86%, avg=206587.91, stdev=52386.21, samples=101
   iops        : min=   30, max=   66, avg=50.41, stdev=12.79, samples=101
  lat (msec)   : 20=73.95%, 50=25.66%, 100=0.39%
  cpu          : usr=8.05%, sys=2.49%, ctx=6279, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=10.0GiB (10.7GB), run=50688-50688msec

Disk stats (read/write):
  sda: ios=0/15387, merge=0/5300, ticks=0/68447, in_queue=74332, util=98.02%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 10281 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10282 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10283 MiB / 0 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10283 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10283 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10283 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10284 MiB / 0 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10284 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10284 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10284 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10285 MiB / 0 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10285 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=744MiB/s][r=186 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1884: Wed Aug 18 20:00:30 2021
  read: IOPS=191, BW=765MiB/s (802MB/s)(10.0GiB/13388msec)
    clat (usec): min=4033, max=11698, avg=5220.23, stdev=1042.83
     lat (usec): min=4034, max=11700, avg=5221.76, stdev=1042.93
    clat percentiles (usec):
     |  1.00th=[ 4228],  5.00th=[ 4359], 10.00th=[ 4359], 20.00th=[ 4490],
     | 30.00th=[ 4555], 40.00th=[ 4686], 50.00th=[ 4817], 60.00th=[ 5014],
     | 70.00th=[ 5407], 80.00th=[ 5866], 90.00th=[ 6652], 95.00th=[ 7308],
     | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[10552], 99.95th=[11207],
     | 99.99th=[11731]
   bw (  KiB/s): min=745472, max=811008, per=100.00%, avg=783850.35, stdev=18468.76, samples=26
   iops        : min=  182, max=  198, avg=191.35, stdev= 4.50, samples=26
  lat (msec)   : 10=99.73%, 20=0.27%
  cpu          : usr=0.23%, sys=4.47%, ctx=2562, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=765MiB/s (802MB/s), 765MiB/s-765MiB/s (802MB/s-802MB/s), io=10.0GiB (10.7GB), run=13388-13388msec

Disk stats (read/write):
  sda: ios=10064/11, merge=0/4, ticks=43758/12, in_queue=43904, util=99.31%
0 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10240 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10287 MiB / 10240 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10287 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10287 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10287 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10287 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10288 MiB / 10240 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10288 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10288 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10288 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10289 MiB / 10240 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10289 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1866 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...42624 MiB / 45856 MiB / 42622 MiB / 32 MiB / 20 MiB ( 42624 MiB / 45856 MiB / 42622 MiB / 32 MiB / 20 MiB )
2 min passed...42656 MiB / 45952 MiB / 42644 MiB / 32 MiB / 20 MiB ( 32 MiB / 96 MiB / 22 MiB / 0 MiB / 0 MiB )
3 min passed...42688 MiB / 46016 MiB / 42666 MiB / 32 MiB / 21 MiB ( 32 MiB / 64 MiB / 22 MiB / 0 MiB / 1 MiB )
4 min passed...42720 MiB / 46240 MiB / 42686 MiB / 32 MiB / 21 MiB ( 32 MiB / 224 MiB / 20 MiB / 0 MiB / 0 MiB )
5 min passed...42720 MiB / 46688 MiB / 42706 MiB / 32 MiB / 21 MiB ( 0 MiB / 448 MiB / 20 MiB / 0 MiB / 0 MiB )
6 min passed...42720 MiB / 47264 MiB / 42725 MiB / 32 MiB / 21 MiB ( 0 MiB / 576 MiB / 19 MiB / 0 MiB / 0 MiB )
7 min passed...42752 MiB / 47360 MiB / 42745 MiB / 32 MiB / 22 MiB ( 32 MiB / 96 MiB / 20 MiB / 0 MiB / 1 MiB )
8 min passed...42752 MiB / 47616 MiB / 42764 MiB / 32 MiB / 22 MiB ( 0 MiB / 256 MiB / 19 MiB / 0 MiB / 0 MiB )
9 min passed...42784 MiB / 47680 MiB / 42783 MiB / 32 MiB / 22 MiB ( 32 MiB / 64 MiB / 19 MiB / 0 MiB / 0 MiB )
10 min passed...42784 MiB / 48160 MiB / 42806 MiB / 32 MiB / 22 MiB ( 0 MiB / 480 MiB / 23 MiB / 0 MiB / 0 MiB )
11 min passed...42816 MiB / 48672 MiB / 42828 MiB / 32 MiB / 23 MiB ( 32 MiB / 512 MiB / 22 MiB / 0 MiB / 1 MiB )
12 min passed...42880 MiB / 48896 MiB / 42849 MiB / 32 MiB / 23 MiB ( 64 MiB / 224 MiB / 21 MiB / 0 MiB / 0 MiB )
13 min passed...42912 MiB / 49088 MiB / 42868 MiB / 32 MiB / 23 MiB ( 32 MiB / 192 MiB / 19 MiB / 0 MiB / 0 MiB )
14 min passed...42912 MiB / 49248 MiB / 42889 MiB / 32 MiB / 23 MiB ( 0 MiB / 160 MiB / 21 MiB / 0 MiB / 0 MiB )
15 min passed...42912 MiB / 49504 MiB / 42912 MiB / 32 MiB / 24 MiB ( 0 MiB / 256 MiB / 23 MiB / 0 MiB / 1 MiB )
16 min passed...42912 MiB / 50144 MiB / 42933 MiB / 10816 MiB / 10833 MiB ( 0 MiB / 640 MiB / 21 MiB / 10784 MiB / 10809 MiB )
17 min passed...42944 MiB / 50368 MiB / 42952 MiB / 10816 MiB / 10833 MiB ( 32 MiB / 224 MiB / 19 MiB / 0 MiB / 0 MiB )
18 min passed...42976 MiB / 50528 MiB / 42971 MiB / 10816 MiB / 10834 MiB ( 32 MiB / 160 MiB / 19 MiB / 0 MiB / 1 MiB )
19 min passed...42976 MiB / 50848 MiB / 42989 MiB / 10816 MiB / 10834 MiB ( 0 MiB / 320 MiB / 18 MiB / 0 MiB / 0 MiB )
20 min passed...43008 MiB / 50944 MiB / 43009 MiB / 10816 MiB / 10834 MiB ( 32 MiB / 96 MiB / 20 MiB / 0 MiB / 0 MiB )
21 min passed...43040 MiB / 51104 MiB / 43028 MiB / 10816 MiB / 10834 MiB ( 32 MiB / 160 MiB / 19 MiB / 0 MiB / 0 MiB )
22 min passed...43104 MiB / 51136 MiB / 43048 MiB / 10816 MiB / 10835 MiB ( 64 MiB / 32 MiB / 20 MiB / 0 MiB / 1 MiB )
23 min passed...43104 MiB / 51168 MiB / 43068 MiB / 10816 MiB / 10835 MiB ( 0 MiB / 32 MiB / 20 MiB / 0 MiB / 0 MiB )
24 min passed...43104 MiB / 51456 MiB / 43087 MiB / 10816 MiB / 10835 MiB ( 0 MiB / 288 MiB / 19 MiB / 0 MiB / 0 MiB )
25 min passed...43104 MiB / 51584 MiB / 43106 MiB / 10816 MiB / 10835 MiB ( 0 MiB / 128 MiB / 19 MiB / 0 MiB / 0 MiB )
26 min passed...43104 MiB / 51712 MiB / 43127 MiB / 10816 MiB / 10836 MiB ( 0 MiB / 128 MiB / 21 MiB / 0 MiB / 1 MiB )
27 min passed...43136 MiB / 52128 MiB / 43147 MiB / 10816 MiB / 10836 MiB ( 32 MiB / 416 MiB / 20 MiB / 0 MiB / 0 MiB )
28 min passed...43136 MiB / 52160 MiB / 43165 MiB / 10816 MiB / 10836 MiB ( 0 MiB / 32 MiB / 18 MiB / 0 MiB / 0 MiB )
29 min passed...43200 MiB / 52320 MiB / 43184 MiB / 10816 MiB / 10837 MiB ( 64 MiB / 160 MiB / 19 MiB / 0 MiB / 1 MiB )
30 min passed...43232 MiB / 52352 MiB / 43201 MiB / 10816 MiB / 10837 MiB ( 32 MiB / 32 MiB / 17 MiB / 0 MiB / 0 MiB )
31 min passed...43264 MiB / 52480 MiB / 43225 MiB / 10816 MiB / 10837 MiB ( 32 MiB / 128 MiB / 24 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=124MiB/s][w=31.8k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2129: Wed Aug 18 20:45:06 2021
  write: IOPS=30.7k, BW=120MiB/s (126MB/s)(1024MiB/8548msec); 0 zone resets
    slat (usec): min=6, max=4191, avg=21.84, stdev=21.36
    clat (usec): min=432, max=32942, avg=2057.76, stdev=1102.51
     lat (usec): min=441, max=32958, avg=2081.10, stdev=1104.88
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1696],
     | 30.00th=[ 1729], 40.00th=[ 1762], 50.00th=[ 1811], 60.00th=[ 1844],
     | 70.00th=[ 1926], 80.00th=[ 2057], 90.00th=[ 2540], 95.00th=[ 3032],
     | 99.00th=[ 7767], 99.50th=[ 8979], 99.90th=[12125], 99.95th=[13698],
     | 99.99th=[30278]
   bw (  KiB/s): min=69848, max=147040, per=99.89%, avg=122537.88, stdev=27356.61, samples=17
   iops        : min=17462, max=36760, avg=30634.47, stdev=6839.15, samples=17
  lat (usec)   : 500=0.01%, 750=0.03%, 1000=0.21%
  lat (msec)   : 2=76.48%, 4=20.19%, 10=2.81%, 20=0.24%, 50=0.03%
  cpu          : usr=17.59%, sys=54.32%, ctx=220974, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=1024MiB (1074MB), run=8548-8548msec

Disk stats (read/write):
  sda: ios=0/256431, merge=0/664, ticks=0/83985, in_queue=87512, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 1026 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1031 MiB / 0 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1031 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1032 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=258MiB/s][r=66.1k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2374: Wed Aug 18 21:00:10 2021
  read: IOPS=69.7k, BW=272MiB/s (285MB/s)(1024MiB/3761msec)
    slat (usec): min=3, max=189, avg= 7.52, stdev= 3.99
    clat (usec): min=196, max=4956, avg=905.66, stdev=112.00
     lat (usec): min=201, max=4964, avg=914.37, stdev=112.90
    clat percentiles (usec):
     |  1.00th=[  766],  5.00th=[  799], 10.00th=[  816], 20.00th=[  832],
     | 30.00th=[  848], 40.00th=[  865], 50.00th=[  881], 60.00th=[  889],
     | 70.00th=[  914], 80.00th=[  947], 90.00th=[ 1074], 95.00th=[ 1106],
     | 99.00th=[ 1205], 99.50th=[ 1287], 99.90th=[ 1729], 99.95th=[ 2278],
     | 99.99th=[ 3228]
   bw (  KiB/s): min=230112, max=295680, per=99.83%, avg=278339.43, stdev=22221.33, samples=7
   iops        : min=57528, max=73920, avg=69584.86, stdev=5555.33, samples=7
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.36%, 1000=84.35%
  lat (msec)   : 2=15.20%, 4=0.07%, 10=0.01%
  cpu          : usr=43.01%, sys=56.60%, ctx=395, majf=0, minf=72
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=1024MiB (1074MB), run=3761-3761msec

Disk stats (read/write):
  sda: ios=253989/2, merge=0/3, ticks=56846/5, in_queue=57296, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1035 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1813 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3072 MiB / 3264 MiB / 3085 MiB / 672 MiB / 637 MiB ( 3072 MiB / 3264 MiB / 3085 MiB / 672 MiB / 637 MiB )
2 min passed...3072 MiB / 3712 MiB / 3098 MiB / 672 MiB / 637 MiB ( 0 MiB / 448 MiB / 13 MiB / 0 MiB / 0 MiB )
3 min passed...3072 MiB / 4000 MiB / 3114 MiB / 672 MiB / 637 MiB ( 0 MiB / 288 MiB / 16 MiB / 0 MiB / 0 MiB )
4 min passed...3104 MiB / 4448 MiB / 3126 MiB / 672 MiB / 637 MiB ( 32 MiB / 448 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...3104 MiB / 4480 MiB / 3142 MiB / 672 MiB / 637 MiB ( 0 MiB / 32 MiB / 16 MiB / 0 MiB / 0 MiB )
6 min passed...3104 MiB / 4608 MiB / 3153 MiB / 672 MiB / 637 MiB ( 0 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
7 min passed...3136 MiB / 4800 MiB / 3167 MiB / 672 MiB / 638 MiB ( 32 MiB / 192 MiB / 14 MiB / 0 MiB / 1 MiB )
8 min passed...3136 MiB / 4928 MiB / 3180 MiB / 672 MiB / 638 MiB ( 0 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
9 min passed...3136 MiB / 5280 MiB / 3194 MiB / 672 MiB / 638 MiB ( 0 MiB / 352 MiB / 14 MiB / 0 MiB / 0 MiB )
10 min passed...3136 MiB / 5504 MiB / 3206 MiB / 672 MiB / 638 MiB ( 0 MiB / 224 MiB / 12 MiB / 0 MiB / 0 MiB )
11 min passed...3200 MiB / 5696 MiB / 3220 MiB / 672 MiB / 638 MiB ( 64 MiB / 192 MiB / 14 MiB / 0 MiB / 0 MiB )
12 min passed...3200 MiB / 5856 MiB / 3232 MiB / 672 MiB / 638 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
13 min passed...3200 MiB / 6048 MiB / 3247 MiB / 672 MiB / 638 MiB ( 0 MiB / 192 MiB / 15 MiB / 0 MiB / 0 MiB )
14 min passed...3232 MiB / 6208 MiB / 3256 MiB / 672 MiB / 638 MiB ( 32 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
15 min passed...3232 MiB / 6688 MiB / 3267 MiB / 672 MiB / 638 MiB ( 0 MiB / 480 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...3264 MiB / 6752 MiB / 3283 MiB / 2656 MiB / 2681 MiB ( 32 MiB / 64 MiB / 16 MiB / 1984 MiB / 2043 MiB )
17 min passed...3264 MiB / 6752 MiB / 3294 MiB / 2656 MiB / 2681 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
18 min passed...3264 MiB / 7008 MiB / 3307 MiB / 2656 MiB / 2681 MiB ( 0 MiB / 256 MiB / 13 MiB / 0 MiB / 0 MiB )
19 min passed...3264 MiB / 7296 MiB / 3317 MiB / 2656 MiB / 2681 MiB ( 0 MiB / 288 MiB / 10 MiB / 0 MiB / 0 MiB )
20 min passed...3296 MiB / 7712 MiB / 3331 MiB / 2656 MiB / 2682 MiB ( 32 MiB / 416 MiB / 14 MiB / 0 MiB / 1 MiB )
21 min passed...3296 MiB / 7712 MiB / 3344 MiB / 2656 MiB / 2682 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...3328 MiB / 7936 MiB / 3358 MiB / 2656 MiB / 2682 MiB ( 32 MiB / 224 MiB / 14 MiB / 0 MiB / 0 MiB )
23 min passed...3328 MiB / 8192 MiB / 3372 MiB / 2656 MiB / 2682 MiB ( 0 MiB / 256 MiB / 14 MiB / 0 MiB / 0 MiB )
24 min passed...3328 MiB / 8672 MiB / 3384 MiB / 2656 MiB / 2682 MiB ( 0 MiB / 480 MiB / 12 MiB / 0 MiB / 0 MiB )
25 min passed...3360 MiB / 8896 MiB / 3398 MiB / 2656 MiB / 2682 MiB ( 32 MiB / 224 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...3392 MiB / 9024 MiB / 3409 MiB / 2656 MiB / 2682 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
27 min passed...3392 MiB / 9184 MiB / 3423 MiB / 2656 MiB / 2682 MiB ( 0 MiB / 160 MiB / 14 MiB / 0 MiB / 0 MiB )
28 min passed...3392 MiB / 9280 MiB / 3433 MiB / 2656 MiB / 2683 MiB ( 0 MiB / 96 MiB / 10 MiB / 0 MiB / 1 MiB )
29 min passed...3424 MiB / 9600 MiB / 3448 MiB / 2656 MiB / 2683 MiB ( 32 MiB / 320 MiB / 15 MiB / 0 MiB / 0 MiB )
30 min passed...3424 MiB / 10272 MiB / 3461 MiB / 2656 MiB / 2683 MiB ( 0 MiB / 672 MiB / 13 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2634: Wed Aug 18 22:10:51 2021
  write: IOPS=27.5k, BW=430MiB/s (451MB/s)(1024MiB/2383msec); 0 zone resets
    slat (usec): min=7, max=14013, avg=20.40, stdev=58.59
    clat (usec): min=48, max=24063, avg=2291.91, stdev=1315.97
     lat (usec): min=79, max=24083, avg=2313.76, stdev=1319.38
    clat percentiles (usec):
     |  1.00th=[ 1795],  5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 1975],
     | 30.00th=[ 2008], 40.00th=[ 2040], 50.00th=[ 2073], 60.00th=[ 2114],
     | 70.00th=[ 2147], 80.00th=[ 2343], 90.00th=[ 2638], 95.00th=[ 3032],
     | 99.00th=[ 5407], 99.50th=[11863], 99.90th=[19530], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min=319520, max=498752, per=100.00%, avg=449376.00, stdev=86723.22, samples=4
   iops        : min=19970, max=31172, avg=28086.00, stdev=5420.20, samples=4
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=29.71%, 4=68.55%, 10=1.14%, 20=0.46%, 50=0.10%
  cpu          : usr=22.59%, sys=53.36%, ctx=63736, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=1024MiB (1074MB), run=2383-2383msec

Disk stats (read/write):
  sda: ios=0/59371, merge=0/0, ticks=0/10329, in_queue=10456, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1027 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2858: Wed Aug 18 22:25:53 2021
  read: IOPS=50.6k, BW=790MiB/s (829MB/s)(1024MiB/1296msec)
    slat (usec): min=4, max=331, avg=10.73, stdev=14.99
    clat (usec): min=338, max=6791, avg=1247.64, stdev=402.28
     lat (usec): min=349, max=6799, avg=1259.84, stdev=404.90
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  807], 10.00th=[  889], 20.00th=[  963],
     | 30.00th=[ 1020], 40.00th=[ 1090], 50.00th=[ 1172], 60.00th=[ 1221],
     | 70.00th=[ 1254], 80.00th=[ 1516], 90.00th=[ 1876], 95.00th=[ 1991],
     | 99.00th=[ 2474], 99.50th=[ 2868], 99.90th=[ 3785], 99.95th=[ 4424],
     | 99.99th=[ 5735]
   bw (  KiB/s): min=649536, max=966592, per=99.87%, avg=808064.00, stdev=224192.45, samples=2
   iops        : min=40596, max=60412, avg=50504.00, stdev=14012.03, samples=2
  lat (usec)   : 500=0.05%, 750=3.07%, 1000=23.61%
  lat (msec)   : 2=68.34%, 4=4.85%, 10=0.08%
  cpu          : usr=36.14%, sys=62.93%, ctx=186, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=790MiB/s (829MB/s), 790MiB/s-790MiB/s (829MB/s-829MB/s), io=1024MiB (1074MB), run=1296-1296msec

Disk stats (read/write):
  sda: ios=57997/0, merge=0/0, ticks=26454/0, in_queue=27996, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1027 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1029 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2624 MiB / 3232 MiB / 2637 MiB / 0 MiB / 0 MiB ( 2624 MiB / 3232 MiB / 2637 MiB / 0 MiB / 0 MiB )
2 min passed...2624 MiB / 3520 MiB / 2649 MiB / 0 MiB / 0 MiB ( 0 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
3 min passed...2624 MiB / 3872 MiB / 2663 MiB / 0 MiB / 0 MiB ( 0 MiB / 352 MiB / 14 MiB / 0 MiB / 0 MiB )
4 min passed...2624 MiB / 4032 MiB / 2676 MiB / 0 MiB / 1 MiB ( 0 MiB / 160 MiB / 13 MiB / 0 MiB / 1 MiB )
5 min passed...2656 MiB / 4352 MiB / 2689 MiB / 0 MiB / 1 MiB ( 32 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
6 min passed...2656 MiB / 4768 MiB / 2704 MiB / 0 MiB / 1 MiB ( 0 MiB / 416 MiB / 15 MiB / 0 MiB / 0 MiB )
7 min passed...2720 MiB / 5216 MiB / 2718 MiB / 0 MiB / 1 MiB ( 64 MiB / 448 MiB / 14 MiB / 0 MiB / 0 MiB )
8 min passed...2720 MiB / 5536 MiB / 2731 MiB / 0 MiB / 1 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
9 min passed...2720 MiB / 5696 MiB / 2744 MiB / 0 MiB / 1 MiB ( 0 MiB / 160 MiB / 13 MiB / 0 MiB / 0 MiB )
10 min passed...2720 MiB / 5984 MiB / 2757 MiB / 0 MiB / 1 MiB ( 0 MiB / 288 MiB / 13 MiB / 0 MiB / 0 MiB )
11 min passed...2752 MiB / 6272 MiB / 2770 MiB / 0 MiB / 2 MiB ( 32 MiB / 288 MiB / 13 MiB / 0 MiB / 1 MiB )
12 min passed...2752 MiB / 6784 MiB / 2783 MiB / 0 MiB / 2 MiB ( 0 MiB / 512 MiB / 13 MiB / 0 MiB / 0 MiB )
13 min passed...2816 MiB / 7072 MiB / 2795 MiB / 0 MiB / 2 MiB ( 64 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
14 min passed...2816 MiB / 7488 MiB / 2807 MiB / 0 MiB / 2 MiB ( 0 MiB / 416 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...2816 MiB / 7776 MiB / 2820 MiB / 1056 MiB / 1027 MiB ( 0 MiB / 288 MiB / 13 MiB / 1056 MiB / 1025 MiB )
16 min passed...2816 MiB / 8000 MiB / 2833 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 1 MiB )
17 min passed...2816 MiB / 8320 MiB / 2845 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
18 min passed...2816 MiB / 8768 MiB / 2855 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 448 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...2848 MiB / 9152 MiB / 2868 MiB / 1056 MiB / 1028 MiB ( 32 MiB / 384 MiB / 13 MiB / 0 MiB / 0 MiB )
20 min passed...2848 MiB / 9408 MiB / 2878 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 256 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...2848 MiB / 9792 MiB / 2891 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 384 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...2848 MiB / 10112 MiB / 2902 MiB / 1056 MiB / 1028 MiB ( 0 MiB / 320 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...2912 MiB / 10688 MiB / 2913 MiB / 1056 MiB / 1028 MiB ( 64 MiB / 576 MiB / 11 MiB / 0 MiB / 0 MiB )
24 min passed...2912 MiB / 11104 MiB / 2925 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 416 MiB / 12 MiB / 0 MiB / 1 MiB )
25 min passed...2912 MiB / 11456 MiB / 2936 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 352 MiB / 11 MiB / 0 MiB / 0 MiB )
26 min passed...2912 MiB / 11680 MiB / 2949 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 224 MiB / 13 MiB / 0 MiB / 0 MiB )
27 min passed...2944 MiB / 12064 MiB / 2959 MiB / 1056 MiB / 1029 MiB ( 32 MiB / 384 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...2944 MiB / 12544 MiB / 2972 MiB / 1056 MiB / 1029 MiB ( 0 MiB / 480 MiB / 13 MiB / 0 MiB / 0 MiB )
29 min passed...2976 MiB / 12928 MiB / 2983 MiB / 1056 MiB / 1029 MiB ( 32 MiB / 384 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...3008 MiB / 13376 MiB / 2997 MiB / 1056 MiB / 1029 MiB ( 32 MiB / 448 MiB / 14 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 18, 2021

Round 2 - Run 7 - Test 6: async 4M sequential read/write

Gast:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][90.5%][eta 00m:02s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3096: Wed Aug 18 23:00:25 2021
  write: IOPS=132, BW=528MiB/s (554MB/s)(10.0GiB/19378msec); 0 zone resets
    slat (usec): min=86, max=29752, avg=248.49, stdev=864.37
    clat (msec): min=24, max=7452, avg=482.05, stdev=1097.78
     lat (msec): min=24, max=7452, avg=482.30, stdev=1097.76
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   67], 10.00th=[   77], 20.00th=[   79],
     | 30.00th=[   81], 40.00th=[   84], 50.00th=[   89], 60.00th=[  103],
     | 70.00th=[  126], 80.00th=[  236], 90.00th=[ 1603], 95.00th=[ 2836],
     | 99.00th=[ 5873], 99.50th=[ 5873], 99.90th=[ 5940], 99.95th=[ 5940],
     | 99.99th=[ 7483]
   bw (  MiB/s): min=   24, max= 2600, per=100.00%, avg=1040.36, stdev=1044.21, samples=18
   iops        : min=    6, max=  650, avg=260.06, stdev=261.09, samples=18
  lat (msec)   : 50=0.98%, 100=57.89%, 250=21.56%, 500=4.02%, 750=0.59%
  lat (msec)   : 1000=2.66%
  cpu          : usr=14.92%, sys=3.22%, ctx=1193, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=528MiB/s (554MB/s), 528MiB/s-528MiB/s (554MB/s-554MB/s), io=10.0GiB (10.7GB), run=19378-19378msec

Disk stats (read/write):
  sda: ios=0/10026, merge=0/175, ticks=0/3607893, in_queue=865676, util=34.45%
0 min passed...Guest Writes/Reads (Changes): 10240 MiB / 0 MiB ( 10240 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10242 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10243 MiB / 0 MiB ( 1 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1513MiB/s][r=378 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3330: Wed Aug 18 23:15:32 2021
  read: IOPS=398, BW=1593MiB/s (1670MB/s)(10.0GiB/6429msec)
    slat (usec): min=86, max=36171, avg=282.42, stdev=1008.92
    clat (msec): min=19, max=360, avg=159.54, stdev=69.65
     lat (msec): min=19, max=360, avg=159.82, stdev=69.57
    clat percentiles (msec):
     |  1.00th=[   38],  5.00th=[   55], 10.00th=[   70], 20.00th=[   86],
     | 30.00th=[  109], 40.00th=[  131], 50.00th=[  161], 60.00th=[  186],
     | 70.00th=[  211], 80.00th=[  230], 90.00th=[  249], 95.00th=[  266],
     | 99.00th=[  288], 99.50th=[  342], 99.90th=[  347], 99.95th=[  347],
     | 99.99th=[  359]
   bw (  MiB/s): min= 1160, max= 2760, per=99.02%, avg=1577.24, stdev=393.41, samples=12
   iops        : min=  290, max=  690, avg=394.25, stdev=98.37, samples=12
  lat (msec)   : 20=0.04%, 50=4.45%, 100=21.80%, 250=64.14%, 500=9.57%
  cpu          : usr=1.51%, sys=11.14%, ctx=1661, majf=0, minf=650
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1593MiB/s (1670MB/s), 1593MiB/s-1593MiB/s (1670MB/s-1670MB/s), io=10.0GiB (10.7GB), run=6429-6429msec

Disk stats (read/write):
  sda: ios=10178/3, merge=0/3, ticks=1200299/387, in_queue=1204532, util=99.63%
0 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10245 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1827 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...25952 MiB / 28416 MiB / 25924 MiB / 352 MiB / 307 MiB ( 25952 MiB / 28416 MiB / 25924 MiB / 352 MiB / 307 MiB )
2 min passed...25952 MiB / 28864 MiB / 25941 MiB / 352 MiB / 307 MiB ( 0 MiB / 448 MiB / 17 MiB / 0 MiB / 0 MiB )
3 min passed...25984 MiB / 28992 MiB / 25955 MiB / 352 MiB / 307 MiB ( 32 MiB / 128 MiB / 14 MiB / 0 MiB / 0 MiB )
4 min passed...26016 MiB / 29344 MiB / 25968 MiB / 352 MiB / 307 MiB ( 32 MiB / 352 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...26016 MiB / 29792 MiB / 25984 MiB / 352 MiB / 307 MiB ( 0 MiB / 448 MiB / 16 MiB / 0 MiB / 0 MiB )
6 min passed...26016 MiB / 30240 MiB / 25996 MiB / 352 MiB / 308 MiB ( 0 MiB / 448 MiB / 12 MiB / 0 MiB / 1 MiB )
7 min passed...26016 MiB / 30848 MiB / 26011 MiB / 352 MiB / 308 MiB ( 0 MiB / 608 MiB / 15 MiB / 0 MiB / 0 MiB )
8 min passed...26016 MiB / 31040 MiB / 26023 MiB / 352 MiB / 308 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
9 min passed...26016 MiB / 31232 MiB / 26038 MiB / 352 MiB / 308 MiB ( 0 MiB / 192 MiB / 15 MiB / 0 MiB / 0 MiB )
10 min passed...26016 MiB / 31840 MiB / 26052 MiB / 352 MiB / 308 MiB ( 0 MiB / 608 MiB / 14 MiB / 0 MiB / 0 MiB )
11 min passed...26016 MiB / 32192 MiB / 26064 MiB / 352 MiB / 308 MiB ( 0 MiB / 352 MiB / 12 MiB / 0 MiB / 0 MiB )
12 min passed...26080 MiB / 32832 MiB / 26078 MiB / 352 MiB / 308 MiB ( 64 MiB / 640 MiB / 14 MiB / 0 MiB / 0 MiB )
13 min passed...26080 MiB / 32864 MiB / 26091 MiB / 352 MiB / 308 MiB ( 0 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...26144 MiB / 32864 MiB / 26105 MiB / 352 MiB / 309 MiB ( 64 MiB / 0 MiB / 14 MiB / 0 MiB / 1 MiB )
15 min passed...26144 MiB / 33440 MiB / 26118 MiB / 352 MiB / 309 MiB ( 0 MiB / 576 MiB / 13 MiB / 0 MiB / 0 MiB )
16 min passed...26144 MiB / 33824 MiB / 26133 MiB / 9600 MiB / 9619 MiB ( 0 MiB / 384 MiB / 15 MiB / 9248 MiB / 9310 MiB )
17 min passed...26176 MiB / 34496 MiB / 26147 MiB / 9600 MiB / 9619 MiB ( 32 MiB / 672 MiB / 14 MiB / 0 MiB / 0 MiB )
18 min passed...26208 MiB / 34848 MiB / 26162 MiB / 9600 MiB / 9619 MiB ( 32 MiB / 352 MiB / 15 MiB / 0 MiB / 0 MiB )
19 min passed...26208 MiB / 34880 MiB / 26177 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 32 MiB / 15 MiB / 0 MiB / 1 MiB )
20 min passed...26208 MiB / 35072 MiB / 26187 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 192 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...26208 MiB / 35488 MiB / 26200 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 416 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...26208 MiB / 36096 MiB / 26211 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 608 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...26208 MiB / 36384 MiB / 26223 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 288 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...26208 MiB / 36544 MiB / 26237 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 160 MiB / 14 MiB / 0 MiB / 0 MiB )
25 min passed...26208 MiB / 36704 MiB / 26248 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
26 min passed...26208 MiB / 37280 MiB / 26260 MiB / 9600 MiB / 9620 MiB ( 0 MiB / 576 MiB / 12 MiB / 0 MiB / 0 MiB )
27 min passed...26240 MiB / 37760 MiB / 26273 MiB / 9600 MiB / 9621 MiB ( 32 MiB / 480 MiB / 13 MiB / 0 MiB / 1 MiB )
28 min passed...26336 MiB / 38048 MiB / 26287 MiB / 9600 MiB / 9621 MiB ( 96 MiB / 288 MiB / 14 MiB / 0 MiB / 0 MiB )
29 min passed...26336 MiB / 38272 MiB / 26301 MiB / 9600 MiB / 9621 MiB ( 0 MiB / 224 MiB / 14 MiB / 0 MiB / 0 MiB )
30 min passed...26336 MiB / 38304 MiB / 26316 MiB / 9600 MiB / 9621 MiB ( 0 MiB / 32 MiB / 15 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 19, 2021

Round 2 - Run 7 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=4338: Thu Aug 19 15:30:59 2021
  write: IOPS=30.0k, BW=469MiB/s (492MB/s)(1024MiB/2183msec); 0 zone resets
    slat (usec): min=7, max=2637, avg=20.44, stdev=14.91
    clat (usec): min=74, max=7408, avg=2098.29, stdev=257.29
     lat (usec): min=97, max=7496, avg=2120.14, stdev=260.15
    clat percentiles (usec):
     |  1.00th=[ 1762],  5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 1958],
     | 30.00th=[ 1991], 40.00th=[ 2040], 50.00th=[ 2057], 60.00th=[ 2114],
     | 70.00th=[ 2147], 80.00th=[ 2212], 90.00th=[ 2278], 95.00th=[ 2376],
     | 99.00th=[ 2769], 99.50th=[ 3556], 99.90th=[ 5080], 99.95th=[ 6128],
     | 99.99th=[ 7177]
   bw (  KiB/s): min=453536, max=501696, per=100.00%, avg=481184.00, stdev=20548.40, samples=4
   iops        : min=28346, max=31356, avg=30074.00, stdev=1284.28, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=30.35%, 4=69.29%, 10=0.31%
  cpu          : usr=23.65%, sys=53.21%, ctx=68261, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=469MiB/s (492MB/s), 469MiB/s-469MiB/s (492MB/s-492MB/s), io=1024MiB (1074MB), run=2183-2183msec

Disk stats (read/write):
  sda: ios=0/64376, merge=0/171, ticks=0/3509, in_queue=4636, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=4550: Thu Aug 19 15:46:01 2021
  read: IOPS=66.5k, BW=1039MiB/s (1089MB/s)(1024MiB/986msec)
    slat (usec): min=4, max=116, avg= 8.12, stdev= 4.03
    clat (usec): min=215, max=3643, avg=948.55, stdev=81.74
     lat (usec): min=227, max=3656, avg=957.89, stdev=82.43
    clat percentiles (usec):
     |  1.00th=[  832],  5.00th=[  865], 10.00th=[  881], 20.00th=[  906],
     | 30.00th=[  914], 40.00th=[  930], 50.00th=[  947], 60.00th=[  955],
     | 70.00th=[  971], 80.00th=[  988], 90.00th=[ 1012], 95.00th=[ 1037],
     | 99.00th=[ 1106], 99.50th=[ 1188], 99.90th=[ 1975], 99.95th=[ 2180],
     | 99.99th=[ 2868]
   bw (  KiB/s): min=1047744, max=1047744, per=98.52%, avg=1047744.00, stdev= 0.00, samples=1
   iops        : min=65484, max=65484, avg=65484.00, stdev= 0.00, samples=1
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.03%, 1000=86.02%
  lat (msec)   : 2=13.84%, 4=0.08%
  cpu          : usr=42.94%, sys=57.16%, ctx=11, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1039MiB/s (1089MB/s), 1039MiB/s-1039MiB/s (1089MB/s-1089MB/s), io=1024MiB (1074MB), run=986-986msec

Disk stats (read/write):
  sda: ios=59426/0, merge=0/0, ticks=13708/0, in_queue=13712, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3200 MiB / 3360 MiB / 3182 MiB / 0 MiB / 2 MiB ( 3200 MiB / 3360 MiB / 3182 MiB / 0 MiB / 2 MiB )
2 min passed...3200 MiB / 3616 MiB / 3182 MiB / 0 MiB / 2 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...3200 MiB / 3904 MiB / 3182 MiB / 0 MiB / 2 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...3200 MiB / 3968 MiB / 3182 MiB / 0 MiB / 2 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...3200 MiB / 3968 MiB / 3190 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 1 MiB )
6 min passed...3200 MiB / 3968 MiB / 3190 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...3200 MiB / 4032 MiB / 3190 MiB / 0 MiB / 3 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...3200 MiB / 4288 MiB / 3190 MiB / 0 MiB / 3 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...3200 MiB / 4288 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...3200 MiB / 4288 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...3200 MiB / 4288 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...3200 MiB / 4288 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...3200 MiB / 4416 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...3200 MiB / 4768 MiB / 3194 MiB / 0 MiB / 3 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...3232 MiB / 4768 MiB / 3202 MiB / 480 MiB / 512 MiB ( 32 MiB / 0 MiB / 8 MiB / 480 MiB / 509 MiB )
16 min passed...3232 MiB / 4768 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...3232 MiB / 4768 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...3232 MiB / 4768 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...3232 MiB / 5376 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 608 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...3232 MiB / 5696 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...3232 MiB / 5696 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...3232 MiB / 5696 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...3232 MiB / 5696 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...3232 MiB / 5728 MiB / 3202 MiB / 480 MiB / 512 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...3232 MiB / 6624 MiB / 3209 MiB / 480 MiB / 512 MiB ( 0 MiB / 896 MiB / 7 MiB / 0 MiB / 0 MiB )
26 min passed...3232 MiB / 6912 MiB / 3209 MiB / 480 MiB / 512 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...3232 MiB / 6912 MiB / 3209 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...3232 MiB / 6912 MiB / 3209 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...3232 MiB / 6912 MiB / 3209 MiB / 480 MiB / 512 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...3264 MiB / 7008 MiB / 3218 MiB / 480 MiB / 512 MiB ( 32 MiB / 96 MiB / 9 MiB / 0 MiB / 0 MiB )

	logbias=throughput	logbias=latency
sync 4k random	1,45 MiB/s	1,47 MiB/s
sync 4M sequential	184,00 MiB/s	179,00 MiB/s
4x sync 32K sequential	18,80 MiB/s	19,10 MiB/s

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

We value your privacy