how to best benchmark SSDs?

Dunuin · Aug 22, 2021

Round 2 - Run 8 - Test X7: 4x sync sequential read/write 32K

Guest (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=239MiB/s][r=7637 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6232: Sun Aug 22 01:13:46 2021
  read: IOPS=1930, BW=60.3MiB/s (63.3MB/s)(512MiB/8486msec)
    clat (usec): min=263, max=6307, avg=508.68, stdev=255.40
     lat (usec): min=265, max=6309, avg=510.54, stdev=255.41
    clat percentiles (usec):
     |  1.00th=[  379],  5.00th=[  408], 10.00th=[  420], 20.00th=[  437],
     | 30.00th=[  453], 40.00th=[  465], 50.00th=[  478], 60.00th=[  490],
     | 70.00th=[  502], 80.00th=[  523], 90.00th=[  553], 95.00th=[  594],
     | 99.00th=[ 1844], 99.50th=[ 2737], 99.90th=[ 3294], 99.95th=[ 3589],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=58496, max=66496, per=25.13%, avg=61948.00, stdev=2286.21, samples=16
   iops        : min= 1828, max= 2078, avg=1935.88, stdev=71.44, samples=16
  lat (usec)   : 500=68.35%, 750=29.82%, 1000=0.32%
  lat (msec)   : 2=0.57%, 4=0.90%, 10=0.04%
  cpu          : usr=2.85%, sys=8.71%, ctx=16390, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6233: Sun Aug 22 01:13:46 2021
  read: IOPS=1936, BW=60.5MiB/s (63.5MB/s)(512MiB/8460msec)
    clat (usec): min=282, max=7797, avg=506.96, stdev=237.36
     lat (usec): min=284, max=7799, avg=508.84, stdev=237.38
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  408], 10.00th=[  420], 20.00th=[  441],
     | 30.00th=[  453], 40.00th=[  465], 50.00th=[  478], 60.00th=[  490],
     | 70.00th=[  502], 80.00th=[  523], 90.00th=[  553], 95.00th=[  594],
     | 99.00th=[ 1614], 99.50th=[ 2671], 99.90th=[ 3163], 99.95th=[ 3261],
     | 99.99th=[ 5735]
   bw (  KiB/s): min=59584, max=66496, per=25.22%, avg=62160.50, stdev=1764.35, samples=16
   iops        : min= 1862, max= 2078, avg=1942.50, stdev=55.16, samples=16
  lat (usec)   : 500=67.92%, 750=30.22%, 1000=0.45%
  lat (msec)   : 2=0.52%, 4=0.87%, 10=0.01%
  cpu          : usr=2.88%, sys=8.80%, ctx=16387, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6234: Sun Aug 22 01:13:46 2021
  read: IOPS=1925, BW=60.2MiB/s (63.1MB/s)(512MiB/8507msec)
    clat (usec): min=254, max=8193, avg=510.02, stdev=263.78
     lat (usec): min=255, max=8195, avg=511.86, stdev=263.81
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  404], 10.00th=[  420], 20.00th=[  437],
     | 30.00th=[  453], 40.00th=[  465], 50.00th=[  474], 60.00th=[  490],
     | 70.00th=[  502], 80.00th=[  523], 90.00th=[  553], 95.00th=[  594],
     | 99.00th=[ 1909], 99.50th=[ 2769], 99.90th=[ 3359], 99.95th=[ 4359],
     | 99.99th=[ 5997]
   bw (  KiB/s): min=56256, max=65536, per=25.00%, avg=61632.29, stdev=2231.34, samples=17
   iops        : min= 1758, max= 2048, avg=1926.00, stdev=69.73, samples=17
  lat (usec)   : 500=68.49%, 750=29.47%, 1000=0.45%
  lat (msec)   : 2=0.63%, 4=0.90%, 10=0.06%
  cpu          : usr=3.16%, sys=8.19%, ctx=16388, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6235: Sun Aug 22 01:13:46 2021
  read: IOPS=1930, BW=60.3MiB/s (63.3MB/s)(512MiB/8485msec)
    clat (usec): min=275, max=6092, avg=508.51, stdev=239.63
     lat (usec): min=276, max=6094, avg=510.37, stdev=239.64
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  408], 10.00th=[  424], 20.00th=[  441],
     | 30.00th=[  453], 40.00th=[  465], 50.00th=[  478], 60.00th=[  490],
     | 70.00th=[  506], 80.00th=[  523], 90.00th=[  553], 95.00th=[  594],
     | 99.00th=[ 1663], 99.50th=[ 2606], 99.90th=[ 3228], 99.95th=[ 3818],
     | 99.99th=[ 5538]
   bw (  KiB/s): min=58688, max=65344, per=25.16%, avg=62028.44, stdev=2154.04, samples=16
   iops        : min= 1834, max= 2042, avg=1938.37, stdev=67.33, samples=16
  lat (usec)   : 500=67.38%, 750=30.62%, 1000=0.53%
  lat (msec)   : 2=0.63%, 4=0.81%, 10=0.04%
  cpu          : usr=3.34%, sys=8.47%, ctx=16396, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=241MiB/s (252MB/s), 60.2MiB/s-60.5MiB/s (63.1MB/s-63.5MB/s), io=2048MiB (2147MB), run=8460-8507msec

Disk stats (read/write):
  sda: ios=64918/6, merge=0/3, ticks=30754/4, in_queue=32936, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2780 MiB / 2048 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2781 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Script ran 1900 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_9.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...9824 MiB / 10528 MiB / 9843 MiB / 64 MiB / 119 MiB ( 9824 MiB / 10528 MiB / 9843 MiB / 64 MiB / 119 MiB )
2 min passed...15616 MiB / 16832 MiB / 15617 MiB / 160 MiB / 186 MiB ( 5792 MiB / 6304 MiB / 5774 MiB / 96 MiB / 67 MiB )
3 min passed...15616 MiB / 16960 MiB / 15631 MiB / 160 MiB / 186 MiB ( 0 MiB / 128 MiB / 14 MiB / 0 MiB / 0 MiB )
4 min passed...15616 MiB / 17056 MiB / 15644 MiB / 160 MiB / 186 MiB ( 0 MiB / 96 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...15616 MiB / 17536 MiB / 15654 MiB / 160 MiB / 186 MiB ( 0 MiB / 480 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...15648 MiB / 17664 MiB / 15667 MiB / 160 MiB / 186 MiB ( 32 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
7 min passed...15648 MiB / 17824 MiB / 15678 MiB / 160 MiB / 187 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 1 MiB )
8 min passed...15648 MiB / 17984 MiB / 15689 MiB / 160 MiB / 187 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 0 MiB )
9 min passed...15680 MiB / 18112 MiB / 15702 MiB / 160 MiB / 187 MiB ( 32 MiB / 128 MiB / 13 MiB / 0 MiB / 0 MiB )
10 min passed...15712 MiB / 18656 MiB / 15712 MiB / 160 MiB / 187 MiB ( 32 MiB / 544 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...15712 MiB / 18848 MiB / 15723 MiB / 160 MiB / 187 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
12 min passed...15712 MiB / 18912 MiB / 15734 MiB / 160 MiB / 187 MiB ( 0 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
13 min passed...15712 MiB / 18944 MiB / 15744 MiB / 160 MiB / 187 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
14 min passed...15712 MiB / 19168 MiB / 15755 MiB / 160 MiB / 187 MiB ( 0 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
15 min passed...15712 MiB / 19520 MiB / 15765 MiB / 160 MiB / 187 MiB ( 0 MiB / 352 MiB / 10 MiB / 0 MiB / 0 MiB )
16 min passed...15712 MiB / 19872 MiB / 15776 MiB / 160 MiB / 187 MiB ( 0 MiB / 352 MiB / 11 MiB / 0 MiB / 0 MiB )
17 min passed...15744 MiB / 20224 MiB / 15787 MiB / 2176 MiB / 2236 MiB ( 32 MiB / 352 MiB / 11 MiB / 2016 MiB / 2049 MiB )
18 min passed...15776 MiB / 20288 MiB / 15799 MiB / 2176 MiB / 2236 MiB ( 32 MiB / 64 MiB / 12 MiB / 0 MiB / 0 MiB )
19 min passed...15808 MiB / 20448 MiB / 15811 MiB / 2176 MiB / 2236 MiB ( 32 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...15808 MiB / 20704 MiB / 15824 MiB / 2176 MiB / 2236 MiB ( 0 MiB / 256 MiB / 13 MiB / 0 MiB / 0 MiB )
21 min passed...15840 MiB / 21216 MiB / 15840 MiB / 2176 MiB / 2236 MiB ( 32 MiB / 512 MiB / 16 MiB / 0 MiB / 0 MiB )
22 min passed...15840 MiB / 21216 MiB / 15851 MiB / 2176 MiB / 2236 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...15840 MiB / 21248 MiB / 15863 MiB / 2176 MiB / 2236 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...15904 MiB / 21248 MiB / 15876 MiB / 2176 MiB / 2236 MiB ( 64 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...15904 MiB / 21280 MiB / 15890 MiB / 2176 MiB / 2236 MiB ( 0 MiB / 32 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...15904 MiB / 21888 MiB / 15903 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 608 MiB / 13 MiB / 0 MiB / 1 MiB )
27 min passed...15904 MiB / 21952 MiB / 15915 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 64 MiB / 12 MiB / 0 MiB / 0 MiB )
28 min passed...15904 MiB / 21984 MiB / 15928 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 32 MiB / 13 MiB / 0 MiB / 0 MiB )
29 min passed...15904 MiB / 22112 MiB / 15942 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 128 MiB / 14 MiB / 0 MiB / 0 MiB )
30 min passed...15904 MiB / 22240 MiB / 15954 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 128 MiB / 12 MiB / 0 MiB / 0 MiB )
31 min passed...15904 MiB / 22560 MiB / 15968 MiB / 2176 MiB / 2237 MiB ( 0 MiB / 320 MiB / 14 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 24, 2021

I wanted to see if adding a SLOG might reduce the write amplification:

Total Write Amplification (factor):

	5 disk raidz1 without slog	5 disk raidz1 + 1 disk slog
T1: sync 4K random	44,88	35,28
T2: sync 16K random	13,79	11,66
T8: sync 32K random	11,48	8,66
T3: sync 4M sequential	3,04	2,76
TX7: 4x sync 32K sequential on host	8,34	7,63

So looks like adding a SLOG helps to lower write amplification. And yes, the writes to the slog are included in the total write amplification.

For performance I'm not sure how comparable they are. The tests wouthout slog where created of a raidz1 of 4x S3710 + 1x S3700 and the tests with slog used 6x S3710. And the S3710 is on paper faster than the S3700.

Write Performance (MiB/s):

	5 disk raidz1 without slog	5 disk raidz1 + 1 disk slog
T1: sync 4K random	1,17	1,26
T2: sync 16K random	5,50	5,40
T8: sync 32K random	12,40	11,60
T3: sync 4M sequential	262,00	183,00
TX7: 4x sync 32K sequential on host	34,10	32,20

So here a SLOG doesn't help with performance at all.

guletz · Aug 26, 2021

Dunuin said:
I wanted to see if adding a SLOG might reduce the write amplification:

Total Write Amplification (factor):

5 disk raidz1 without slog 5 disk raidz1 + 1 disk slog
T1: sync 4K random 44,88 35,28
T2: sync 16K random 13,79 11,66
T8: sync 32K random 11,48 8,66
T3: sync 4M sequential 3,04 2,76
TX7: 4x sync 32K sequential on host 8,34 7,63

So looks like adding a SLOG helps to lower write amplification. And yes, the writes to the slog are included in the total write amplification.

For performance I'm not sure how comparable they are. The tests wouthout slog where created of a raidz1 of 4x S3710 + 1x S3700 and the tests with slog used 6x S3710. And the S3710 is on paper faster than the S3700.

Write Performance (MiB/s):

5 disk raidz1 without slog 5 disk raidz1 + 1 disk slog
T1: sync 4K random 1,17 1,26
T2: sync 16K random 5,50 5,40
T8: sync 32K random 12,40 11,60
T3: sync 4M sequential 262,00 183,00
TX7: 4x sync 32K sequential on host 34,10 32,20

So here a SLOG doesn't help with performance at all.

Hi @Dunuin

Thx. a lot for your tests, conclusions and time that you spent for many of us.

Congratulations!

Good luck / Bafta !

Dunuin · Aug 26, 2021

Dunuin said:
guletz said:

Hi @Dunuin

Thx. a lot for your tests, conclusions and time that you spent for many of us.

Congratulations!

Good luck / Bafta !

Click to expand...

Thanks. Right now I'm only missing a LUKS encrypted LVM benchmark for camparison with ZFS. Then I will choose how I should use my 8 SSDs and import my VMs again. After that I will try to create a new Thread where I will summarize the results and create some more diagrams.

Dunuin · Aug 28, 2021

This is the LVM thin and VM that will be used in Round 2 Run 9:

Host is a single disk LUKS encrypted LVM thin of 1x S3710 200GB.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 8 are bold)

Commands used to create LVM thin:

Code:

# create LUKS encrypted partition
cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN-part1
# unlock partition
cryptsetup luksOpen /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN-part1 lukslvm
# create LVM
pvcreate /dev/mapper/lukslvm
vgcreate vgluks /dev/mapper/lukslvm
# create LVM thin
lvcreate -l99%FREE -n lvthin vgluks
lvconvert --type thin-pool vgluks/lvthin
# manually added to PVE as LVM thin storage using WebUI

LVM infos:

Code:

lvdisplay vgluks/lvthin
  --- Logical volume ---
  LV Name                lvthin
  VG Name                vgluks
  LV UUID                4czlTI-O0Tk-OJfA-zkiV-4HHO-HsXz-EFRCEi
  LV Write Access        read/write
  LV Creation host, time Hypervisor, 2021-08-28 20:13:41 +0200
  LV Pool metadata       lvthin_tmeta
  LV Pool data           lvthin_tdata
  LV Status              available
  # open                 3
  LV Size                <184.43 GiB
  Allocated pool data    1.43%
  Allocated metadata     11.05%
  Current LE             47214
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:6


vgdisplay vgluks
  --- Volume group ---
  VG Name               vgluks
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  11
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               186.29 GiB
  PE Size               4.00 MiB
  Total PE              47691
  Alloc PE / Size       47262 / <184.62 GiB
  Free  PE / Size       429 / <1.68 GiB
  VG UUID               RGMn5D-N3ng-a4e8-c0P5-1zWQ-LhfI-B8ZVwe
 
  pvdisplay /dev/mapper/lukslvm
  --- Physical volume ---
  PV Name               /dev/mapper/lukslvm
  VG Name               vgluks
  PV Size               186.29 GiB / not usable 1.19 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              47691
  Free PE               429
  Allocated PE          47262
  PV UUID               HkV5DJ-83KS-fByM-XslJ-8iJZ-8USC-wmXATC

Dunuin · Aug 28, 2021

Round 2 - Run 9 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=2698KiB/s][w=674 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=693: Sun Aug 29 17:01:25 2021
  write: IOPS=727, BW=2911KiB/s (2981kB/s)(1024MiB/360161msec); 0 zone resets
    clat (usec): min=581, max=6781, avg=1360.18, stdev=457.51
     lat (usec): min=582, max=6783, avg=1362.24, stdev=457.60
    clat percentiles (usec):
     |  1.00th=[  799],  5.00th=[  865], 10.00th=[  922], 20.00th=[ 1029],
     | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1270], 60.00th=[ 1352],
     | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1876], 95.00th=[ 2147],
     | 99.00th=[ 3261], 99.50th=[ 3458], 99.90th=[ 3752], 99.95th=[ 3884],
     | 99.99th=[ 4883]
   bw (  KiB/s): min= 1216, max= 5680, per=99.99%, avg=2910.84, stdev=613.76, samples=720
   iops        : min=  304, max= 1420, avg=727.70, stdev=153.44, samples=720
  lat (usec)   : 750=0.29%, 1000=17.14%
  lat (msec)   : 2=75.31%, 4=7.23%, 10=0.03%
  cpu          : usr=1.70%, sys=10.04%, ctx=594251, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2911KiB/s (2981kB/s), 2911KiB/s-2911KiB/s (2981kB/s-2981kB/s), io=1024MiB (1074MB), run=360161-360161msec

Disk stats (read/write):
  sda: ios=0/800205, merge=0/531924, ticks=0/303760, in_queue=360452, util=99.89%
0 min passed...Guest Writes/Reads (Changes): 5187 MiB / 17 MiB ( 5187 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5188 MiB / 17 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=16.3MiB/s][r=4171 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=898: Sun Aug 29 17:17:31 2021
  read: IOPS=4028, BW=15.7MiB/s (16.5MB/s)(1024MiB/65067msec)
    clat (usec): min=142, max=3320, avg=240.07, stdev=63.52
     lat (usec): min=143, max=3322, avg=241.70, stdev=63.55
    clat percentiles (usec):
     |  1.00th=[  202],  5.00th=[  208], 10.00th=[  212], 20.00th=[  217],
     | 30.00th=[  223], 40.00th=[  227], 50.00th=[  231], 60.00th=[  235],
     | 70.00th=[  241], 80.00th=[  255], 90.00th=[  285], 95.00th=[  302],
     | 99.00th=[  318], 99.50th=[  338], 99.90th=[  799], 99.95th=[ 1795],
     | 99.99th=[ 2835]
   bw (  KiB/s): min=13840, max=18307, per=99.99%, avg=16113.66, stdev=862.00, samples=130
   iops        : min= 3460, max= 4576, avg=4028.39, stdev=215.48, samples=130
  lat (usec)   : 250=77.29%, 500=22.55%, 750=0.05%, 1000=0.03%
  lat (msec)   : 2=0.04%, 4=0.04%
  cpu          : usr=5.49%, sys=11.56%, ctx=262161, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=15.7MiB/s (16.5MB/s), 15.7MiB/s-15.7MiB/s (16.5MB/s-16.5MB/s), io=1024MiB (1074MB), run=65067-65067msec

Disk stats (read/write):
  sda: ios=261440/2, merge=0/6, ticks=57227/2, in_queue=64984, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5188 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Script ran 2228 seconds or 37 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1952 MiB / 2080 MiB / 1951 MiB / 32 MiB / 21 MiB ( 1952 MiB / 2080 MiB / 1951 MiB / 32 MiB / 21 MiB )
2 min passed...2912 MiB / 3168 MiB / 2896 MiB / 32 MiB / 25 MiB ( 960 MiB / 1088 MiB / 945 MiB / 0 MiB / 4 MiB )
3 min passed...3776 MiB / 4352 MiB / 3765 MiB / 32 MiB / 30 MiB ( 864 MiB / 1184 MiB / 869 MiB / 0 MiB / 5 MiB )
4 min passed...4704 MiB / 5312 MiB / 4691 MiB / 32 MiB / 34 MiB ( 928 MiB / 960 MiB / 926 MiB / 0 MiB / 4 MiB )
5 min passed...5600 MiB / 6592 MiB / 5586 MiB / 32 MiB / 39 MiB ( 896 MiB / 1280 MiB / 895 MiB / 0 MiB / 5 MiB )
6 min passed...6496 MiB / 7616 MiB / 6503 MiB / 32 MiB / 43 MiB ( 896 MiB / 1024 MiB / 917 MiB / 0 MiB / 4 MiB )
7 min passed...6528 MiB / 7840 MiB / 6534 MiB / 64 MiB / 49 MiB ( 32 MiB / 224 MiB / 31 MiB / 32 MiB / 6 MiB )
8 min passed...6528 MiB / 8000 MiB / 6534 MiB / 64 MiB / 54 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...6528 MiB / 8000 MiB / 6534 MiB / 64 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...6528 MiB / 8000 MiB / 6534 MiB / 64 MiB / 63 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...6528 MiB / 8064 MiB / 6534 MiB / 64 MiB / 67 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...6528 MiB / 8288 MiB / 6534 MiB / 64 MiB / 72 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...6528 MiB / 8288 MiB / 6534 MiB / 64 MiB / 76 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...6528 MiB / 8288 MiB / 6534 MiB / 96 MiB / 81 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
15 min passed...6528 MiB / 8288 MiB / 6534 MiB / 96 MiB / 85 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...6528 MiB / 8320 MiB / 6534 MiB / 96 MiB / 90 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 5 MiB )
17 min passed...6528 MiB / 8320 MiB / 6534 MiB / 96 MiB / 94 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
18 min passed...6528 MiB / 8320 MiB / 6534 MiB / 96 MiB / 99 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
19 min passed...6528 MiB / 8320 MiB / 6534 MiB / 96 MiB / 103 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
20 min passed...6528 MiB / 8384 MiB / 6534 MiB / 96 MiB / 108 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
21 min passed...6528 MiB / 8608 MiB / 6534 MiB / 128 MiB / 112 MiB ( 0 MiB / 224 MiB / 0 MiB / 32 MiB / 4 MiB )
22 min passed...6528 MiB / 8640 MiB / 6534 MiB / 1056 MiB / 1042 MiB ( 0 MiB / 32 MiB / 0 MiB / 928 MiB / 930 MiB )
23 min passed...6528 MiB / 8640 MiB / 6534 MiB / 1152 MiB / 1145 MiB ( 0 MiB / 0 MiB / 0 MiB / 96 MiB / 103 MiB )
24 min passed...6528 MiB / 8640 MiB / 6534 MiB / 1152 MiB / 1150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
25 min passed...6528 MiB / 8896 MiB / 6534 MiB / 1152 MiB / 1154 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
26 min passed...6528 MiB / 8992 MiB / 6534 MiB / 1152 MiB / 1159 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )
27 min passed...6528 MiB / 8992 MiB / 6534 MiB / 1152 MiB / 1163 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
28 min passed...6528 MiB / 8992 MiB / 6534 MiB / 1184 MiB / 1168 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
29 min passed...6528 MiB / 9088 MiB / 6534 MiB / 1184 MiB / 1172 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
30 min passed...6528 MiB / 9344 MiB / 6534 MiB / 1184 MiB / 1177 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
31 min passed...6528 MiB / 9344 MiB / 6534 MiB / 1184 MiB / 1181 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
32 min passed...6528 MiB / 9344 MiB / 6534 MiB / 1184 MiB / 1186 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
33 min passed...6528 MiB / 9344 MiB / 6534 MiB / 1184 MiB / 1190 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
34 min passed...6528 MiB / 9536 MiB / 6534 MiB / 1184 MiB / 1195 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
35 min passed...6528 MiB / 9664 MiB / 6534 MiB / 1216 MiB / 1200 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 5 MiB )
36 min passed...6528 MiB / 9664 MiB / 6534 MiB / 1216 MiB / 1204 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
37 min passed...6528 MiB / 9664 MiB / 6534 MiB / 1216 MiB / 1209 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )

Dunuin · Aug 28, 2021

Round 2 - Run 9 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=11.9MiB/s][w=759 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1133: Sun Aug 29 17:38:07 2021
  write: IOPS=622, BW=9960KiB/s (10.2MB/s)(1024MiB/105274msec); 0 zone resets
    clat (usec): min=764, max=7086, avg=1587.69, stdev=626.15
     lat (usec): min=766, max=7088, avg=1589.75, stdev=626.22
    clat percentiles (usec):
     |  1.00th=[  873],  5.00th=[  930], 10.00th=[ 1004], 20.00th=[ 1139],
     | 30.00th=[ 1237], 40.00th=[ 1319], 50.00th=[ 1401], 60.00th=[ 1500],
     | 70.00th=[ 1631], 80.00th=[ 1860], 90.00th=[ 2737], 95.00th=[ 3097],
     | 99.00th=[ 3490], 99.50th=[ 3654], 99.90th=[ 3884], 99.95th=[ 3982],
     | 99.99th=[ 6259]
   bw (  KiB/s): min= 5440, max=16512, per=100.00%, avg=9960.27, stdev=2424.25, samples=210
   iops        : min=  340, max= 1032, avg=622.50, stdev=151.51, samples=210
  lat (usec)   : 1000=9.70%
  lat (msec)   : 2=73.70%, 4=16.56%, 10=0.05%
  cpu          : usr=1.99%, sys=8.11%, ctx=151447, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9960KiB/s (10.2MB/s), 9960KiB/s-9960KiB/s (10.2MB/s-10.2MB/s), io=1024MiB (1074MB), run=105274-105274msec

Disk stats (read/write):
  sda: ios=0/196806, merge=0/131994, ticks=0/90424, in_queue=105016, util=99.85%
0 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 2053 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2053 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2054 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=38.5MiB/s][r=2465 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=1333: Sun Aug 29 17:53:33 2021
  read: IOPS=2563, BW=40.0MiB/s (41.0MB/s)(1024MiB/25569msec)
    clat (usec): min=254, max=2961, avg=380.31, stdev=60.48
     lat (usec): min=256, max=2963, avg=382.13, stdev=60.60
    clat percentiles (usec):
     |  1.00th=[  285],  5.00th=[  302], 10.00th=[  310], 20.00th=[  326],
     | 30.00th=[  343], 40.00th=[  359], 50.00th=[  379], 60.00th=[  396],
     | 70.00th=[  412], 80.00th=[  424], 90.00th=[  453], 95.00th=[  478],
     | 99.00th=[  553], 99.50th=[  578], 99.90th=[  611], 99.95th=[  627],
     | 99.99th=[  889]
   bw (  KiB/s): min=35200, max=51520, per=99.99%, avg=41004.33, stdev=3826.31, samples=51
   iops        : min= 2200, max= 3220, avg=2562.75, stdev=239.15, samples=51
  lat (usec)   : 500=96.37%, 750=3.61%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=4.35%, sys=12.83%, ctx=65549, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.0MiB/s (41.0MB/s), 40.0MiB/s-40.0MiB/s (41.0MB/s-41.0MB/s), io=1024MiB (1074MB), run=25569-25569msec

Disk stats (read/write):
  sda: ios=65068/0, merge=0/0, ticks=22372/0, in_queue=25472, util=99.98%
0 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2054 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1932 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2176 MiB / 2368 MiB / 2172 MiB / 32 MiB / 5 MiB ( 2176 MiB / 2368 MiB / 2172 MiB / 32 MiB / 5 MiB )
2 min passed...3296 MiB / 3840 MiB / 3295 MiB / 32 MiB / 9 MiB ( 1120 MiB / 1472 MiB / 1123 MiB / 0 MiB / 4 MiB )
3 min passed...3296 MiB / 3904 MiB / 3295 MiB / 32 MiB / 14 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
4 min passed...3296 MiB / 3904 MiB / 3295 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
5 min passed...3296 MiB / 3904 MiB / 3295 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
6 min passed...3296 MiB / 3936 MiB / 3295 MiB / 32 MiB / 27 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
7 min passed...3296 MiB / 4224 MiB / 3295 MiB / 32 MiB / 32 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 5 MiB )
8 min passed...3296 MiB / 4256 MiB / 3295 MiB / 64 MiB / 36 MiB ( 0 MiB / 32 MiB / 0 MiB / 32 MiB / 4 MiB )
9 min passed...3296 MiB / 4256 MiB / 3295 MiB / 64 MiB / 41 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
10 min passed...3296 MiB / 4256 MiB / 3295 MiB / 64 MiB / 45 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
11 min passed...3296 MiB / 4416 MiB / 3295 MiB / 64 MiB / 50 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
12 min passed...3296 MiB / 4576 MiB / 3295 MiB / 64 MiB / 54 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
13 min passed...3296 MiB / 4576 MiB / 3295 MiB / 64 MiB / 59 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
14 min passed...3296 MiB / 4576 MiB / 3295 MiB / 64 MiB / 63 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
15 min passed...3296 MiB / 4608 MiB / 3295 MiB / 96 MiB / 68 MiB ( 0 MiB / 32 MiB / 0 MiB / 32 MiB / 5 MiB )
16 min passed...3296 MiB / 4736 MiB / 3295 MiB / 96 MiB / 72 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
17 min passed...3296 MiB / 4736 MiB / 3295 MiB / 544 MiB / 529 MiB ( 0 MiB / 0 MiB / 0 MiB / 448 MiB / 457 MiB )
18 min passed...3296 MiB / 4736 MiB / 3295 MiB / 1120 MiB / 1105 MiB ( 0 MiB / 0 MiB / 0 MiB / 576 MiB / 576 MiB )
19 min passed...3296 MiB / 4736 MiB / 3295 MiB / 1120 MiB / 1110 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...3296 MiB / 4896 MiB / 3295 MiB / 1120 MiB / 1114 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...3296 MiB / 5024 MiB / 3295 MiB / 1120 MiB / 1119 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 5 MiB )
22 min passed...3296 MiB / 5024 MiB / 3295 MiB / 1120 MiB / 1123 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...3296 MiB / 5024 MiB / 3295 MiB / 1152 MiB / 1128 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
24 min passed...3296 MiB / 5088 MiB / 3295 MiB / 1152 MiB / 1132 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...3296 MiB / 5344 MiB / 3295 MiB / 1152 MiB / 1137 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...3296 MiB / 5344 MiB / 3295 MiB / 1152 MiB / 1141 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
27 min passed...3296 MiB / 5344 MiB / 3295 MiB / 1152 MiB / 1146 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...3296 MiB / 5344 MiB / 3295 MiB / 1152 MiB / 1150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
29 min passed...3296 MiB / 5536 MiB / 3295 MiB / 1152 MiB / 1155 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
30 min passed...3296 MiB / 5696 MiB / 3296 MiB / 1184 MiB / 1159 MiB ( 0 MiB / 160 MiB / 1 MiB / 32 MiB / 4 MiB )
31 min passed...3296 MiB / 5696 MiB / 3296 MiB / 1184 MiB / 1164 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
32 min passed...3296 MiB / 5696 MiB / 3296 MiB / 1184 MiB / 1168 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )

Dunuin · Aug 28, 2021

Round 2 - Run 9 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=236MiB/s][w=59 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1578: Sun Aug 29 18:35:06 2021
  write: IOPS=58, BW=236MiB/s (247MB/s)(10.0GiB/43470msec); 0 zone resets
    clat (usec): min=10901, max=21313, avg=15302.14, stdev=1208.51
     lat (usec): min=10904, max=21315, avg=15304.90, stdev=1208.50
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[13829], 10.00th=[13960], 20.00th=[14222],
     | 30.00th=[14615], 40.00th=[14877], 50.00th=[15139], 60.00th=[15401],
     | 70.00th=[15664], 80.00th=[16450], 90.00th=[16909], 95.00th=[17433],
     | 99.00th=[18744], 99.50th=[19268], 99.90th=[20317], 99.95th=[20579],
     | 99.99th=[21365]
   bw (  KiB/s): min=229376, max=253952, per=99.98%, avg=241176.30, stdev=6473.75, samples=86
   iops        : min=   56, max=   62, avg=58.86, stdev= 1.57, samples=86
  lat (msec)   : 20=99.80%, 50=0.20%
  cpu          : usr=10.08%, sys=2.06%, ctx=6270, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=236MiB/s (247MB/s), 236MiB/s-236MiB/s (247MB/s-247MB/s), io=10.0GiB (10.7GB), run=43470-43470msec

Disk stats (read/write):
  sda: ios=0/15344, merge=0/5291, ticks=0/98685, in_queue=102956, util=99.10%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 10281 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=424MiB/s][r=106 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=1781: Sun Aug 29 18:50:30 2021
  read: IOPS=106, BW=427MiB/s (448MB/s)(10.0GiB/23955msec)
    clat (usec): min=8267, max=10750, avg=9345.99, stdev=227.26
     lat (usec): min=8268, max=10752, avg=9348.05, stdev=227.28
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 8979], 10.00th=[ 9110], 20.00th=[ 9241],
     | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9372], 60.00th=[ 9372],
     | 70.00th=[ 9503], 80.00th=[ 9503], 90.00th=[ 9634], 95.00th=[ 9634],
     | 99.00th=[ 9765], 99.50th=[ 9896], 99.90th=[10552], 99.95th=[10683],
     | 99.99th=[10814]
   bw (  KiB/s): min=433309, max=458752, per=99.97%, avg=437587.49, stdev=5850.70, samples=47
   iops        : min=  105, max=  112, avg=106.77, stdev= 1.45, samples=47
  lat (msec)   : 10=99.65%, 20=0.35%
  cpu          : usr=0.06%, sys=3.64%, ctx=2569, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=10.0GiB (10.7GB), run=23955-23955msec

Disk stats (read/write):
  sda: ios=10215/0, merge=0/0, ticks=69956/0, in_queue=70056, util=99.34%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10281 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1868 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...10400 MiB / 11008 MiB / 10382 MiB / 0 MiB / 4 MiB ( 10400 MiB / 11008 MiB / 10382 MiB / 0 MiB / 4 MiB )
2 min passed...10400 MiB / 11008 MiB / 10382 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
3 min passed...10400 MiB / 11264 MiB / 10382 MiB / 0 MiB / 13 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...10400 MiB / 11328 MiB / 10382 MiB / 0 MiB / 18 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...10400 MiB / 11328 MiB / 10382 MiB / 0 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...10400 MiB / 11328 MiB / 10382 MiB / 0 MiB / 27 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...10400 MiB / 11488 MiB / 10382 MiB / 32 MiB / 31 MiB ( 0 MiB / 160 MiB / 0 MiB / 32 MiB / 4 MiB )
8 min passed...10400 MiB / 11680 MiB / 10382 MiB / 32 MiB / 36 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...10400 MiB / 11680 MiB / 10382 MiB / 32 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...10400 MiB / 11680 MiB / 10382 MiB / 32 MiB / 45 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...10400 MiB / 11712 MiB / 10382 MiB / 32 MiB / 49 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...10400 MiB / 11968 MiB / 10382 MiB / 32 MiB / 54 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...10400 MiB / 11968 MiB / 10382 MiB / 32 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...10400 MiB / 11968 MiB / 10382 MiB / 64 MiB / 63 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
15 min passed...10400 MiB / 11968 MiB / 10382 MiB / 64 MiB / 67 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...10400 MiB / 12064 MiB / 10382 MiB / 5632 MiB / 5637 MiB ( 0 MiB / 96 MiB / 0 MiB / 5568 MiB / 5570 MiB )
17 min passed...10400 MiB / 12064 MiB / 10382 MiB / 10304 MiB / 10319 MiB ( 0 MiB / 0 MiB / 0 MiB / 4672 MiB / 4682 MiB )
18 min passed...10400 MiB / 12064 MiB / 10382 MiB / 10304 MiB / 10323 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
19 min passed...10400 MiB / 12064 MiB / 10382 MiB / 10304 MiB / 10328 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...10400 MiB / 12096 MiB / 10382 MiB / 10304 MiB / 10332 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...10400 MiB / 12384 MiB / 10382 MiB / 10336 MiB / 10337 MiB ( 0 MiB / 288 MiB / 0 MiB / 32 MiB / 5 MiB )
22 min passed...10400 MiB / 12384 MiB / 10382 MiB / 10336 MiB / 10341 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...10400 MiB / 12384 MiB / 10382 MiB / 10336 MiB / 10346 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
24 min passed...10400 MiB / 12384 MiB / 10382 MiB / 10336 MiB / 10350 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...10400 MiB / 12448 MiB / 10382 MiB / 10336 MiB / 10355 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...10400 MiB / 12448 MiB / 10382 MiB / 10336 MiB / 10359 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
27 min passed...10400 MiB / 12448 MiB / 10382 MiB / 10336 MiB / 10364 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...10400 MiB / 12448 MiB / 10382 MiB / 10368 MiB / 10368 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
29 min passed...10400 MiB / 12480 MiB / 10382 MiB / 10368 MiB / 10373 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 5 MiB )
30 min passed...10400 MiB / 12736 MiB / 10382 MiB / 10368 MiB / 10377 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
31 min passed...10400 MiB / 12768 MiB / 10382 MiB / 10368 MiB / 10382 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 5 MiB )

Dunuin · Aug 28, 2021

Round 2 - Run 9 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=149MiB/s][w=38.1k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2012: Sun Aug 29 19:07:30 2021
  write: IOPS=25.2k, BW=98.4MiB/s (103MB/s)(1024MiB/10406msec); 0 zone resets
    slat (usec): min=6, max=5578, avg=21.77, stdev=21.54
    clat (usec): min=65, max=38971, avg=2512.22, stdev=3299.45
     lat (usec): min=82, max=38990, avg=2535.38, stdev=3302.39
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1549], 10.00th=[ 1582], 20.00th=[ 1631],
     | 30.00th=[ 1663], 40.00th=[ 1696], 50.00th=[ 1729], 60.00th=[ 1795],
     | 70.00th=[ 1926], 80.00th=[ 2040], 90.00th=[ 2802], 95.00th=[ 5538],
     | 99.00th=[23200], 99.50th=[27395], 99.90th=[31851], 99.95th=[33817],
     | 99.99th=[38011]
   bw (  KiB/s): min= 9912, max=152960, per=98.26%, avg=99008.00, stdev=56066.74, samples=20
   iops        : min= 2478, max=38240, avg=24752.00, stdev=14016.69, samples=20
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=76.82%, 4=16.62%, 10=3.50%, 20=1.84%, 50=1.19%
  cpu          : usr=11.38%, sys=46.33%, ctx=255788, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=98.4MiB/s (103MB/s), 98.4MiB/s-98.4MiB/s (103MB/s-103MB/s), io=1024MiB (1074MB), run=10406-10406msec

Disk stats (read/write):
  sda: ios=0/261637, merge=0/1162, ticks=0/228765, in_queue=227080, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 0 MiB ( 1029 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 4 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1033 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=228MiB/s][r=58.4k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=2327: Sun Aug 29 19:22:35 2021
  read: IOPS=59.8k, BW=234MiB/s (245MB/s)(1024MiB/4384msec)
    slat (usec): min=3, max=118, avg= 4.88, stdev= 2.94
    clat (usec): min=485, max=3242, avg=1061.37, stdev=161.28
     lat (usec): min=490, max=3251, avg=1067.08, stdev=161.21
    clat percentiles (usec):
     |  1.00th=[  734],  5.00th=[  832], 10.00th=[  889], 20.00th=[  947],
     | 30.00th=[  979], 40.00th=[ 1012], 50.00th=[ 1045], 60.00th=[ 1074],
     | 70.00th=[ 1106], 80.00th=[ 1172], 90.00th=[ 1254], 95.00th=[ 1352],
     | 99.00th=[ 1549], 99.50th=[ 1663], 99.90th=[ 1958], 99.95th=[ 2114],
     | 99.99th=[ 2638]
   bw (  KiB/s): min=213472, max=253848, per=100.00%, avg=242126.00, stdev=16183.94, samples=8
   iops        : min=53368, max=63462, avg=60531.50, stdev=4045.99, samples=8
  lat (usec)   : 500=0.01%, 750=1.39%, 1000=34.49%
  lat (msec)   : 2=64.03%, 4=0.08%
  cpu          : usr=31.76%, sys=36.30%, ctx=10461, majf=0, minf=72
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=1024MiB (1074MB), run=4384-4384msec

Disk stats (read/write):
  sda: ios=249460/0, merge=0/0, ticks=228838/0, in_queue=229292, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1033 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1817 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2048 MiB / 2304 MiB / 2036 MiB / 0 MiB / 5 MiB ( 2048 MiB / 2304 MiB / 2036 MiB / 0 MiB / 5 MiB )
2 min passed...2048 MiB / 2464 MiB / 2036 MiB / 32 MiB / 9 MiB ( 0 MiB / 160 MiB / 0 MiB / 32 MiB / 4 MiB )
3 min passed...2048 MiB / 2464 MiB / 2036 MiB / 32 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
4 min passed...2048 MiB / 2464 MiB / 2036 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
5 min passed...2048 MiB / 2528 MiB / 2036 MiB / 32 MiB / 23 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
6 min passed...2048 MiB / 2816 MiB / 2036 MiB / 32 MiB / 27 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
7 min passed...2048 MiB / 2816 MiB / 2036 MiB / 32 MiB / 32 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
8 min passed...2048 MiB / 2816 MiB / 2036 MiB / 32 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
9 min passed...2048 MiB / 2816 MiB / 2036 MiB / 64 MiB / 41 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
10 min passed...2048 MiB / 2976 MiB / 2036 MiB / 64 MiB / 45 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
11 min passed...2048 MiB / 3136 MiB / 2036 MiB / 64 MiB / 50 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
12 min passed...2048 MiB / 3136 MiB / 2036 MiB / 64 MiB / 54 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
13 min passed...2048 MiB / 3136 MiB / 2036 MiB / 64 MiB / 59 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
14 min passed...2048 MiB / 3168 MiB / 2036 MiB / 64 MiB / 63 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
15 min passed...2048 MiB / 3360 MiB / 2036 MiB / 64 MiB / 68 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
16 min passed...2048 MiB / 3360 MiB / 2036 MiB / 1120 MiB / 1096 MiB ( 0 MiB / 0 MiB / 0 MiB / 1056 MiB / 1028 MiB )
17 min passed...2048 MiB / 3360 MiB / 2036 MiB / 1120 MiB / 1101 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
18 min passed...2048 MiB / 3360 MiB / 2036 MiB / 1120 MiB / 1105 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
19 min passed...2048 MiB / 3520 MiB / 2036 MiB / 1120 MiB / 1110 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...2048 MiB / 3680 MiB / 2036 MiB / 1120 MiB / 1114 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...2048 MiB / 3680 MiB / 2036 MiB / 1120 MiB / 1119 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
22 min passed...2048 MiB / 3680 MiB / 2036 MiB / 1120 MiB / 1123 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...2048 MiB / 3712 MiB / 2036 MiB / 1152 MiB / 1128 MiB ( 0 MiB / 32 MiB / 0 MiB / 32 MiB / 5 MiB )
24 min passed...2048 MiB / 4000 MiB / 2036 MiB / 1152 MiB / 1132 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...2048 MiB / 4000 MiB / 2036 MiB / 1152 MiB / 1137 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...2048 MiB / 4000 MiB / 2036 MiB / 1152 MiB / 1141 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
27 min passed...2048 MiB / 4000 MiB / 2036 MiB / 1152 MiB / 1146 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...2048 MiB / 4224 MiB / 2037 MiB / 1152 MiB / 1150 MiB ( 0 MiB / 224 MiB / 1 MiB / 0 MiB / 4 MiB )
29 min passed...2048 MiB / 4352 MiB / 2037 MiB / 1152 MiB / 1155 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 5 MiB )
30 min passed...2048 MiB / 4352 MiB / 2037 MiB / 1184 MiB / 1159 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=288MiB/s][w=18.4k IOPS][eta 00m:00s]
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2568: Sun Aug 29 19:56:12 2021
  write: IOPS=10.3k, BW=160MiB/s (168MB/s)(1024MiB/6388msec); 0 zone resets
    slat (usec): min=7, max=5768, avg=29.37, stdev=29.35
    clat (usec): min=804, max=42614, avg=6194.06, stdev=5756.64
     lat (usec): min=834, max=42642, avg=6224.93, stdev=5761.50
    clat percentiles (usec):
     |  1.00th=[ 1631],  5.00th=[ 2057], 10.00th=[ 2343], 20.00th=[ 2737],
     | 30.00th=[ 3163], 40.00th=[ 3589], 50.00th=[ 4113], 60.00th=[ 4752],
     | 70.00th=[ 5669], 80.00th=[ 7832], 90.00th=[12780], 95.00th=[21627],
     | 99.00th=[28443], 99.50th=[31327], 99.90th=[34866], 99.95th=[35914],
     | 99.99th=[38011]
   bw (  KiB/s): min=42272, max=314592, per=95.00%, avg=155941.33, stdev=101577.44, samples=12
   iops        : min= 2642, max=19662, avg=9746.33, stdev=6348.59, samples=12
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=4.09%, 4=43.75%, 10=37.59%, 20=8.87%, 50=5.70%
  cpu          : usr=11.29%, sys=25.93%, ctx=64256, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1024MiB (1074MB), run=6388-6388msec

Disk stats (read/write):
  sda: ios=0/61062, merge=0/166, ticks=0/365273, in_queue=366212, util=99.68%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=2840: Sun Aug 29 20:11:15 2021
  read: IOPS=28.3k, BW=442MiB/s (463MB/s)(1024MiB/2317msec)
    slat (usec): min=3, max=146, avg= 7.84, stdev= 3.91
    clat (usec): min=317, max=17972, avg=2249.34, stdev=933.61
     lat (usec): min=331, max=17977, avg=2258.08, stdev=933.60
    clat percentiles (usec):
     |  1.00th=[  914],  5.00th=[ 1172], 10.00th=[ 1319], 20.00th=[ 1549],
     | 30.00th=[ 1745], 40.00th=[ 1942], 50.00th=[ 2114], 60.00th=[ 2311],
     | 70.00th=[ 2540], 80.00th=[ 2769], 90.00th=[ 3130], 95.00th=[ 3687],
     | 99.00th=[ 5800], 99.50th=[ 6718], 99.90th=[ 9372], 99.95th=[10945],
     | 99.99th=[14484]
   bw (  KiB/s): min=448512, max=455680, per=99.81%, avg=451712.00, stdev=3325.13, samples=4
   iops        : min=28032, max=28480, avg=28232.00, stdev=207.82, samples=4
  lat (usec)   : 500=0.07%, 750=0.26%, 1000=1.43%
  lat (msec)   : 2=41.43%, 4=53.07%, 10=3.67%, 20=0.07%
  cpu          : usr=17.18%, sys=32.73%, ctx=37497, majf=0, minf=263
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=442MiB/s (463MB/s), 442MiB/s-442MiB/s (463MB/s-463MB/s), io=1024MiB (1074MB), run=2317-2317msec

Disk stats (read/write):
  sda: ios=60697/0, merge=0/0, ticks=136153/0, in_queue=136796, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1810 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1920 MiB / 2016 MiB / 1920 MiB / 0 MiB / 4 MiB ( 1920 MiB / 2016 MiB / 1920 MiB / 0 MiB / 4 MiB )
2 min passed...1920 MiB / 2240 MiB / 1920 MiB / 32 MiB / 9 MiB ( 0 MiB / 224 MiB / 0 MiB / 32 MiB / 5 MiB )
3 min passed...1920 MiB / 2240 MiB / 1920 MiB / 32 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...1920 MiB / 2240 MiB / 1920 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...1920 MiB / 2240 MiB / 1920 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...1920 MiB / 2368 MiB / 1920 MiB / 32 MiB / 27 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...1920 MiB / 2592 MiB / 1920 MiB / 32 MiB / 31 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...1920 MiB / 2592 MiB / 1920 MiB / 32 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...1920 MiB / 2592 MiB / 1920 MiB / 32 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...1920 MiB / 2592 MiB / 1920 MiB / 64 MiB / 45 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
11 min passed...1920 MiB / 2848 MiB / 1920 MiB / 64 MiB / 49 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...1920 MiB / 2912 MiB / 1920 MiB / 64 MiB / 54 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...1920 MiB / 2912 MiB / 1920 MiB / 64 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...1920 MiB / 2912 MiB / 1920 MiB / 64 MiB / 63 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...1920 MiB / 3040 MiB / 1920 MiB / 64 MiB / 67 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...1920 MiB / 3232 MiB / 1920 MiB / 1088 MiB / 1096 MiB ( 0 MiB / 192 MiB / 0 MiB / 1024 MiB / 1029 MiB )
17 min passed...1920 MiB / 3232 MiB / 1920 MiB / 1120 MiB / 1100 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
18 min passed...1920 MiB / 3232 MiB / 1920 MiB / 1120 MiB / 1105 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
19 min passed...1920 MiB / 3232 MiB / 1920 MiB / 1120 MiB / 1109 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
20 min passed...1920 MiB / 3488 MiB / 1920 MiB / 1120 MiB / 1114 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
21 min passed...1920 MiB / 3584 MiB / 1920 MiB / 1120 MiB / 1118 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
22 min passed...1920 MiB / 3584 MiB / 1920 MiB / 1120 MiB / 1123 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
23 min passed...1920 MiB / 3584 MiB / 1920 MiB / 1120 MiB / 1127 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
24 min passed...1920 MiB / 3680 MiB / 1920 MiB / 1152 MiB / 1132 MiB ( 0 MiB / 96 MiB / 0 MiB / 32 MiB / 5 MiB )
25 min passed...1920 MiB / 3936 MiB / 1920 MiB / 1152 MiB / 1136 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
26 min passed...1920 MiB / 3936 MiB / 1920 MiB / 1152 MiB / 1141 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
27 min passed...1920 MiB / 3936 MiB / 1920 MiB / 1152 MiB / 1145 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
28 min passed...1920 MiB / 3936 MiB / 1920 MiB / 1152 MiB / 1150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
29 min passed...1920 MiB / 4160 MiB / 1920 MiB / 1152 MiB / 1154 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 4 MiB )
30 min passed...1920 MiB / 4256 MiB / 1920 MiB / 1152 MiB / 1159 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test 6: async 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=244MiB/s][w=61 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3076: Sun Aug 29 20:34:11 2021
  write: IOPS=70, BW=283MiB/s (297MB/s)(10.0GiB/36206msec); 0 zone resets
    slat (usec): min=71, max=830056, avg=730.65, stdev=19309.51
    clat (msec): min=68, max=3406, avg=901.99, stdev=431.89
     lat (msec): min=68, max=3406, avg=902.72, stdev=431.61
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[  121], 10.00th=[  230], 20.00th=[  405],
     | 30.00th=[  693], 40.00th=[  894], 50.00th=[ 1020], 60.00th=[ 1099],
     | 70.00th=[ 1234], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1418],
     | 99.00th=[ 1485], 99.50th=[ 1770], 99.90th=[ 2702], 99.95th=[ 2869],
     | 99.99th=[ 3406]
   bw (  KiB/s): min= 8192, max=811008, per=100.00%, avg=346651.07, stdev=249786.61, samples=59
   iops        : min=    2, max=  198, avg=84.59, stdev=60.96, samples=59
  lat (msec)   : 100=3.98%, 250=6.95%, 500=12.30%, 750=9.45%, 1000=14.61%
  cpu          : usr=8.54%, sys=1.39%, ctx=2027, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=283MiB/s (297MB/s), 283MiB/s-283MiB/s (297MB/s-297MB/s), io=10.0GiB (10.7GB), run=36206-36206msec

Disk stats (read/write):
  sda: ios=0/10198, merge=0/103, ticks=0/6805505, in_queue=5474864, util=78.46%
0 min passed...Guest Writes/Reads (Changes): 10240 MiB / 0 MiB ( 10240 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=416MiB/s][r=104 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=3333: Sun Aug 29 20:49:31 2021
  read: IOPS=134, BW=540MiB/s (566MB/s)(10.0GiB/18977msec)
    slat (usec): min=79, max=1575, avg=206.11, stdev=93.84
    clat (msec): min=42, max=896, avg=473.47, stdev=94.52
     lat (msec): min=43, max=896, avg=473.67, stdev=94.48
    clat percentiles (msec):
     |  1.00th=[  188],  5.00th=[  330], 10.00th=[  368], 20.00th=[  414],
     | 30.00th=[  439], 40.00th=[  460], 50.00th=[  472], 60.00th=[  489],
     | 70.00th=[  506], 80.00th=[  531], 90.00th=[  592], 95.00th=[  625],
     | 99.00th=[  726], 99.50th=[  743], 99.90th=[  885], 99.95th=[  894],
     | 99.99th=[  894]
   bw (  KiB/s): min=344064, max=655360, per=99.41%, avg=549274.54, stdev=60786.92, samples=37
   iops        : min=   84, max=  160, avg=134.08, stdev=14.83, samples=37
  lat (msec)   : 50=0.16%, 100=0.35%, 250=1.17%, 500=65.16%, 750=32.77%
  lat (msec)   : 1000=0.39%
  cpu          : usr=0.17%, sys=2.88%, ctx=1836, majf=0, minf=649
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=540MiB/s (566MB/s), 540MiB/s-540MiB/s (566MB/s-566MB/s), io=10.0GiB (10.7GB), run=18977-18977msec

Disk stats (read/write):
  sda: ios=10187/0, merge=0/0, ticks=3934815/0, in_queue=3874120, util=97.65%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10241 MiB / 10240 MiB ( 0 MiB / 0 MiB )
Script ran 1856 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:/dev/vgluks# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...10240 MiB / 11104 MiB / 10245 MiB / 0 MiB / 4 MiB ( 10240 MiB / 11104 MiB / 10245 MiB / 0 MiB / 4 MiB )
2 min passed...10240 MiB / 11104 MiB / 10245 MiB / 0 MiB / 9 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
3 min passed...10240 MiB / 11104 MiB / 10245 MiB / 0 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...10240 MiB / 11104 MiB / 10245 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...10240 MiB / 11360 MiB / 10245 MiB / 0 MiB / 22 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...10240 MiB / 11456 MiB / 10245 MiB / 0 MiB / 27 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...10240 MiB / 11456 MiB / 10245 MiB / 0 MiB / 31 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...10240 MiB / 11456 MiB / 10245 MiB / 32 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
9 min passed...10240 MiB / 11552 MiB / 10245 MiB / 32 MiB / 40 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...10240 MiB / 11776 MiB / 10245 MiB / 32 MiB / 45 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...10240 MiB / 11776 MiB / 10245 MiB / 32 MiB / 49 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...10240 MiB / 11776 MiB / 10245 MiB / 32 MiB / 54 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...10240 MiB / 11776 MiB / 10245 MiB / 32 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...10240 MiB / 12000 MiB / 10245 MiB / 32 MiB / 63 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...10240 MiB / 12128 MiB / 10245 MiB / 64 MiB / 67 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 4 MiB )
16 min passed...10240 MiB / 12128 MiB / 10245 MiB / 10304 MiB / 10314 MiB ( 0 MiB / 0 MiB / 0 MiB / 10240 MiB / 10247 MiB )
17 min passed...10240 MiB / 12128 MiB / 10245 MiB / 10304 MiB / 10319 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
18 min passed...10240 MiB / 12224 MiB / 10245 MiB / 10304 MiB / 10323 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
19 min passed...10240 MiB / 12416 MiB / 10245 MiB / 10304 MiB / 10328 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...10240 MiB / 12448 MiB / 10245 MiB / 10304 MiB / 10332 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...10240 MiB / 12448 MiB / 10245 MiB / 10336 MiB / 10337 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
22 min passed...10240 MiB / 12448 MiB / 10245 MiB / 10336 MiB / 10341 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...10240 MiB / 12672 MiB / 10245 MiB / 10336 MiB / 10346 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
24 min passed...10240 MiB / 12800 MiB / 10245 MiB / 10336 MiB / 10350 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...10240 MiB / 12800 MiB / 10245 MiB / 10336 MiB / 10355 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...10240 MiB / 12800 MiB / 10245 MiB / 10336 MiB / 10359 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
27 min passed...10240 MiB / 12896 MiB / 10245 MiB / 10336 MiB / 10364 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...10240 MiB / 13120 MiB / 10245 MiB / 10368 MiB / 10368 MiB ( 0 MiB / 224 MiB / 0 MiB / 32 MiB / 4 MiB )
29 min passed...10240 MiB / 13120 MiB / 10245 MiB / 10368 MiB / 10373 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
30 min passed...10240 MiB / 13120 MiB / 10245 MiB / 10368 MiB / 10377 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][75.0%][w=277MiB/s][w=17.7k IOPS][eta 00m:02s]
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=3908: Mon Aug 30 05:06:08 2021
  write: IOPS=9677, BW=151MiB/s (159MB/s)(1024MiB/6772msec); 0 zone resets
    slat (usec): min=7, max=4936, avg=27.61, stdev=26.95
    clat (usec): min=590, max=47853, avg=6572.87, stdev=6515.86
     lat (usec): min=613, max=47875, avg=6601.97, stdev=6521.85
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1860], 10.00th=[ 2212], 20.00th=[ 2737],
     | 30.00th=[ 3195], 40.00th=[ 3654], 50.00th=[ 4228], 60.00th=[ 4883],
     | 70.00th=[ 5932], 80.00th=[ 8356], 90.00th=[13960], 95.00th=[23725],
     | 99.00th=[32113], 99.50th=[34866], 99.90th=[40109], 99.95th=[43779],
     | 99.99th=[47449]
   bw (  KiB/s): min=36352, max=306112, per=96.29%, avg=149092.62, stdev=103360.12, samples=13
   iops        : min= 2272, max=19132, avg=9318.23, stdev=6459.69, samples=13
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=6.72%, 4=39.70%, 10=37.79%, 20=9.02%, 50=6.75%
  cpu          : usr=9.07%, sys=23.50%, ctx=63417, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=1024MiB (1074MB), run=6772-6772msec

Disk stats (read/write):
  sda: ios=0/63257, merge=0/172, ticks=0/404796, in_queue=402540, util=99.54%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1025 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 0 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=4174: Mon Aug 30 05:21:11 2021
  read: IOPS=28.9k, BW=451MiB/s (473MB/s)(1024MiB/2269msec)
    slat (usec): min=4, max=127, avg= 7.24, stdev= 3.21
    clat (usec): min=538, max=26678, avg=2203.11, stdev=842.67
     lat (usec): min=545, max=26693, avg=2211.19, stdev=842.67
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1205], 10.00th=[ 1352], 20.00th=[ 1565],
     | 30.00th=[ 1745], 40.00th=[ 1926], 50.00th=[ 2114], 60.00th=[ 2278],
     | 70.00th=[ 2474], 80.00th=[ 2704], 90.00th=[ 3032], 95.00th=[ 3359],
     | 99.00th=[ 5342], 99.50th=[ 6390], 99.90th=[ 8717], 99.95th=[10028],
     | 99.99th=[11863]
   bw (  KiB/s): min=458464, max=464512, per=99.85%, avg=461432.00, stdev=2481.10, samples=4
   iops        : min=28654, max=29032, avg=28839.50, stdev=155.07, samples=4
  lat (usec)   : 750=0.04%, 1000=1.00%
  lat (msec)   : 2=42.99%, 4=53.46%, 10=2.45%, 20=0.05%, 50=0.01%
  cpu          : usr=15.61%, sys=30.60%, ctx=41367, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=451MiB/s (473MB/s), 451MiB/s-451MiB/s (473MB/s-473MB/s), io=1024MiB (1074MB), run=2269-2269msec

Disk stats (read/write):
  sda: ios=61950/0, merge=0/0, ticks=136327/0, in_queue=136852, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1812 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2016 MiB / 2304 MiB / 2020 MiB / 0 MiB / 4 MiB ( 2016 MiB / 2304 MiB / 2020 MiB / 0 MiB / 4 MiB )
2 min passed...2016 MiB / 2464 MiB / 2020 MiB / 0 MiB / 9 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
3 min passed...2016 MiB / 2464 MiB / 2020 MiB / 0 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...2016 MiB / 2464 MiB / 2020 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...2016 MiB / 2560 MiB / 2020 MiB / 0 MiB / 22 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...2016 MiB / 2816 MiB / 2020 MiB / 32 MiB / 27 MiB ( 0 MiB / 256 MiB / 0 MiB / 32 MiB / 5 MiB )
7 min passed...2016 MiB / 2816 MiB / 2020 MiB / 32 MiB / 31 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...2016 MiB / 2816 MiB / 2020 MiB / 32 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...2016 MiB / 2816 MiB / 2020 MiB / 32 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...2016 MiB / 3040 MiB / 2020 MiB / 32 MiB / 45 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...2016 MiB / 3136 MiB / 2020 MiB / 32 MiB / 49 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...2016 MiB / 3136 MiB / 2020 MiB / 32 MiB / 54 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...2016 MiB / 3136 MiB / 2020 MiB / 64 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
14 min passed...2016 MiB / 3232 MiB / 2020 MiB / 64 MiB / 63 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...2016 MiB / 3360 MiB / 2020 MiB / 64 MiB / 67 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...2016 MiB / 3360 MiB / 2020 MiB / 1088 MiB / 1096 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1029 MiB )
17 min passed...2016 MiB / 3360 MiB / 2020 MiB / 1088 MiB / 1100 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
18 min passed...2016 MiB / 3360 MiB / 2020 MiB / 1088 MiB / 1105 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
19 min passed...2016 MiB / 3584 MiB / 2020 MiB / 1088 MiB / 1109 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 4 MiB )
20 min passed...2016 MiB / 3680 MiB / 2020 MiB / 1120 MiB / 1114 MiB ( 0 MiB / 96 MiB / 0 MiB / 32 MiB / 5 MiB )
21 min passed...2016 MiB / 3680 MiB / 2020 MiB / 1120 MiB / 1118 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
22 min passed...2016 MiB / 3680 MiB / 2020 MiB / 1120 MiB / 1123 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
23 min passed...2016 MiB / 3776 MiB / 2020 MiB / 1120 MiB / 1127 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
24 min passed...2016 MiB / 4032 MiB / 2020 MiB / 1120 MiB / 1132 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
25 min passed...2016 MiB / 4032 MiB / 2020 MiB / 1120 MiB / 1136 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
26 min passed...2016 MiB / 4032 MiB / 2020 MiB / 1120 MiB / 1141 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
27 min passed...2016 MiB / 4032 MiB / 2020 MiB / 1152 MiB / 1145 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
28 min passed...2016 MiB / 4288 MiB / 2020 MiB / 1152 MiB / 1150 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
29 min passed...2016 MiB / 4352 MiB / 2020 MiB / 1152 MiB / 1154 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 4 MiB )
30 min passed...2016 MiB / 4352 MiB / 2020 MiB / 1152 MiB / 1159 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=16.1MiB/s][w=516 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4405: Mon Aug 30 05:38:52 2021
  write: IOPS=480, BW=15.0MiB/s (15.7MB/s)(1024MiB/68251msec); 0 zone resets
    clat (usec): min=895, max=8331, avg=2056.44, stdev=738.42
     lat (usec): min=897, max=8334, avg=2058.70, stdev=738.46
    clat percentiles (usec):
     |  1.00th=[ 1139],  5.00th=[ 1254], 10.00th=[ 1336], 20.00th=[ 1467],
     | 30.00th=[ 1565], 40.00th=[ 1680], 50.00th=[ 1795], 60.00th=[ 1942],
     | 70.00th=[ 2212], 80.00th=[ 2835], 90.00th=[ 3326], 95.00th=[ 3556],
     | 99.00th=[ 3884], 99.50th=[ 3982], 99.90th=[ 4146], 99.95th=[ 4293],
     | 99.99th=[ 7111]
   bw (  KiB/s): min= 9344, max=22528, per=99.97%, avg=15357.74, stdev=2866.29, samples=136
   iops        : min=  292, max=  704, avg=479.90, stdev=89.56, samples=136
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=62.62%, 4=36.96%, 10=0.38%
  cpu          : usr=1.98%, sys=7.08%, ctx=71577, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=15.0MiB/s (15.7MB/s), 15.0MiB/s-15.0MiB/s (15.7MB/s-15.7MB/s), io=1024MiB (1074MB), run=68251-68251msec

Disk stats (read/write):
  sda: ios=0/98227, merge=0/66496, ticks=0/59342, in_queue=68204, util=99.96%
0 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 1540 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1540 MiB / 0 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=54.6MiB/s][r=1748 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=4605: Mon Aug 30 05:54:11 2021
  read: IOPS=1762, BW=55.1MiB/s (57.8MB/s)(1024MiB/18589msec)
    clat (usec): min=350, max=2534, avg=556.60, stdev=73.73
     lat (usec): min=351, max=2536, avg=558.49, stdev=73.84
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  429], 10.00th=[  457], 20.00th=[  494],
     | 30.00th=[  519], 40.00th=[  537], 50.00th=[  562], 60.00th=[  578],
     | 70.00th=[  603], 80.00th=[  619], 90.00th=[  644], 95.00th=[  668],
     | 99.00th=[  709], 99.50th=[  717], 99.90th=[  742], 99.95th=[  832],
     | 99.99th=[ 1647]
   bw (  KiB/s): min=50240, max=67712, per=100.00%, avg=56436.54, stdev=4447.42, samples=37
   iops        : min= 1570, max= 2116, avg=1763.62, stdev=139.00, samples=37
  lat (usec)   : 500=22.28%, 750=77.63%, 1000=0.06%
  lat (msec)   : 2=0.02%, 4=0.01%
  cpu          : usr=3.35%, sys=9.50%, ctx=32771, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=55.1MiB/s (57.8MB/s), 55.1MiB/s-55.1MiB/s (57.8MB/s-57.8MB/s), io=1024MiB (1074MB), run=18589-18589msec

Disk stats (read/write):
  sda: ios=32459/0, merge=0/0, ticks=16761/0, in_queue=18496, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1540 MiB / 1024 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1541 MiB / 1024 MiB ( 0 MiB / 0 MiB )
Script ran 1889 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2400 MiB / 3008 MiB / 2381 MiB / 0 MiB / 5 MiB ( 2400 MiB / 3008 MiB / 2381 MiB / 0 MiB / 5 MiB )
2 min passed...2688 MiB / 3328 MiB / 2689 MiB / 32 MiB / 10 MiB ( 288 MiB / 320 MiB / 308 MiB / 32 MiB / 5 MiB )
3 min passed...2688 MiB / 3328 MiB / 2689 MiB / 32 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...2688 MiB / 3328 MiB / 2689 MiB / 32 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...2688 MiB / 3488 MiB / 2689 MiB / 32 MiB / 23 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...2688 MiB / 3680 MiB / 2689 MiB / 32 MiB / 28 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...2688 MiB / 3680 MiB / 2689 MiB / 32 MiB / 32 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...2688 MiB / 3680 MiB / 2689 MiB / 32 MiB / 37 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...2688 MiB / 3680 MiB / 2689 MiB / 64 MiB / 41 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
10 min passed...2688 MiB / 3936 MiB / 2689 MiB / 64 MiB / 46 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...2688 MiB / 4032 MiB / 2689 MiB / 64 MiB / 50 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...2688 MiB / 4032 MiB / 2689 MiB / 64 MiB / 55 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...2688 MiB / 4032 MiB / 2689 MiB / 64 MiB / 59 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...2688 MiB / 4160 MiB / 2689 MiB / 64 MiB / 64 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...2688 MiB / 4352 MiB / 2689 MiB / 64 MiB / 68 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...2688 MiB / 4352 MiB / 2689 MiB / 96 MiB / 73 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
17 min passed...2688 MiB / 4352 MiB / 2689 MiB / 1120 MiB / 1101 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1028 MiB )
18 min passed...2688 MiB / 4352 MiB / 2690 MiB / 1120 MiB / 1106 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 5 MiB )
19 min passed...2688 MiB / 4608 MiB / 2690 MiB / 1120 MiB / 1110 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 4 MiB )
20 min passed...2688 MiB / 4704 MiB / 2690 MiB / 1120 MiB / 1115 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 5 MiB )
21 min passed...2688 MiB / 4704 MiB / 2690 MiB / 1120 MiB / 1119 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
22 min passed...2688 MiB / 4704 MiB / 2690 MiB / 1120 MiB / 1124 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
23 min passed...2688 MiB / 4832 MiB / 2690 MiB / 1152 MiB / 1128 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 4 MiB )
24 min passed...2688 MiB / 5024 MiB / 2690 MiB / 1152 MiB / 1133 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
25 min passed...2688 MiB / 5024 MiB / 2690 MiB / 1152 MiB / 1137 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
26 min passed...2688 MiB / 5024 MiB / 2690 MiB / 1152 MiB / 1142 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
27 min passed...2688 MiB / 5056 MiB / 2690 MiB / 1152 MiB / 1146 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
28 min passed...2688 MiB / 5344 MiB / 2690 MiB / 1152 MiB / 1151 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 5 MiB )
29 min passed...2688 MiB / 5376 MiB / 2690 MiB / 1152 MiB / 1155 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
30 min passed...2688 MiB / 5376 MiB / 2690 MiB / 1152 MiB / 1160 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
31 min passed...2688 MiB / 5376 MiB / 2690 MiB / 1184 MiB / 1164 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][85.7%][w=250MiB/s][w=8002 IOPS][eta 00m:01s]
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=4837: Mon Aug 30 06:12:54 2021
  write: IOPS=5070, BW=158MiB/s (166MB/s)(1024MiB/6462msec); 0 zone resets
    slat (usec): min=10, max=9996, avg=42.13, stdev=58.55
    clat (usec): min=528, max=45157, avg=12550.67, stdev=7955.03
     lat (usec): min=556, max=45185, avg=12594.90, stdev=7958.60
    clat percentiles (usec):
     |  1.00th=[ 3195],  5.00th=[ 4621], 10.00th=[ 5473], 20.00th=[ 6718],
     | 30.00th=[ 7701], 40.00th=[ 8586], 50.00th=[ 9765], 60.00th=[11207],
     | 70.00th=[13435], 80.00th=[17695], 90.00th=[25822], 95.00th=[30016],
     | 99.00th=[36963], 99.50th=[40109], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[44827]
   bw (  KiB/s): min=74112, max=279168, per=94.90%, avg=154000.00, stdev=69220.61, samples=12
   iops        : min= 2316, max= 8724, avg=4812.50, stdev=2163.14, samples=12
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.06%, 4=2.30%, 10=49.75%, 20=31.43%, 50=16.46%
  cpu          : usr=9.30%, sys=20.01%, ctx=31985, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=1024MiB (1074MB), run=6462-6462msec

Disk stats (read/write):
  sda: ios=0/32181, merge=0/113, ticks=0/399400, in_queue=400264, util=99.68%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 0 MiB ( 1024 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1 MiB ( 1 MiB / 1 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=5221: Mon Aug 30 06:27:57 2021
  read: IOPS=15.2k, BW=474MiB/s (497MB/s)(1024MiB/2160msec)
    slat (usec): min=4, max=129, avg=10.40, stdev= 4.28
    clat (usec): min=1091, max=37505, avg=4200.30, stdev=1859.47
     lat (usec): min=1122, max=37518, avg=4211.66, stdev=1859.42
    clat percentiles (usec):
     |  1.00th=[ 1958],  5.00th=[ 2278], 10.00th=[ 2507], 20.00th=[ 2868],
     | 30.00th=[ 3195], 40.00th=[ 3523], 50.00th=[ 3884], 60.00th=[ 4228],
     | 70.00th=[ 4621], 80.00th=[ 5145], 90.00th=[ 5932], 95.00th=[ 7308],
     | 99.00th=[11207], 99.50th=[13435], 99.90th=[19006], 99.95th=[22152],
     | 99.99th=[33817]
   bw (  KiB/s): min=479104, max=488768, per=99.78%, avg=484366.00, stdev=4053.58, samples=4
   iops        : min=14972, max=15274, avg=15136.25, stdev=126.71, samples=4
  lat (msec)   : 2=1.20%, 4=52.50%, 10=44.72%, 20=1.51%, 50=0.07%
  cpu          : usr=9.72%, sys=24.07%, ctx=29543, majf=0, minf=522
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=1024MiB (1074MB), run=2160-2160msec

Disk stats (read/write):
  sda: ios=32558/0, merge=0/0, ticks=136593/0, in_queue=137200, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1025 MiB ( 0 MiB / 0 MiB )
Script ran 1810 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1920 MiB / 2048 MiB / 1922 MiB / 0 MiB / 5 MiB ( 1920 MiB / 2048 MiB / 1922 MiB / 0 MiB / 5 MiB )
2 min passed...1920 MiB / 2336 MiB / 1922 MiB / 0 MiB / 9 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
3 min passed...1920 MiB / 2400 MiB / 1922 MiB / 32 MiB / 14 MiB ( 0 MiB / 64 MiB / 0 MiB / 32 MiB / 5 MiB )
4 min passed...1920 MiB / 2400 MiB / 1922 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
5 min passed...1920 MiB / 2400 MiB / 1922 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
6 min passed...1920 MiB / 2560 MiB / 1922 MiB / 32 MiB / 27 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
7 min passed...1920 MiB / 2720 MiB / 1922 MiB / 32 MiB / 32 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
8 min passed...1920 MiB / 2720 MiB / 1922 MiB / 32 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
9 min passed...1920 MiB / 2720 MiB / 1922 MiB / 32 MiB / 41 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
10 min passed...1920 MiB / 2752 MiB / 1922 MiB / 64 MiB / 45 MiB ( 0 MiB / 32 MiB / 0 MiB / 32 MiB / 4 MiB )
11 min passed...1920 MiB / 3008 MiB / 1922 MiB / 64 MiB / 50 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 5 MiB )
12 min passed...1920 MiB / 3072 MiB / 1922 MiB / 64 MiB / 54 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 4 MiB )
13 min passed...1920 MiB / 3072 MiB / 1923 MiB / 64 MiB / 61 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 7 MiB )
14 min passed...1920 MiB / 3072 MiB / 1923 MiB / 64 MiB / 65 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
15 min passed...1920 MiB / 3232 MiB / 1923 MiB / 64 MiB / 70 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
16 min passed...1920 MiB / 3392 MiB / 1923 MiB / 1120 MiB / 1098 MiB ( 0 MiB / 160 MiB / 0 MiB / 1056 MiB / 1028 MiB )
17 min passed...1920 MiB / 3392 MiB / 1923 MiB / 1120 MiB / 1103 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
18 min passed...1920 MiB / 3392 MiB / 1923 MiB / 1120 MiB / 1107 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
19 min passed...1920 MiB / 3424 MiB / 1923 MiB / 1120 MiB / 1112 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...1920 MiB / 3616 MiB / 1923 MiB / 1120 MiB / 1116 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...1920 MiB / 3744 MiB / 1923 MiB / 1120 MiB / 1121 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 5 MiB )
22 min passed...1920 MiB / 3744 MiB / 1923 MiB / 1120 MiB / 1125 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...1920 MiB / 3744 MiB / 1923 MiB / 1152 MiB / 1130 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
24 min passed...1920 MiB / 3904 MiB / 1923 MiB / 1152 MiB / 1134 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...1920 MiB / 4064 MiB / 1923 MiB / 1152 MiB / 1139 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...1920 MiB / 4096 MiB / 1923 MiB / 1152 MiB / 1144 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 5 MiB )
27 min passed...1920 MiB / 4096 MiB / 1923 MiB / 1152 MiB / 1148 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
28 min passed...1920 MiB / 4096 MiB / 1923 MiB / 1152 MiB / 1153 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
29 min passed...1920 MiB / 4384 MiB / 1923 MiB / 1152 MiB / 1157 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
30 min passed...1920 MiB / 4416 MiB / 1923 MiB / 1184 MiB / 1162 MiB ( 0 MiB / 32 MiB / 0 MiB / 32 MiB / 5 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test X6: 4x async sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_6.sh
async_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
async_seq_32K: Laying out IO file (1 file / 3072MiB)
Jobs: 4 (f=4): [W(4)][98.9%][w=135MiB/s][w=4329 IOPS][eta 00m:01s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5453: Mon Aug 30 06:50:49 2021
  write: IOPS=1098, BW=34.3MiB/s (36.0MB/s)(3072MiB/89451msec); 0 zone resets
    slat (usec): min=14, max=1952, avg=68.42, stdev=65.17
    clat (usec): min=3, max=12925, avg=813.00, stdev=786.61
     lat (usec): min=232, max=12959, avg=885.01, stdev=780.53
    clat percentiles (usec):
     |  1.00th=[   69],  5.00th=[  204], 10.00th=[  273], 20.00th=[  302],
     | 30.00th=[  326], 40.00th=[  355], 50.00th=[  396], 60.00th=[  478],
     | 70.00th=[  930], 80.00th=[ 1582], 90.00th=[ 1909], 95.00th=[ 2343],
     | 99.00th=[ 3195], 99.50th=[ 3752], 99.90th=[ 5997], 99.95th=[ 7111],
     | 99.99th=[ 8356]
   bw (  KiB/s): min=29440, max=48576, per=25.10%, avg=35159.69, stdev=1796.48, samples=178
   iops        : min=  920, max= 1518, avg=1098.67, stdev=56.15, samples=178
  lat (usec)   : 4=0.08%, 10=0.47%, 20=0.01%, 50=0.17%, 100=0.90%
  lat (usec)   : 250=5.87%, 500=54.02%, 750=6.98%, 1000=1.96%
  lat (msec)   : 2=21.12%, 4=8.04%, 10=0.39%, 20=0.01%
  cpu          : usr=2.65%, sys=5.19%, ctx=139367, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5454: Mon Aug 30 06:50:49 2021
  write: IOPS=1096, BW=34.3MiB/s (35.9MB/s)(3072MiB/89680msec); 0 zone resets
    slat (usec): min=14, max=773, avg=68.53, stdev=64.33
    clat (usec): min=2, max=13648, avg=815.35, stdev=790.65
     lat (usec): min=246, max=13682, avg=887.32, stdev=784.59
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[  202], 10.00th=[  269], 20.00th=[  302],
     | 30.00th=[  326], 40.00th=[  355], 50.00th=[  396], 60.00th=[  478],
     | 70.00th=[  963], 80.00th=[ 1582], 90.00th=[ 1909], 95.00th=[ 2343],
     | 99.00th=[ 3228], 99.50th=[ 3785], 99.90th=[ 6128], 99.95th=[ 7177],
     | 99.99th=[ 8455]
   bw (  KiB/s): min=29440, max=38336, per=25.03%, avg=35055.43, stdev=1334.49, samples=179
   iops        : min=  920, max= 1198, avg=1095.44, stdev=41.68, samples=179
  lat (usec)   : 4=0.07%, 10=0.49%, 20=0.01%, 50=0.15%, 100=0.87%
  lat (usec)   : 250=6.04%, 500=53.78%, 750=6.99%, 1000=1.83%
  lat (msec)   : 2=21.21%, 4=8.16%, 10=0.39%, 20=0.01%
  cpu          : usr=2.61%, sys=5.22%, ctx=139401, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5455: Mon Aug 30 06:50:49 2021
  write: IOPS=1094, BW=34.2MiB/s (35.9MB/s)(3072MiB/89844msec); 0 zone resets
    slat (usec): min=16, max=1952, avg=68.78, stdev=65.37
    clat (usec): min=2, max=13116, avg=816.89, stdev=795.07
     lat (usec): min=216, max=13223, avg=889.04, stdev=788.44
    clat percentiles (usec):
     |  1.00th=[   69],  5.00th=[  204], 10.00th=[  273], 20.00th=[  302],
     | 30.00th=[  326], 40.00th=[  355], 50.00th=[  396], 60.00th=[  478],
     | 70.00th=[  922], 80.00th=[ 1582], 90.00th=[ 1926], 95.00th=[ 2343],
     | 99.00th=[ 3261], 99.50th=[ 3818], 99.90th=[ 5997], 99.95th=[ 7242],
     | 99.99th=[ 8586]
   bw (  KiB/s): min=29184, max=38272, per=24.98%, avg=34980.17, stdev=1317.51, samples=179
   iops        : min=  912, max= 1196, avg=1093.04, stdev=41.20, samples=179
  lat (usec)   : 4=0.06%, 10=0.49%, 20=0.01%, 50=0.17%, 100=0.94%
  lat (usec)   : 250=5.80%, 500=54.18%, 750=7.05%, 1000=1.77%
  lat (msec)   : 2=20.84%, 4=8.30%, 10=0.38%, 20=0.01%
  cpu          : usr=2.79%, sys=5.05%, ctx=138377, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5456: Mon Aug 30 06:50:49 2021
  write: IOPS=1096, BW=34.3MiB/s (35.9MB/s)(3072MiB/89648msec); 0 zone resets
    slat (usec): min=15, max=1890, avg=68.64, stdev=64.80
    clat (usec): min=2, max=12859, avg=815.02, stdev=786.20
     lat (usec): min=224, max=12893, avg=887.00, stdev=780.48
    clat percentiles (usec):
     |  1.00th=[   71],  5.00th=[  200], 10.00th=[  269], 20.00th=[  302],
     | 30.00th=[  326], 40.00th=[  355], 50.00th=[  396], 60.00th=[  482],
     | 70.00th=[  979], 80.00th=[ 1582], 90.00th=[ 1909], 95.00th=[ 2343],
     | 99.00th=[ 3195], 99.50th=[ 3785], 99.90th=[ 5866], 99.95th=[ 7111],
     | 99.99th=[ 8455]
   bw (  KiB/s): min=29504, max=38400, per=25.04%, avg=35065.52, stdev=1328.30, samples=179
   iops        : min=  922, max= 1200, avg=1095.69, stdev=41.55, samples=179
  lat (usec)   : 4=0.08%, 10=0.49%, 20=0.01%, 50=0.16%, 100=0.83%
  lat (usec)   : 250=6.07%, 500=53.64%, 750=6.89%, 1000=1.97%
  lat (msec)   : 2=21.44%, 4=8.03%, 10=0.38%, 20=0.01%
  cpu          : usr=2.62%, sys=5.20%, ctx=139890, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,98304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=137MiB/s (143MB/s), 34.2MiB/s-34.3MiB/s (35.9MB/s-36.0MB/s), io=12.0GiB (12.9GB), run=89451-89844msec

Disk stats (read/write):
  sda: ios=0/392948, merge=0/339, ticks=0/313231, in_queue=328764, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 12290 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12290 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test X6: 4x async sequential read/write 32K

Guest (part 2):

Code:

async_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 1 (f=1): [_(2),R(1),_(1)][100.0%][r=199MiB/s][r=6365 IOPS][eta 00m:00s]
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5664: Mon Aug 30 07:06:36 2021
  read: IOPS=2165, BW=67.7MiB/s (70.9MB/s)(3072MiB/45401msec)
    slat (usec): min=8, max=177, avg=28.27, stdev=10.66
    clat (usec): min=209, max=3646, avg=424.90, stdev=74.41
     lat (usec): min=273, max=3684, avg=454.92, stdev=76.24
    clat percentiles (usec):
     |  1.00th=[  310],  5.00th=[  334], 10.00th=[  347], 20.00th=[  371],
     | 30.00th=[  388], 40.00th=[  404], 50.00th=[  420], 60.00th=[  437],
     | 70.00th=[  453], 80.00th=[  474], 90.00th=[  502], 95.00th=[  529],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  725], 99.95th=[  971],
     | 99.99th=[ 2868]
   bw (  KiB/s): min=61376, max=76032, per=25.53%, avg=69272.11, stdev=2972.36, samples=90
   iops        : min= 1918, max= 2376, avg=2164.70, stdev=92.91, samples=90
  lat (usec)   : 250=0.01%, 500=89.38%, 750=10.53%, 1000=0.04%
  lat (msec)   : 2=0.01%, 4=0.03%
  cpu          : usr=3.50%, sys=9.20%, ctx=98364, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5665: Mon Aug 30 07:06:36 2021
  read: IOPS=2155, BW=67.4MiB/s (70.6MB/s)(3072MiB/45602msec)
    slat (usec): min=8, max=388, avg=28.74, stdev=10.67
    clat (usec): min=179, max=3350, avg=426.26, stdev=73.77
     lat (usec): min=279, max=3389, avg=456.82, stdev=75.87
    clat percentiles (usec):
     |  1.00th=[  314],  5.00th=[  334], 10.00th=[  351], 20.00th=[  371],
     | 30.00th=[  392], 40.00th=[  408], 50.00th=[  424], 60.00th=[  437],
     | 70.00th=[  457], 80.00th=[  474], 90.00th=[  502], 95.00th=[  529],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  734], 99.95th=[  906],
     | 99.99th=[ 2835]
   bw (  KiB/s): min=63936, max=75008, per=25.42%, avg=68983.79, stdev=2837.84, samples=91
   iops        : min= 1998, max= 2344, avg=2155.70, stdev=88.70, samples=91
  lat (usec)   : 250=0.01%, 500=89.28%, 750=10.63%, 1000=0.04%
  lat (msec)   : 2=0.01%, 4=0.03%
  cpu          : usr=3.99%, sys=8.90%, ctx=98362, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5666: Mon Aug 30 07:06:36 2021
  read: IOPS=2119, BW=66.2MiB/s (69.5MB/s)(3072MiB/46372msec)
    slat (usec): min=7, max=340, avg=30.78, stdev=11.06
    clat (usec): min=49, max=3262, avg=431.80, stdev=77.06
     lat (usec): min=289, max=3292, avg=464.40, stdev=79.16
    clat percentiles (usec):
     |  1.00th=[  314],  5.00th=[  338], 10.00th=[  355], 20.00th=[  375],
     | 30.00th=[  396], 40.00th=[  412], 50.00th=[  429], 60.00th=[  445],
     | 70.00th=[  461], 80.00th=[  482], 90.00th=[  510], 95.00th=[  537],
     | 99.00th=[  603], 99.50th=[  627], 99.90th=[  742], 99.95th=[ 1012],
     | 99.99th=[ 2900]
   bw (  KiB/s): min=55268, max=75264, per=25.02%, avg=67901.41, stdev=3074.78, samples=92
   iops        : min= 1727, max= 2352, avg=2121.87, stdev=96.08, samples=92
  lat (usec)   : 50=0.01%, 500=87.14%, 750=12.76%, 1000=0.05%
  lat (msec)   : 2=0.02%, 4=0.04%
  cpu          : usr=3.79%, sys=9.68%, ctx=98353, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
async_seq_32K: (groupid=0, jobs=1): err= 0: pid=5667: Mon Aug 30 07:06:36 2021
  read: IOPS=2148, BW=67.1MiB/s (70.4MB/s)(3072MiB/45755msec)
    slat (usec): min=7, max=242, avg=29.37, stdev=10.81
    clat (usec): min=248, max=3724, avg=427.10, stdev=75.61
     lat (usec): min=286, max=3752, avg=458.28, stdev=77.68
    clat percentiles (usec):
     |  1.00th=[  310],  5.00th=[  334], 10.00th=[  351], 20.00th=[  371],
     | 30.00th=[  392], 40.00th=[  408], 50.00th=[  424], 60.00th=[  441],
     | 70.00th=[  457], 80.00th=[  474], 90.00th=[  506], 95.00th=[  529],
     | 99.00th=[  586], 99.50th=[  619], 99.90th=[  717], 99.95th=[  889],
     | 99.99th=[ 2835]
   bw (  KiB/s): min=61056, max=75456, per=25.35%, avg=68785.32, stdev=3134.82, samples=91
   iops        : min= 1908, max= 2358, avg=2149.51, stdev=97.96, samples=91
  lat (usec)   : 250=0.01%, 500=88.83%, 750=11.08%, 1000=0.04%
  lat (msec)   : 2=0.01%, 4=0.04%
  cpu          : usr=3.80%, sys=9.40%, ctx=98354, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=265MiB/s (278MB/s), 66.2MiB/s-67.7MiB/s (69.5MB/s-70.9MB/s), io=12.0GiB (12.9GB), run=45401-46372msec

Disk stats (read/write):
  sda: ios=392786/0, merge=0/0, ticks=167350/0, in_queue=178492, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 12288 MiB )
1 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 12290 MiB / 12288 MiB ( 0 MiB / 0 MiB )
Script ran 1938 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...15072 MiB / 16000 MiB / 15089 MiB / 32 MiB / 4 MiB ( 15072 MiB / 16000 MiB / 15089 MiB / 32 MiB / 4 MiB )
2 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 9 MiB ( 9472 MiB / 10016 MiB / 9457 MiB / 0 MiB / 5 MiB )
3 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 13 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 22 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
6 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 27 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...24544 MiB / 26016 MiB / 24546 MiB / 32 MiB / 31 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...24544 MiB / 26016 MiB / 24546 MiB / 64 MiB / 36 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
9 min passed...24544 MiB / 26016 MiB / 24546 MiB / 64 MiB / 40 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...24544 MiB / 26176 MiB / 24546 MiB / 64 MiB / 45 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...24544 MiB / 26208 MiB / 24546 MiB / 64 MiB / 49 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...24544 MiB / 26208 MiB / 24546 MiB / 64 MiB / 54 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
13 min passed...24544 MiB / 26208 MiB / 24546 MiB / 64 MiB / 58 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...24544 MiB / 26208 MiB / 24546 MiB / 64 MiB / 63 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...24544 MiB / 26528 MiB / 24546 MiB / 96 MiB / 67 MiB ( 0 MiB / 320 MiB / 0 MiB / 32 MiB / 4 MiB )
16 min passed...24544 MiB / 26528 MiB / 24546 MiB / 96 MiB / 72 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
17 min passed...24544 MiB / 26528 MiB / 24546 MiB / 7072 MiB / 7050 MiB ( 0 MiB / 0 MiB / 0 MiB / 6976 MiB / 6978 MiB )
18 min passed...24544 MiB / 26528 MiB / 24546 MiB / 12384 MiB / 12372 MiB ( 0 MiB / 0 MiB / 0 MiB / 5312 MiB / 5322 MiB )
19 min passed...24544 MiB / 26720 MiB / 24546 MiB / 12384 MiB / 12377 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
20 min passed...24544 MiB / 26880 MiB / 24546 MiB / 12384 MiB / 12381 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...24544 MiB / 26880 MiB / 24546 MiB / 12384 MiB / 12386 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
22 min passed...24544 MiB / 26880 MiB / 24546 MiB / 12416 MiB / 12390 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
23 min passed...24544 MiB / 26880 MiB / 24546 MiB / 12416 MiB / 12395 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
24 min passed...24544 MiB / 27168 MiB / 24546 MiB / 12416 MiB / 12399 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...24544 MiB / 27232 MiB / 24546 MiB / 12416 MiB / 12404 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...24544 MiB / 27232 MiB / 24546 MiB / 12416 MiB / 12408 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
27 min passed...24544 MiB / 27232 MiB / 24546 MiB / 12416 MiB / 12413 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...24544 MiB / 27360 MiB / 24546 MiB / 12416 MiB / 12417 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
29 min passed...24544 MiB / 27552 MiB / 24546 MiB / 12448 MiB / 12422 MiB ( 0 MiB / 192 MiB / 0 MiB / 32 MiB / 5 MiB )
30 min passed...24544 MiB / 27552 MiB / 24546 MiB / 12448 MiB / 12426 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
31 min passed...24544 MiB / 27552 MiB / 24546 MiB / 12448 MiB / 12431 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
32 min passed...24544 MiB / 27584 MiB / 24546 MiB / 12448 MiB / 12435 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 4 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test X7: 4x sync sequential read/write 32K

Guest (part 1):

Code:

root@DebianTest2:~# bash /root/benchmark_x_7.sh
sync_seq_32K: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
sync_seq_32K: Laying out IO file (1 file / 512MiB)
Jobs: 2 (f=2): [W(2),_(2)][100.0%][w=27.9MiB/s][w=893 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5904: Mon Aug 30 07:34:57 2021
  write: IOPS=399, BW=12.5MiB/s (13.1MB/s)(512MiB/41012msec); 0 zone resets
    clat (usec): min=1059, max=10345, avg=2478.30, stdev=821.21
     lat (usec): min=1061, max=10347, avg=2480.53, stdev=821.22
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1598], 10.00th=[ 1680], 20.00th=[ 1811],
     | 30.00th=[ 1909], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2376],
     | 70.00th=[ 2835], 80.00th=[ 3228], 90.00th=[ 3687], 95.00th=[ 4047],
     | 99.00th=[ 4752], 99.50th=[ 5014], 99.90th=[ 5735], 99.95th=[ 6915],
     | 99.99th=[ 9634]
   bw (  KiB/s): min=10304, max=16832, per=24.74%, avg=12649.28, stdev=994.63, samples=81
   iops        : min=  322, max=  526, avg=395.25, stdev=31.11, samples=81
  lat (msec)   : 2=38.04%, 4=56.43%, 10=5.52%, 20=0.01%
  cpu          : usr=1.67%, sys=5.10%, ctx=50212, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5905: Mon Aug 30 07:34:57 2021
  write: IOPS=400, BW=12.5MiB/s (13.1MB/s)(512MiB/40955msec); 0 zone resets
    clat (usec): min=1116, max=10693, avg=2474.77, stdev=792.24
     lat (usec): min=1118, max=10696, avg=2476.99, stdev=792.24
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1614], 10.00th=[ 1696], 20.00th=[ 1811],
     | 30.00th=[ 1926], 40.00th=[ 2024], 50.00th=[ 2180], 60.00th=[ 2409],
     | 70.00th=[ 2835], 80.00th=[ 3228], 90.00th=[ 3654], 95.00th=[ 3949],
     | 99.00th=[ 4686], 99.50th=[ 4948], 99.90th=[ 5800], 99.95th=[ 6915],
     | 99.99th=[10421]
   bw (  KiB/s): min=10155, max=14912, per=24.78%, avg=12673.65, stdev=813.97, samples=81
   iops        : min=  317, max=  466, avg=396.01, stdev=25.48, samples=81
  lat (msec)   : 2=37.11%, 4=58.41%, 10=4.47%, 20=0.01%
  cpu          : usr=1.57%, sys=5.01%, ctx=50279, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5906: Mon Aug 30 07:34:57 2021
  write: IOPS=420, BW=13.1MiB/s (13.8MB/s)(512MiB/38935msec); 0 zone resets
    clat (usec): min=1293, max=8894, avg=2351.18, stdev=626.64
     lat (usec): min=1295, max=8896, avg=2353.43, stdev=626.63
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 1827],
     | 30.00th=[ 1942], 40.00th=[ 2040], 50.00th=[ 2180], 60.00th=[ 2343],
     | 70.00th=[ 2606], 80.00th=[ 2868], 90.00th=[ 3195], 95.00th=[ 3523],
     | 99.00th=[ 4178], 99.50th=[ 4555], 99.90th=[ 5538], 99.95th=[ 6259],
     | 99.99th=[ 8094]
   bw (  KiB/s): min=10411, max=15744, per=26.32%, avg=13460.71, stdev=1199.60, samples=77
   iops        : min=  325, max=  492, avg=420.64, stdev=37.50, samples=77
  lat (msec)   : 2=36.17%, 4=62.29%, 10=1.54%
  cpu          : usr=1.75%, sys=5.63%, ctx=49390, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=5907: Mon Aug 30 07:34:57 2021
  write: IOPS=424, BW=13.3MiB/s (13.9MB/s)(512MiB/38553msec); 0 zone resets
    clat (usec): min=1235, max=7561, avg=2327.71, stdev=581.06
     lat (usec): min=1237, max=7563, avg=2330.00, stdev=581.06
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 1827],
     | 30.00th=[ 1942], 40.00th=[ 2040], 50.00th=[ 2180], 60.00th=[ 2343],
     | 70.00th=[ 2573], 80.00th=[ 2835], 90.00th=[ 3130], 95.00th=[ 3392],
     | 99.00th=[ 3982], 99.50th=[ 4228], 99.90th=[ 4883], 99.95th=[ 5407],
     | 99.99th=[ 7242]
   bw (  KiB/s): min=10475, max=16768, per=26.59%, avg=13596.39, stdev=1186.54, samples=77
   iops        : min=  327, max=  524, avg=424.87, stdev=37.10, samples=77
  lat (msec)   : 2=35.88%, 4=63.13%, 10=0.99%
  cpu          : usr=1.62%, sys=5.83%, ctx=49315, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=49.9MiB/s (52.4MB/s), 12.5MiB/s-13.3MiB/s (13.1MB/s-13.9MB/s), io=2048MiB (2147MB), run=38553-41012msec

Disk stats (read/write):
  sda: ios=0/135169, merge=0/138746, ticks=0/71285, in_queue=77168, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2863 MiB / 0 MiB ( 2863 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2864 MiB / 0 MiB ( 0 MiB / 0 MiB )

Dunuin · Aug 29, 2021

Round 2 - Run 9 - Test X7: 4x sync sequential read/write 32K

Guest (part 2):

Code:

sync_seq_32K: (g=0): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=263MiB/s][r=8407 IOPS][eta 00m:00s]
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6110: Mon Aug 30 07:50:06 2021
  read: IOPS=2109, BW=65.9MiB/s (69.1MB/s)(512MiB/7765msec)
    clat (usec): min=302, max=2643, avg=465.17, stdev=69.84
     lat (usec): min=303, max=2645, avg=466.89, stdev=69.92
    clat percentiles (usec):
     |  1.00th=[  347],  5.00th=[  371], 10.00th=[  388], 20.00th=[  412],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  461], 60.00th=[  478],
     | 70.00th=[  494], 80.00th=[  515], 90.00th=[  545], 95.00th=[  570],
     | 99.00th=[  619], 99.50th=[  644], 99.90th=[  775], 99.95th=[ 1352],
     | 99.99th=[ 2343]
   bw (  KiB/s): min=64256, max=70784, per=25.13%, avg=67369.73, stdev=2195.66, samples=15
   iops        : min= 2008, max= 2212, avg=2105.27, stdev=68.56, samples=15
  lat (usec)   : 500=73.44%, 750=26.44%, 1000=0.05%
  lat (msec)   : 2=0.05%, 4=0.02%
  cpu          : usr=3.63%, sys=8.23%, ctx=16391, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6111: Mon Aug 30 07:50:06 2021
  read: IOPS=2094, BW=65.4MiB/s (68.6MB/s)(512MiB/7824msec)
    clat (usec): min=297, max=2924, avg=468.70, stdev=76.93
     lat (usec): min=299, max=2926, avg=470.42, stdev=77.01
    clat percentiles (usec):
     |  1.00th=[  347],  5.00th=[  375], 10.00th=[  392], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  449], 50.00th=[  465], 60.00th=[  482],
     | 70.00th=[  498], 80.00th=[  519], 90.00th=[  545], 95.00th=[  570],
     | 99.00th=[  627], 99.50th=[  652], 99.90th=[  799], 99.95th=[ 1106],
     | 99.99th=[ 2835]
   bw (  KiB/s): min=63360, max=70848, per=25.04%, avg=67105.47, stdev=1964.34, samples=15
   iops        : min= 1980, max= 2214, avg=2097.00, stdev=61.35, samples=15
  lat (usec)   : 500=71.53%, 750=28.33%, 1000=0.09%
  lat (msec)   : 2=0.01%, 4=0.04%
  cpu          : usr=3.02%, sys=8.96%, ctx=16389, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6112: Mon Aug 30 07:50:06 2021
  read: IOPS=2106, BW=65.8MiB/s (69.0MB/s)(512MiB/7779msec)
    clat (usec): min=310, max=3077, avg=465.95, stdev=77.57
     lat (usec): min=311, max=3079, avg=467.71, stdev=77.63
    clat percentiles (usec):
     |  1.00th=[  351],  5.00th=[  375], 10.00th=[  392], 20.00th=[  412],
     | 30.00th=[  433], 40.00th=[  449], 50.00th=[  461], 60.00th=[  478],
     | 70.00th=[  494], 80.00th=[  515], 90.00th=[  545], 95.00th=[  570],
     | 99.00th=[  627], 99.50th=[  652], 99.90th=[  766], 99.95th=[ 1844],
     | 99.99th=[ 2802]
   bw (  KiB/s): min=63808, max=72064, per=25.15%, avg=67404.60, stdev=2502.28, samples=15
   iops        : min= 1994, max= 2252, avg=2106.33, stdev=78.25, samples=15
  lat (usec)   : 500=73.55%, 750=26.34%, 1000=0.04%
  lat (msec)   : 2=0.02%, 4=0.05%
  cpu          : usr=2.76%, sys=9.18%, ctx=16388, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
sync_seq_32K: (groupid=0, jobs=1): err= 0: pid=6113: Mon Aug 30 07:50:06 2021
  read: IOPS=2146, BW=67.1MiB/s (70.3MB/s)(512MiB/7633msec)
    clat (usec): min=308, max=2813, avg=457.27, stdev=72.10
     lat (usec): min=310, max=2815, avg=459.00, stdev=72.13
    clat percentiles (usec):
     |  1.00th=[  347],  5.00th=[  367], 10.00th=[  383], 20.00th=[  404],
     | 30.00th=[  420], 40.00th=[  437], 50.00th=[  453], 60.00th=[  469],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  529], 95.00th=[  562],
     | 99.00th=[  611], 99.50th=[  635], 99.90th=[  734], 99.95th=[ 1205],
     | 99.99th=[ 2606]
   bw (  KiB/s): min=63168, max=72000, per=25.60%, avg=68624.67, stdev=2887.33, samples=15
   iops        : min= 1974, max= 2250, avg=2144.47, stdev=90.27, samples=15
  lat (usec)   : 500=77.72%, 750=22.20%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.03%
  cpu          : usr=2.63%, sys=8.98%, ctx=16393, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=262MiB/s (274MB/s), 65.4MiB/s-67.1MiB/s (68.6MB/s-70.3MB/s), io=2048MiB (2147MB), run=7633-7824msec

Disk stats (read/write):
  sda: ios=64632/0, merge=0/0, ticks=27968/0, in_queue=29752, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 2048 MiB )
1 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2864 MiB / 2048 MiB ( 0 MiB / 0 MiB )
Script ran 1851 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/bench/smartctl_13.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4192 MiB / 4800 MiB / 4218 MiB / 0 MiB / 5 MiB ( 4192 MiB / 4800 MiB / 4218 MiB / 0 MiB / 5 MiB )
2 min passed...4192 MiB / 4800 MiB / 4218 MiB / 0 MiB / 10 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
3 min passed...4192 MiB / 4800 MiB / 4218 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
4 min passed...4192 MiB / 4800 MiB / 4218 MiB / 0 MiB / 19 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
5 min passed...4192 MiB / 4800 MiB / 4218 MiB / 32 MiB / 23 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
6 min passed...4192 MiB / 5024 MiB / 4218 MiB / 32 MiB / 28 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 5 MiB )
7 min passed...4192 MiB / 5120 MiB / 4218 MiB / 32 MiB / 32 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 4 MiB )
8 min passed...4192 MiB / 5120 MiB / 4218 MiB / 32 MiB / 37 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
9 min passed...4192 MiB / 5120 MiB / 4218 MiB / 32 MiB / 41 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
10 min passed...4192 MiB / 5280 MiB / 4218 MiB / 32 MiB / 46 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 5 MiB )
11 min passed...4192 MiB / 5472 MiB / 4218 MiB / 32 MiB / 50 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 4 MiB )
12 min passed...4192 MiB / 5472 MiB / 4218 MiB / 64 MiB / 55 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 5 MiB )
13 min passed...4192 MiB / 5472 MiB / 4218 MiB / 64 MiB / 59 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
14 min passed...4192 MiB / 5472 MiB / 4218 MiB / 64 MiB / 64 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
15 min passed...4192 MiB / 5760 MiB / 4218 MiB / 64 MiB / 68 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
16 min passed...4192 MiB / 5824 MiB / 4218 MiB / 2112 MiB / 2121 MiB ( 0 MiB / 64 MiB / 0 MiB / 2048 MiB / 2053 MiB )
17 min passed...4192 MiB / 5824 MiB / 4218 MiB / 2112 MiB / 2126 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
18 min passed...4192 MiB / 5824 MiB / 4218 MiB / 2112 MiB / 2130 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
19 min passed...4192 MiB / 5952 MiB / 4218 MiB / 2144 MiB / 2135 MiB ( 0 MiB / 128 MiB / 0 MiB / 32 MiB / 5 MiB )
20 min passed...4192 MiB / 6144 MiB / 4218 MiB / 2144 MiB / 2139 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 4 MiB )
21 min passed...4192 MiB / 6144 MiB / 4218 MiB / 2144 MiB / 2144 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
22 min passed...4192 MiB / 6144 MiB / 4218 MiB / 2144 MiB / 2148 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )
23 min passed...4192 MiB / 6144 MiB / 4218 MiB / 2144 MiB / 2153 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
24 min passed...4192 MiB / 6432 MiB / 4218 MiB / 2144 MiB / 2157 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 4 MiB )
25 min passed...4192 MiB / 6496 MiB / 4218 MiB / 2144 MiB / 2162 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 5 MiB )
26 min passed...4192 MiB / 6496 MiB / 4218 MiB / 2176 MiB / 2166 MiB ( 0 MiB / 0 MiB / 0 MiB / 32 MiB / 4 MiB )
27 min passed...4192 MiB / 6496 MiB / 4218 MiB / 2176 MiB / 2171 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 5 MiB )
28 min passed...4192 MiB / 6624 MiB / 4218 MiB / 2176 MiB / 2175 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 4 MiB )
29 min passed...4192 MiB / 6816 MiB / 4218 MiB / 2176 MiB / 2180 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 5 MiB )
30 min passed...4192 MiB / 6816 MiB / 4218 MiB / 2176 MiB / 2184 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 4 MiB )

Dunuin · Aug 29, 2021

Anything else I should test? Otherwise I'm done now and will look what will be best for my needs and go back to production use.

Dunuin · Aug 30, 2021

In Round 2 Run 10 I will run all the fio scripts on the host and fio will write directly to a single S3710 200GB SSD on block level.
So this should show what the SSD is capable of.

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Famous Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member