how to best benchmark SSDs?

Special Test Nr. 17: Like Round 2 Run 3 Test 4 (async 4K on 2 disk mirror with 4k volblocksize) but with xfs with journaling instead of ext4 with journaling

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=132MiB/s][w=33.8k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=729: Wed Aug 11 21:27:23 2021
  write: IOPS=32.9k, BW=129MiB/s (135MB/s)(1024MiB/7960msec); 0 zone resets
    slat (usec): min=3, max=54698, avg=17.43, stdev=190.03
    clat (usec): min=35, max=58551, avg=1916.67, stdev=1908.16
     lat (usec): min=49, max=58575, avg=1936.18, stdev=1918.46
    clat percentiles (usec):
     |  1.00th=[ 1418],  5.00th=[ 1516], 10.00th=[ 1565], 20.00th=[ 1614],
     | 30.00th=[ 1647], 40.00th=[ 1696], 50.00th=[ 1729], 60.00th=[ 1778],
     | 70.00th=[ 1844], 80.00th=[ 1958], 90.00th=[ 2212], 95.00th=[ 2540],
     | 99.00th=[ 3621], 99.50th=[ 4146], 99.90th=[27657], 99.95th=[56886],
     | 99.99th=[58459]
   bw (  KiB/s): min=97480, max=151632, per=99.54%, avg=131126.40, stdev=15946.08, samples=15
   iops        : min=24370, max=37908, avg=32781.60, stdev=3986.52, samples=15
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=82.42%, 4=16.98%, 10=0.28%, 20=0.07%, 50=0.10%
  lat (msec)   : 100=0.10%
  cpu          : usr=19.99%, sys=35.10%, ctx=131237, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=1024MiB (1074MB), run=7960-7960msec

Disk stats (read/write):
  sdb: ios=1/263181, merge=0/9, ticks=1/62611, in_queue=64548, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 1126 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1126 MiB / 18 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=296MiB/s][r=75.8k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=1303: Wed Aug 11 21:42:28 2021
  read: IOPS=75.5k, BW=295MiB/s (309MB/s)(1024MiB/3471msec)
    slat (usec): min=3, max=219, avg= 6.73, stdev= 3.41
    clat (usec): min=173, max=3943, avg=835.92, stdev=70.31
     lat (usec): min=187, max=3955, avg=843.78, stdev=70.66
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  775], 10.00th=[  783], 20.00th=[  799],
     | 30.00th=[  807], 40.00th=[  824], 50.00th=[  832], 60.00th=[  840],
     | 70.00th=[  848], 80.00th=[  865], 90.00th=[  889], 95.00th=[  906],
     | 99.00th=[  955], 99.50th=[ 1057], 99.90th=[ 1663], 99.95th=[ 2073],
     | 99.99th=[ 2737]
   bw (  KiB/s): min=297408, max=307304, per=100.00%, avg=302238.67, stdev=3186.17, samples=6
   iops        : min=74352, max=76826, avg=75559.67, stdev=796.54, samples=6
  lat (usec)   : 250=0.01%, 500=0.01%, 750=1.27%, 1000=98.11%
  lat (msec)   : 2=0.55%, 4=0.06%
  cpu          : usr=42.62%, sys=57.32%, ctx=12, majf=0, minf=72
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=1024MiB (1074MB), run=3471-3471msec

Disk stats (read/write):
  sdb: ios=256697/0, merge=0/0, ticks=52061/0, in_queue=52112, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1126 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 1813 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4704 MiB / 5216 MiB / 4696 MiB / 0 MiB / 18 MiB ( 4704 MiB / 5216 MiB / 4696 MiB / 0 MiB / 18 MiB )
2 min passed...4704 MiB / 5216 MiB / 4703 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 0 MiB )
3 min passed...4736 MiB / 5280 MiB / 4705 MiB / 0 MiB / 18 MiB ( 32 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...4736 MiB / 5568 MiB / 4708 MiB / 0 MiB / 18 MiB ( 0 MiB / 288 MiB / 3 MiB / 0 MiB / 0 MiB )
5 min passed...4736 MiB / 5696 MiB / 4709 MiB / 0 MiB / 18 MiB ( 0 MiB / 128 MiB / 1 MiB / 0 MiB / 0 MiB )
6 min passed...4736 MiB / 5760 MiB / 4709 MiB / 0 MiB / 18 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...4736 MiB / 5760 MiB / 4709 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...4736 MiB / 5920 MiB / 4714 MiB / 0 MiB / 18 MiB ( 0 MiB / 160 MiB / 5 MiB / 0 MiB / 0 MiB )
9 min passed...4736 MiB / 5920 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...4736 MiB / 6144 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...4736 MiB / 6144 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...4736 MiB / 6240 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...4736 MiB / 6496 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...4736 MiB / 6656 MiB / 4718 MiB / 0 MiB / 18 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...4736 MiB / 6688 MiB / 4724 MiB / 0 MiB / 18 MiB ( 0 MiB / 32 MiB / 6 MiB / 0 MiB / 0 MiB )
16 min passed...4736 MiB / 6688 MiB / 4724 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 0 MiB / 1024 MiB / 1024 MiB )
17 min passed...4736 MiB / 6944 MiB / 4727 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
18 min passed...4736 MiB / 7040 MiB / 4729 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 96 MiB / 2 MiB / 0 MiB / 0 MiB )
19 min passed...4736 MiB / 7232 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 192 MiB / 4 MiB / 0 MiB / 0 MiB )
20 min passed...4736 MiB / 7232 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...4736 MiB / 7328 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...4736 MiB / 7552 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...4736 MiB / 7712 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...4736 MiB / 7712 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...4736 MiB / 7712 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...4736 MiB / 7936 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...4736 MiB / 7936 MiB / 4733 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...4736 MiB / 8224 MiB / 4735 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 288 MiB / 2 MiB / 0 MiB / 0 MiB )
29 min passed...4736 MiB / 8224 MiB / 4740 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
30 min passed...4736 MiB / 8288 MiB / 4740 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
 
4k sync/async write/read on 2 disk ZFS mirror using ashift=12 & volblocksize=4K comparing ext4 with journaling vs ext4 without journaling vs xfs with journaling:
ext4 journal syncext4 no journal syncxfs journal sync
ext4 journal async		ext4 no journal async
xfs journal async
Write Performance:1,87 MiB/s4,26 MiB/s2,4 MiB/s125 MiB/s133 MiB/s129 MiB/s
Read Performance:9,25 MiB/s9,67 MiB/s9,67 MiB/s295 MiB/s282 MiB/s295 MiB/s
W.A. fio -> guest:5,11 x2,24 x2,86 x1,01 x1 x1 x
W.A. guest -> host:10,02 x9,67 x12,34 x3,67 x2,17 x4,20 x
W.A. host -> NAND:1,11 x1,15 x1,09 x1,46 x1,46 x1,41 x
W.A. total:56,84 x25,19 x38,38 x5,44 x3,16 x6,53 x
 
  • Like
Reactions: anch and guletz
16k sync writes/reads that are 50% compressible read/written to a xfs partition on a zvol (volblocksize=8K) on a 4 disk striped mirror (ashift=12):

aes-256-gcm + lz4aes-128-gcm + lz4aes-256-gcm + no compressionno encryption + lz4
Write Performance:8 MiB/s8,09 MiB/s7,78 MiB/s10,1 MiB/s
Read Performance:29,2 MiB/s31,9 MiB/s29,8 MiB/s38,8 MiB/s
W.A. fio -> guest:1,48 x1,48 x1,48 x1,48 x
W.A. guest -> host:7,23 x7,21 x8,1 x3,67 x
W.A. host -> NAND:1,13 x1,15 x1,15 x1,12 x
W.A. total:12,09 x12,25 x13,78 x6,09 x
R.A. total:0,5 x0,5 x1,0 x0,5 x

Conclusion:
Enableing encryption, no matter if 128bit or 256bit will double the write amplification. Using encryption also lowers the IOPS/performance. Its good to enable LZ4 compression because it will lower the write amplification a bit and increase the performance if the data is compressible.
 
Last edited:
  • Like
Reactions: guletz
This is the pool and VM that will be used in Round 2 Run 4:

Host is a striped mirror of 4x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=8K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 3 are bold)

root@Hypervisor:~# zpool status
Code: 
 pool: VMpool11
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool11                                        ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN  ONLINE       0     0     0

errors: No known data errors

root@Hypervisor:~# zpool get all
Code: 
NAME      PROPERTY                       VALUE                          SOURCE
VMpool11  size                           372G                           -
VMpool11  capacity                       0%                             -
VMpool11  altroot                        -                              default
VMpool11  health                         ONLINE                         -
VMpool11  guid                           16300487031154975288           -
VMpool11  version                        -                              default
VMpool11  bootfs                         -                              default
VMpool11  delegation                     on                             default
VMpool11  autoreplace                    off                            default
VMpool11  cachefile                      -                              default
VMpool11  failmode                       wait                           default
VMpool11  listsnapshots                  off                            default
VMpool11  autoexpand                     off                            default
VMpool11  dedupratio                     1.00x                          -
VMpool11  free                           370G                           -
VMpool11  allocated                      2.09G                          -
VMpool11  readonly                       off                            -
VMpool11  ashift                         12                             local
VMpool11  comment                        -                              default
VMpool11  expandsize                     -                              -
VMpool11  freeing                        0                              -
VMpool11  fragmentation                  1%                             -
VMpool11  leaked                         0                              -
VMpool11  multihost                      off                            default
VMpool11  checkpoint                     -                              -
VMpool11  load_guid                      6200048113594507635            -
VMpool11  autotrim                       off                            default
VMpool11  feature@async_destroy          enabled                        local
VMpool11  feature@empty_bpobj            active                         local
VMpool11  feature@lz4_compress           active                         local
VMpool11  feature@multi_vdev_crash_dump  enabled                        local
VMpool11  feature@spacemap_histogram     active                         local
VMpool11  feature@enabled_txg            active                         local
VMpool11  feature@hole_birth             active                         local
VMpool11  feature@extensible_dataset     active                         local
VMpool11  feature@embedded_data          active                         local
VMpool11  feature@bookmarks              enabled                        local
VMpool11  feature@filesystem_limits      enabled                        local
VMpool11  feature@large_blocks           enabled                        local
VMpool11  feature@large_dnode            enabled                        local
VMpool11  feature@sha512                 enabled                        local
VMpool11  feature@skein                  enabled                        local
VMpool11  feature@edonr                  enabled                        local
VMpool11  feature@userobj_accounting     active                         local
VMpool11  feature@encryption             active                         local
VMpool11  feature@project_quota          active                         local
VMpool11  feature@device_removal         enabled                        local
VMpool11  feature@obsolete_counts        enabled                        local
VMpool11  feature@zpool_checkpoint       enabled                        local
VMpool11  feature@spacemap_v2            active                         local
VMpool11  feature@allocation_classes     enabled                        local
VMpool11  feature@resilver_defer         enabled                        local
VMpool11  feature@bookmark_v2            enabled                        local
VMpool11  feature@redaction_bookmarks    enabled                        local
VMpool11  feature@redacted_datasets      enabled                        local
VMpool11  feature@bookmark_written       enabled                        local
VMpool11  feature@log_spacemap           active                         local
VMpool11  feature@livelist               enabled                        local
VMpool11  feature@device_rebuild         enabled                        local
VMpool11  feature@zstd_compress          enabled                        local

root@Hypervisor:~# zfs get all VMpool10/VLT/VM
Code: 
NAME             PROPERTY              VALUE                  SOURCE
VMpool11/VLT/VM  type                  filesystem             -
VMpool11/VLT/VM  creation              Tue Aug 10  0:07 2021  -
VMpool11/VLT/VM  used                  2.07G                  -
VMpool11/VLT/VM  available             286G                   -
VMpool11/VLT/VM  referenced            192K                   -
VMpool11/VLT/VM  compressratio         1.21x                  -
VMpool11/VLT/VM  mounted               yes                    -
VMpool11/VLT/VM  quota                 none                   default
VMpool11/VLT/VM  reservation           none                   default
VMpool11/VLT/VM  recordsize            128K                   default
VMpool11/VLT/VM  mountpoint            /VMpool11/VLT/VM       default
VMpool11/VLT/VM  sharenfs              off                    default
VMpool11/VLT/VM  checksum              on                     default
VMpool11/VLT/VM  compression           lz4                    inherited from VMpool11
VMpool11/VLT/VM  atime                 off                    inherited from VMpool11
VMpool11/VLT/VM  devices               on                     default
VMpool11/VLT/VM  exec                  on                     default
VMpool11/VLT/VM  setuid                on                     default
VMpool11/VLT/VM  readonly              off                    default
VMpool11/VLT/VM  zoned                 off                    default
VMpool11/VLT/VM  snapdir               hidden                 default
VMpool11/VLT/VM  aclmode               discard                default
VMpool11/VLT/VM  aclinherit            restricted             default
VMpool11/VLT/VM  createtxg             37                     -
VMpool11/VLT/VM  canmount              on                     local
VMpool11/VLT/VM  xattr                 on                     default
VMpool11/VLT/VM  copies                1                      default
VMpool11/VLT/VM  version               5                      -
VMpool11/VLT/VM  utf8only              off                    -
VMpool11/VLT/VM  normalization         none                   -
VMpool11/VLT/VM  casesensitivity       sensitive              -
VMpool11/VLT/VM  vscan                 off                    default
VMpool11/VLT/VM  nbmand                off                    default
VMpool11/VLT/VM  sharesmb              off                    default
VMpool11/VLT/VM  refquota              none                   default
VMpool11/VLT/VM  refreservation        none                   default
VMpool11/VLT/VM  guid                  1768884776605799805    -
VMpool11/VLT/VM  primarycache          metadata               inherited from VMpool11
VMpool11/VLT/VM  secondarycache        all                    default
VMpool11/VLT/VM  usedbysnapshots       0B                     -
VMpool11/VLT/VM  usedbydataset         192K                   -
VMpool11/VLT/VM  usedbychildren        2.07G                  -
VMpool11/VLT/VM  usedbyrefreservation  0B                     -
VMpool11/VLT/VM  logbias               latency                default
VMpool11/VLT/VM  objsetid              1667                   -
VMpool11/VLT/VM  dedup                 off                    default
VMpool11/VLT/VM  mlslabel              none                   default
VMpool11/VLT/VM  sync                  standard               inherited from VMpool11
VMpool11/VLT/VM  dnodesize             legacy                 default
VMpool11/VLT/VM  refcompressratio      1.00x                  -
VMpool11/VLT/VM  written               192K                   -
VMpool11/VLT/VM  logicalused           2.49G                  -
VMpool11/VLT/VM  logicalreferenced     69K                    -
VMpool11/VLT/VM  volmode               default                default
VMpool11/VLT/VM  filesystem_limit      none                   default
VMpool11/VLT/VM  snapshot_limit        none                   default
VMpool11/VLT/VM  filesystem_count      none                   default
VMpool11/VLT/VM  snapshot_count        none                   default
VMpool11/VLT/VM  snapdev               hidden                 default
VMpool11/VLT/VM  acltype               off                    default
VMpool11/VLT/VM  context               none                   default
VMpool11/VLT/VM  fscontext             none                   default
VMpool11/VLT/VM  defcontext            none                   default
VMpool11/VLT/VM  rootcontext           none                   default
VMpool11/VLT/VM  relatime              off                    default
VMpool11/VLT/VM  redundant_metadata    all                    default
VMpool11/VLT/VM  overlay               on                     default
VMpool11/VLT/VM  encryption            aes-256-gcm            -
VMpool11/VLT/VM  keylocation           none                   default
VMpool11/VLT/VM  keyformat             passphrase             -
VMpool11/VLT/VM  pbkdf2iters           350000                 -
VMpool11/VLT/VM  encryptionroot        VMpool11/VLT           -
VMpool11/VLT/VM  keystatus             available              -
VMpool11/VLT/VM  special_small_blocks  0                      default

Commands used to create the pool:
Code: 
# create mirror of 4 disks (4x Intel S3710 200GB):
zpool create -f -o ashift=12 VMpool11 mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN
# set ZFS options
zpool set autoexpand=off VMpool11
zpool set feature@encryption=enabled VMpool11
zfs set compression=lz4 VMpool11
zfs set sync=standard VMpool11
zfs set canmount=off VMpool11
zfs set atime=off VMpool11
# set quota to 80% so pool can't get full/slow:
zfs set quota=288G VMpool11
# create encrypted dataset (256bit aes):
zfs create -o encryption=on -o keyformat=passphrase VMpool11/VLT
zfs set canmount=off VMpool11/VLT
# create encrypted dataset (128bit aes):
zfs create -o encryption=aes-128-gcm -o keyformat=passphrase VMpool11/VLT2
zfs set canmount=off VMpool11/VLT2
# create dataset for encrypted VMs
zfs create -o encryption=on VMpool11/VLT/VM
zfs set canmount=on VMpool11/VLT/VM
# create another dataset for encrypted VMs
zfs create -o encryption=on VMpool11/VLT2/VM
zfs set canmount=on VMpool11/VLT2/VM
# create unencrypted dataset:
zfs create VMpool11/NE
zfs set canmount=off VMpool11/NE
# create dataset for unencrypted VMs
zfs create VMpool11/NE/VM
zfs set canmount=on VMpool11/NE/VM
# add dataset to PVE:
pvesm add zfspool VMpool11_VLT_VM -pool VMpool11/VLT/VM --content images,rootdir --sparse 1 --mountpoint /VMpool11/VLT/VM
pvesm add zfspool VMpool11_NE_VM -pool VMpool11/NE/VM --content images,rootdir --sparse 1 --mountpoint /VMpool11/NE/VM
pvesm add zfspool VMpool11_VLT2_VM -pool VMpool11/VLT2/VM --content images,rootdir --sparse 1 --mountpoint /VMpool11/VLT2/VM
# disable ARC caching:
zfs set primarycache=metadata VMpool11
 
Round 2 - Run 4 - Test 1: sync 4K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1116KiB/s][w=279 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=684: Wed Aug 11 23:55:30 2021
  write: IOPS=281, BW=1126KiB/s (1153kB/s)(1024MiB/930999msec); 0 zone resets
    clat (usec): min=1667, max=41859, avg=3536.67, stdev=1059.64
     lat (usec): min=1668, max=41862, avg=3538.86, stdev=1059.67
    clat percentiles (usec):
     |  1.00th=[ 2409],  5.00th=[ 2704], 10.00th=[ 2868], 20.00th=[ 3064],
     | 30.00th=[ 3195], 40.00th=[ 3294], 50.00th=[ 3425], 60.00th=[ 3523],
     | 70.00th=[ 3621], 80.00th=[ 3720], 90.00th=[ 3949], 95.00th=[ 5080],
     | 99.00th=[ 6259], 99.50th=[ 7701], 99.90th=[17171], 99.95th=[20579],
     | 99.99th=[30802]
   bw (  KiB/s): min=  872, max= 1528, per=100.00%, avg=1126.19, stdev=83.99, samples=1861
   iops        : min=  218, max=  382, avg=281.54, stdev=21.00, samples=1861
  lat (msec)   : 2=0.04%, 4=90.63%, 10=9.01%, 20=0.27%, 50=0.06%
  cpu          : usr=0.68%, sys=4.15%, ctx=634862, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1126KiB/s (1153kB/s), 1126KiB/s-1126KiB/s (1153kB/s-1153kB/s), io=1024MiB (1074MB), run=930999-930999msec

Disk stats (read/write):
  sda: ios=25/797872, merge=0/538769, ticks=11/865844, in_queue=929560, util=99.82%
0 min passed...Guest Writes/Reads (Changes): 5209 MiB / 18 MiB ( 5209 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 5210 MiB / 18 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5210 MiB / 18 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5210 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5210 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5210 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 3 MiB / 3 MiB )
7 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5213 MiB / 21 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][99.2%][r=8056KiB/s][r=2014 IOPS][eta 00m:01s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=912: Thu Aug 12 00:12:31 2021
  read: IOPS=2168, BW=8675KiB/s (8883kB/s)(1024MiB/120871msec)
    clat (usec): min=195, max=5372, avg=450.45, stdev=144.78
     lat (usec): min=197, max=5375, avg=452.44, stdev=144.85
    clat percentiles (usec):
     |  1.00th=[  289],  5.00th=[  310], 10.00th=[  326], 20.00th=[  351],
     | 30.00th=[  375], 40.00th=[  400], 50.00th=[  429], 60.00th=[  461],
     | 70.00th=[  502], 80.00th=[  545], 90.00th=[  594], 95.00th=[  627],
     | 99.00th=[  676], 99.50th=[  693], 99.90th=[ 2573], 99.95th=[ 2999],
     | 99.99th=[ 3523]
   bw (  KiB/s): min= 6824, max=12760, per=99.95%, avg=8670.24, stdev=1386.48, samples=241
   iops        : min= 1706, max= 3190, avg=2167.54, stdev=346.62, samples=241
  lat (usec)   : 250=0.05%, 500=69.63%, 750=29.93%, 1000=0.08%
  lat (msec)   : 2=0.15%, 4=0.15%, 10=0.01%
  cpu          : usr=3.61%, sys=11.10%, ctx=262172, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=8675KiB/s (8883kB/s), 8675KiB/s-8675KiB/s (8883kB/s-8883kB/s), io=1024MiB (1074MB), run=120871-120871msec

Disk stats (read/write):
  sda: ios=261577/0, merge=0/0, ticks=108518/0, in_queue=120556, util=99.92%
0 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5213 MiB / 1045 MiB ( 0 MiB / 0 MiB )
Script ran 2854 seconds or 47 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3168 MiB / 3520 MiB / 3275 MiB / 416 MiB / 396 MiB ( 3168 MiB / 3520 MiB / 3275 MiB / 416 MiB / 396 MiB )
2 min passed...6624 MiB / 7328 MiB / 6621 MiB / 832 MiB / 798 MiB ( 3456 MiB / 3808 MiB / 3346 MiB / 416 MiB / 402 MiB )
3 min passed...10016 MiB / 10912 MiB / 10029 MiB / 1216 MiB / 1206 MiB ( 3392 MiB / 3584 MiB / 3408 MiB / 384 MiB / 408 MiB )
4 min passed...13472 MiB / 14752 MiB / 13442 MiB / 1600 MiB / 1609 MiB ( 3456 MiB / 3840 MiB / 3413 MiB / 384 MiB / 403 MiB )
5 min passed...16832 MiB / 18848 MiB / 16845 MiB / 1984 MiB / 2003 MiB ( 3360 MiB / 4096 MiB / 3403 MiB / 384 MiB / 394 MiB )
6 min passed...20192 MiB / 22560 MiB / 20203 MiB / 2368 MiB / 2400 MiB ( 3360 MiB / 3712 MiB / 3358 MiB / 384 MiB / 397 MiB )
7 min passed...23648 MiB / 26528 MiB / 23659 MiB / 2784 MiB / 2801 MiB ( 3456 MiB / 3968 MiB / 3456 MiB / 416 MiB / 401 MiB )
8 min passed...27104 MiB / 30144 MiB / 27113 MiB / 3200 MiB / 3202 MiB ( 3456 MiB / 3616 MiB / 3454 MiB / 416 MiB / 401 MiB )
9 min passed...30528 MiB / 33952 MiB / 30520 MiB / 3616 MiB / 3596 MiB ( 3424 MiB / 3808 MiB / 3407 MiB / 416 MiB / 394 MiB )
10 min passed...33920 MiB / 37792 MiB / 33951 MiB / 4032 MiB / 3995 MiB ( 3392 MiB / 3840 MiB / 3431 MiB / 416 MiB / 399 MiB )
11 min passed...37248 MiB / 41632 MiB / 37285 MiB / 4416 MiB / 4395 MiB ( 3328 MiB / 3840 MiB / 3334 MiB / 384 MiB / 400 MiB )
12 min passed...40672 MiB / 45376 MiB / 40703 MiB / 4800 MiB / 4794 MiB ( 3424 MiB / 3744 MiB / 3418 MiB / 384 MiB / 399 MiB )
13 min passed...44128 MiB / 48992 MiB / 44118 MiB / 5216 MiB / 5192 MiB ( 3456 MiB / 3616 MiB / 3415 MiB / 416 MiB / 398 MiB )
14 min passed...47584 MiB / 52704 MiB / 47605 MiB / 5600 MiB / 5597 MiB ( 3456 MiB / 3712 MiB / 3487 MiB / 384 MiB / 405 MiB )
15 min passed...51008 MiB / 56576 MiB / 51018 MiB / 5984 MiB / 5982 MiB ( 3424 MiB / 3872 MiB / 3413 MiB / 384 MiB / 385 MiB )
16 min passed...52512 MiB / 58592 MiB / 52485 MiB / 6112 MiB / 6147 MiB ( 1504 MiB / 2016 MiB / 1467 MiB / 128 MiB / 165 MiB )
17 min passed...52512 MiB / 58592 MiB / 52489 MiB / 6112 MiB / 6147 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
18 min passed...52512 MiB / 58592 MiB / 52489 MiB / 6112 MiB / 6147 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...52512 MiB / 58592 MiB / 52489 MiB / 6112 MiB / 6147 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...52512 MiB / 58816 MiB / 52498 MiB / 6112 MiB / 6147 MiB ( 0 MiB / 224 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...52512 MiB / 59296 MiB / 52521 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 480 MiB / 23 MiB / 0 MiB / 3 MiB )
22 min passed...52512 MiB / 59424 MiB / 52528 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 128 MiB / 7 MiB / 0 MiB / 0 MiB )
23 min passed...52512 MiB / 59424 MiB / 52528 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...52512 MiB / 59424 MiB / 52528 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...52512 MiB / 59424 MiB / 52528 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...52512 MiB / 59776 MiB / 52534 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 352 MiB / 6 MiB / 0 MiB / 0 MiB )
27 min passed...52512 MiB / 59872 MiB / 52534 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...52512 MiB / 59872 MiB / 52534 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...52512 MiB / 59872 MiB / 52534 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...52512 MiB / 59872 MiB / 52534 MiB / 6112 MiB / 6150 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...52512 MiB / 60128 MiB / 52534 MiB / 6784 MiB / 6792 MiB ( 0 MiB / 256 MiB / 0 MiB / 672 MiB / 642 MiB )
32 min passed...52512 MiB / 60224 MiB / 52534 MiB / 7840 MiB / 7830 MiB ( 0 MiB / 96 MiB / 0 MiB / 1056 MiB / 1038 MiB )
33 min passed...52512 MiB / 60224 MiB / 52534 MiB / 8224 MiB / 8198 MiB ( 0 MiB / 0 MiB / 0 MiB / 384 MiB / 368 MiB )
34 min passed...52512 MiB / 60224 MiB / 52534 MiB / 8224 MiB / 8198 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...52512 MiB / 60224 MiB / 52534 MiB / 8224 MiB / 8198 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...52512 MiB / 60448 MiB / 52542 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 224 MiB / 8 MiB / 0 MiB / 1 MiB )
37 min passed...52512 MiB / 60576 MiB / 52544 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 128 MiB / 2 MiB / 0 MiB / 0 MiB )
38 min passed...52512 MiB / 60576 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 6 MiB / 0 MiB / 0 MiB )
39 min passed...52512 MiB / 60576 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...52512 MiB / 60576 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...52512 MiB / 60768 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
42 min passed...52512 MiB / 60896 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...52512 MiB / 60896 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...52512 MiB / 60896 MiB / 52550 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...52512 MiB / 60896 MiB / 52552 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
46 min passed...52512 MiB / 61024 MiB / 52558 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 128 MiB / 6 MiB / 0 MiB / 0 MiB )
47 min passed...52512 MiB / 61248 MiB / 52558 MiB / 8224 MiB / 8199 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 2: sync 16K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=5765KiB/s][w=360 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=640: Thu Aug 12 00:39:03 2021
  write: IOPS=351, BW=5624KiB/s (5759kB/s)(1024MiB/186444msec); 0 zone resets
    clat (usec): min=1413, max=43234, avg=2824.88, stdev=867.32
     lat (usec): min=1415, max=43236, avg=2827.08, stdev=867.32
    clat percentiles (usec):
     |  1.00th=[ 1991],  5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2474],
     | 30.00th=[ 2573], 40.00th=[ 2671], 50.00th=[ 2737], 60.00th=[ 2802],
     | 70.00th=[ 2900], 80.00th=[ 3032], 90.00th=[ 3163], 95.00th=[ 3359],
     | 99.00th=[ 5342], 99.50th=[ 7308], 99.90th=[13304], 99.95th=[16450],
     | 99.99th=[31851]
   bw (  KiB/s): min= 4352, max= 6944, per=99.98%, avg=5622.80, stdev=460.20, samples=372
   iops        : min=  272, max=  434, avg=351.42, stdev=28.77, samples=372
  lat (msec)   : 2=1.05%, 4=96.04%, 10=2.64%, 20=0.23%, 50=0.03%
  cpu          : usr=1.06%, sys=5.07%, ctx=144972, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5624KiB/s (5759kB/s), 5624KiB/s-5624KiB/s (5759kB/s-5759kB/s), io=1024MiB (1074MB), run=186444-186444msec

Disk stats (read/write):
  sda: ios=0/196863, merge=0/132238, ticks=0/168437, in_queue=186016, util=99.79%
0 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 2054 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2054 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=29.2MiB/s][r=1870 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=841: Thu Aug 12 00:54:37 2021
  read: IOPS=1928, BW=30.1MiB/s (31.6MB/s)(1024MiB/33979msec)
    clat (usec): min=309, max=4661, avg=507.96, stdev=85.77
     lat (usec): min=311, max=4663, avg=509.86, stdev=85.88
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  383], 10.00th=[  404], 20.00th=[  433],
     | 30.00th=[  457], 40.00th=[  478], 50.00th=[  502], 60.00th=[  523],
     | 70.00th=[  553], 80.00th=[  586], 90.00th=[  627], 95.00th=[  652],
     | 99.00th=[  693], 99.50th=[  709], 99.90th=[  725], 99.95th=[  734],
     | 99.99th=[ 1004]
   bw (  KiB/s): min=26752, max=37056, per=100.00%, avg=30882.48, stdev=2916.19, samples=67
   iops        : min= 1672, max= 2316, avg=1930.13, stdev=182.26, samples=67
  lat (usec)   : 500=49.91%, 750=50.06%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.10%, sys=10.46%, ctx=65591, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.1MiB/s (31.6MB/s), 30.1MiB/s-30.1MiB/s (31.6MB/s-31.6MB/s), io=1024MiB (1074MB), run=33979-33979msec

Disk stats (read/write):
  sda: ios=65391/0, merge=0/0, ticks=30724/0, in_queue=33996, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2054 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 2022 seconds or 33 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4224 MiB / 4832 MiB / 4264 MiB / 224 MiB / 172 MiB ( 4224 MiB / 4832 MiB / 4264 MiB / 224 MiB / 172 MiB )
2 min passed...8800 MiB / 9632 MiB / 8819 MiB / 384 MiB / 341 MiB ( 4576 MiB / 4800 MiB / 4555 MiB / 160 MiB / 169 MiB )
3 min passed...13216 MiB / 14368 MiB / 13279 MiB / 544 MiB / 508 MiB ( 4416 MiB / 4736 MiB / 4460 MiB / 160 MiB / 167 MiB )
4 min passed...13824 MiB / 15136 MiB / 13888 MiB / 544 MiB / 530 MiB ( 608 MiB / 768 MiB / 609 MiB / 0 MiB / 22 MiB )
5 min passed...13824 MiB / 15136 MiB / 13888 MiB / 544 MiB / 530 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...13824 MiB / 15360 MiB / 13888 MiB / 544 MiB / 530 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...13824 MiB / 15488 MiB / 13888 MiB / 544 MiB / 530 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...13824 MiB / 15488 MiB / 13890 MiB / 544 MiB / 530 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...13824 MiB / 15552 MiB / 13891 MiB / 544 MiB / 530 MiB ( 0 MiB / 64 MiB / 1 MiB / 0 MiB / 0 MiB )
10 min passed...13856 MiB / 15616 MiB / 13899 MiB / 544 MiB / 530 MiB ( 32 MiB / 64 MiB / 8 MiB / 0 MiB / 0 MiB )
11 min passed...13856 MiB / 15872 MiB / 13899 MiB / 544 MiB / 530 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...13856 MiB / 16192 MiB / 13899 MiB / 544 MiB / 530 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...13856 MiB / 16192 MiB / 13899 MiB / 544 MiB / 530 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...13856 MiB / 16192 MiB / 13899 MiB / 544 MiB / 530 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...13856 MiB / 16384 MiB / 13904 MiB / 544 MiB / 531 MiB ( 0 MiB / 192 MiB / 5 MiB / 0 MiB / 1 MiB )
16 min passed...13856 MiB / 16576 MiB / 13906 MiB / 544 MiB / 531 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
17 min passed...13888 MiB / 16928 MiB / 13910 MiB / 544 MiB / 531 MiB ( 32 MiB / 352 MiB / 4 MiB / 0 MiB / 0 MiB )
18 min passed...13888 MiB / 16960 MiB / 13910 MiB / 544 MiB / 531 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...13888 MiB / 16960 MiB / 13910 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 0 MiB / 0 MiB / 1056 MiB / 1024 MiB )
20 min passed...13888 MiB / 17184 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 224 MiB / 8 MiB / 0 MiB / 0 MiB )
21 min passed...13888 MiB / 17248 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...13888 MiB / 17504 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...13888 MiB / 17504 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...13888 MiB / 17632 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...13888 MiB / 17664 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...13888 MiB / 17696 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...13888 MiB / 18208 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...13888 MiB / 18304 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...13888 MiB / 18464 MiB / 13918 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...13920 MiB / 18528 MiB / 13926 MiB / 1600 MiB / 1555 MiB ( 32 MiB / 64 MiB / 8 MiB / 0 MiB / 0 MiB )
31 min passed...13920 MiB / 18528 MiB / 13926 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...13920 MiB / 18944 MiB / 13926 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...13920 MiB / 19168 MiB / 13926 MiB / 1600 MiB / 1555 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 3: sync 4M sequential read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=228MiB/s][w=57 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=651: Thu Aug 12 01:16:15 2021
  write: IOPS=47, BW=189MiB/s (198MB/s)(10.0GiB/54209msec); 0 zone resets
    clat (usec): min=12807, max=77551, avg=19515.69, stdev=6358.42
     lat (usec): min=12810, max=77554, avg=19519.16, stdev=6358.37
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[14353], 10.00th=[14615], 20.00th=[14877],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[23200], 80.00th=[25822], 90.00th=[28181], 95.00th=[30278],
     | 99.00th=[37487], 99.50th=[43779], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[77071]
   bw (  KiB/s): min=114688, max=253952, per=99.86%, avg=193170.05, stdev=43262.49, samples=108
   iops        : min=   28, max=   62, avg=47.13, stdev=10.57, samples=108
  lat (msec)   : 20=67.70%, 50=32.03%, 100=0.27%
  cpu          : usr=7.91%, sys=2.03%, ctx=5782, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=10.0GiB (10.7GB), run=54209-54209msec

Disk stats (read/write):
  sda: ios=0/15355, merge=0/5285, ticks=0/72584, in_queue=77516, util=97.98%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 10281 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=732MiB/s][r=183 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=857: Thu Aug 12 01:31:30 2021
  read: IOPS=185, BW=740MiB/s (776MB/s)(10.0GiB/13831msec)
    clat (usec): min=4305, max=14962, avg=5392.84, stdev=1063.06
     lat (usec): min=4306, max=14964, avg=5394.41, stdev=1063.06
    clat percentiles (usec):
     |  1.00th=[ 4490],  5.00th=[ 4555], 10.00th=[ 4621], 20.00th=[ 4752],
     | 30.00th=[ 4817], 40.00th=[ 4883], 50.00th=[ 5014], 60.00th=[ 5145],
     | 70.00th=[ 5342], 80.00th=[ 5800], 90.00th=[ 6718], 95.00th=[ 7439],
     | 99.00th=[ 9896], 99.50th=[10421], 99.90th=[12780], 99.95th=[13960],
     | 99.99th=[15008]
   bw (  KiB/s): min=720896, max=794624, per=99.84%, avg=756885.74, stdev=20207.76, samples=27
   iops        : min=  176, max=  194, avg=184.74, stdev= 4.88, samples=27
  lat (msec)   : 10=99.14%, 20=0.86%
  cpu          : usr=0.33%, sys=4.19%, ctx=2563, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=740MiB/s (776MB/s), 740MiB/s-740MiB/s (776MB/s-776MB/s), io=10.0GiB (10.7GB), run=13831-13831msec

Disk stats (read/write):
  sda: ios=10095/0, merge=0/0, ticks=45503/0, in_queue=45824, util=99.53%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10281 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Script ran 1870 seconds or 31 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...42624 MiB / 45184 MiB / 42614 MiB / 64 MiB / 37 MiB ( 42624 MiB / 45184 MiB / 42614 MiB / 64 MiB / 37 MiB )
2 min passed...42624 MiB / 45408 MiB / 42619 MiB / 64 MiB / 37 MiB ( 0 MiB / 224 MiB / 5 MiB / 0 MiB / 0 MiB )
3 min passed...42624 MiB / 45920 MiB / 42625 MiB / 64 MiB / 37 MiB ( 0 MiB / 512 MiB / 6 MiB / 0 MiB / 0 MiB )
4 min passed...42624 MiB / 46080 MiB / 42625 MiB / 64 MiB / 37 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...42624 MiB / 46080 MiB / 42625 MiB / 64 MiB / 37 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...42624 MiB / 46080 MiB / 42625 MiB / 64 MiB / 37 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...42624 MiB / 46336 MiB / 42629 MiB / 64 MiB / 37 MiB ( 0 MiB / 256 MiB / 4 MiB / 0 MiB / 0 MiB )
8 min passed...42624 MiB / 46976 MiB / 42629 MiB / 64 MiB / 37 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...42624 MiB / 47008 MiB / 42629 MiB / 64 MiB / 37 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...42624 MiB / 47008 MiB / 42631 MiB / 64 MiB / 37 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
11 min passed...42624 MiB / 47040 MiB / 42637 MiB / 64 MiB / 37 MiB ( 0 MiB / 32 MiB / 6 MiB / 0 MiB / 0 MiB )
12 min passed...42624 MiB / 47232 MiB / 42637 MiB / 64 MiB / 37 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...42656 MiB / 47552 MiB / 42642 MiB / 64 MiB / 37 MiB ( 32 MiB / 320 MiB / 5 MiB / 0 MiB / 0 MiB )
14 min passed...42656 MiB / 47552 MiB / 42646 MiB / 64 MiB / 37 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
15 min passed...42656 MiB / 47552 MiB / 42648 MiB / 64 MiB / 37 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
16 min passed...42656 MiB / 47744 MiB / 42650 MiB / 7360 MiB / 7398 MiB ( 0 MiB / 192 MiB / 2 MiB / 7296 MiB / 7361 MiB )
17 min passed...42656 MiB / 48000 MiB / 42650 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 256 MiB / 0 MiB / 3872 MiB / 3806 MiB )
18 min passed...42656 MiB / 48384 MiB / 42650 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...42656 MiB / 48384 MiB / 42650 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...42656 MiB / 48384 MiB / 42652 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
21 min passed...42656 MiB / 48544 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
22 min passed...42656 MiB / 48768 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...42656 MiB / 49504 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 736 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...42656 MiB / 49536 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...42656 MiB / 49536 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...42656 MiB / 49632 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...42656 MiB / 49920 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...42656 MiB / 50720 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 800 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...42656 MiB / 50720 MiB / 42656 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...42656 MiB / 50720 MiB / 42658 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
31 min passed...42656 MiB / 50784 MiB / 42679 MiB / 11232 MiB / 11204 MiB ( 0 MiB / 64 MiB / 21 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 4: async 4K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][90.9%][w=66.1MiB/s][w=16.9k IOPS][eta 00m:01s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=637: Thu Aug 12 01:50:08 2021
  write: IOPS=24.1k, BW=94.3MiB/s (98.9MB/s)(1024MiB/10858msec); 0 zone resets
    slat (usec): min=6, max=11268, avg=24.38, stdev=32.57
    clat (usec): min=312, max=39629, avg=2618.24, stdev=1927.47
     lat (usec): min=335, max=39685, avg=2644.10, stdev=1930.11
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1598], 10.00th=[ 1680], 20.00th=[ 1844],
     | 30.00th=[ 1975], 40.00th=[ 2040], 50.00th=[ 2089], 60.00th=[ 2147],
     | 70.00th=[ 2245], 80.00th=[ 2638], 90.00th=[ 3326], 95.00th=[ 7046],
     | 99.00th=[10552], 99.50th=[11731], 99.90th=[22414], 99.95th=[27395],
     | 99.99th=[33817]
   bw (  KiB/s): min=31600, max=128856, per=98.46%, avg=95080.38, stdev=33395.45, samples=21
   iops        : min= 7900, max=32214, avg=23770.10, stdev=8348.86, samples=21
  lat (usec)   : 500=0.01%, 750=0.02%, 1000=0.06%
  lat (msec)   : 2=34.31%, 4=57.46%, 10=6.72%, 20=1.31%, 50=0.11%
  cpu          : usr=16.10%, sys=48.23%, ctx=194985, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=94.3MiB/s (98.9MB/s), 94.3MiB/s-94.3MiB/s (98.9MB/s-98.9MB/s), io=1024MiB (1074MB), run=10858-10858msec

Disk stats (read/write):
  sda: ios=0/254479, merge=0/1242, ticks=0/254480, in_queue=254444, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 1029 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=283MiB/s][r=72.5k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=892: Thu Aug 12 02:05:12 2021
  read: IOPS=72.6k, BW=284MiB/s (298MB/s)(1024MiB/3609msec)
    slat (usec): min=3, max=115, avg= 7.10, stdev= 3.60
    clat (usec): min=204, max=4593, avg=869.08, stdev=76.24
     lat (usec): min=216, max=4598, avg=877.34, stdev=76.64
    clat percentiles (usec):
     |  1.00th=[  766],  5.00th=[  791], 10.00th=[  807], 20.00th=[  824],
     | 30.00th=[  840], 40.00th=[  848], 50.00th=[  865], 60.00th=[  873],
     | 70.00th=[  889], 80.00th=[  906], 90.00th=[  922], 95.00th=[  947],
     | 99.00th=[ 1074], 99.50th=[ 1172], 99.90th=[ 1778], 99.95th=[ 2114],
     | 99.99th=[ 2999]
   bw (  KiB/s): min=288944, max=295416, per=100.00%, avg=291068.57, stdev=2265.42, samples=7
   iops        : min=72236, max=73854, avg=72766.86, stdev=566.59, samples=7
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.39%, 1000=97.67%
  lat (msec)   : 2=1.87%, 4=0.06%, 10=0.01%
  cpu          : usr=43.35%, sys=56.57%, ctx=21, majf=0, minf=73
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=1024MiB (1074MB), run=3609-3609msec

Disk stats (read/write):
  sda: ios=247099/2, merge=0/0, ticks=55903/1, in_queue=55972, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1817 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3712 MiB / 4000 MiB / 3756 MiB / 2976 MiB / 3034 MiB ( 3712 MiB / 4000 MiB / 3756 MiB / 2976 MiB / 3034 MiB )
2 min passed...3712 MiB / 4160 MiB / 3760 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
3 min passed...3712 MiB / 4736 MiB / 3760 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...3712 MiB / 4992 MiB / 3762 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
5 min passed...3712 MiB / 4992 MiB / 3764 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
6 min passed...3776 MiB / 4992 MiB / 3773 MiB / 2976 MiB / 3034 MiB ( 64 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
7 min passed...3776 MiB / 5024 MiB / 3773 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...3776 MiB / 5504 MiB / 3775 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 480 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...3776 MiB / 6048 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 544 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...3776 MiB / 6048 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...3776 MiB / 6048 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...3776 MiB / 6112 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...3776 MiB / 6432 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...3776 MiB / 6688 MiB / 3777 MiB / 2976 MiB / 3034 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...3776 MiB / 6720 MiB / 3779 MiB / 2976 MiB / 3035 MiB ( 0 MiB / 32 MiB / 2 MiB / 0 MiB / 1 MiB )
16 min passed...3808 MiB / 6752 MiB / 3789 MiB / 5056 MiB / 5078 MiB ( 32 MiB / 32 MiB / 10 MiB / 2080 MiB / 2043 MiB )
17 min passed...3808 MiB / 6880 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 128 MiB / 3 MiB / 0 MiB / 0 MiB )
18 min passed...3808 MiB / 7552 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 672 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...3808 MiB / 7936 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...3808 MiB / 8032 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...3808 MiB / 8032 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...3808 MiB / 8256 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...3808 MiB / 8832 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...3808 MiB / 9120 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...3808 MiB / 9216 MiB / 3792 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...3840 MiB / 9216 MiB / 3800 MiB / 5056 MiB / 5078 MiB ( 32 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
27 min passed...3840 MiB / 9440 MiB / 3800 MiB / 5056 MiB / 5078 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...3840 MiB / 9952 MiB / 3809 MiB / 5056 MiB / 5079 MiB ( 0 MiB / 512 MiB / 9 MiB / 0 MiB / 1 MiB )
29 min passed...3840 MiB / 10400 MiB / 3809 MiB / 5056 MiB / 5079 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...3840 MiB / 10528 MiB / 3809 MiB / 5056 MiB / 5079 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Dunuin said:
4k sync/async write/read on 2 disk ZFS mirror using ashift=12 & volblocksize=4K comparing ext4 with journaling vs ext4 without journaling vs xfs with journaling:
ext4 journal syncext4 no journal syncxfs journal sync
ext4 journal async		ext4 no journal async
xfs journal async
Write Performance:1,87 MiB/s4,26 MiB/s2,4 MiB/s125 MiB/s133 MiB/s129 MiB/s
Read Performance:9,25 MiB/s9,67 MiB/s9,67 MiB/s295 MiB/s282 MiB/s295 MiB/s
W.A. fio -> guest:5,11 x2,24 x2,86 x1,01 x1 x1 x
W.A. guest -> host:10,02 x9,67 x12,34 x3,67 x2,17 x4,20 x
W.A. host -> NAND:1,11 x1,15 x1,09 x1,46 x1,46 x1,41 x
W.A. total:56,84 x25,19 x38,38 x5,44 x3,16 x6,53 x
Click to expand...

Hi @Dunuin ,


Thx. again for your work, and time spent for us!
Nice .... so the winner is ext4(no journal) if you want W.A. to be at minimum!


Good luck / Bafta!
 
guletz said:
Hi @Dunuin ,


Thx. again for your work, and time spent for us!
Nice .... so the winner is ext4(no journal) if you want W.A. to be at minimum!


Good luck / Bafta!
Click to expand...
Yes, by far the best. Now its just the question if it is useful not to use a journal for the guests file system. If the unterlaying journal of ZFS isn't enough to keep the guests file systems integrity while power outages and crashes I would still use XFS with journaling even if the write amplification is higher and performance is lower. Atleast for all VMs I care about. I think 90% of all my writes are created by graylog and zabbix all they do is storing logs and metrics to elasticsearch, mongodb and MySQL DBs. Here it would be totally fine if I just restore a backup on problems because I don't care loosing some days of logs and metrics.
 
Round 2 - Run 4 - Test 5: async 16K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=648: Thu Aug 12 13:28:41 2021
  write: IOPS=29.2k, BW=456MiB/s (478MB/s)(1024MiB/2248msec); 0 zone resets
    slat (usec): min=7, max=4078, avg=19.25, stdev=23.22
    clat (usec): min=139, max=15626, avg=2161.60, stdev=677.72
     lat (usec): min=149, max=15649, avg=2182.20, stdev=680.22
    clat percentiles (usec):
     |  1.00th=[ 1713],  5.00th=[ 1827], 10.00th=[ 1860], 20.00th=[ 1909],
     | 30.00th=[ 1942], 40.00th=[ 1991], 50.00th=[ 2008], 60.00th=[ 2057],
     | 70.00th=[ 2114], 80.00th=[ 2212], 90.00th=[ 2573], 95.00th=[ 2999],
     | 99.00th=[ 4146], 99.50th=[ 5997], 99.90th=[10552], 99.95th=[15008],
     | 99.99th=[15401]
   bw (  KiB/s): min=381312, max=515328, per=100.00%, avg=474056.00, stdev=62335.94, samples=4
   iops        : min=23832, max=32208, avg=29628.50, stdev=3896.00, samples=4
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=45.30%, 4=53.51%, 10=0.98%, 20=0.17%
  cpu          : usr=28.57%, sys=49.04%, ctx=66112, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s), io=1024MiB (1074MB), run=2248-2248msec

Disk stats (read/write):
  sda: ios=0/62630, merge=0/0, ticks=0/8135, in_queue=8932, util=99.96%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=864: Thu Aug 12 13:43:43 2021
  read: IOPS=56.6k, BW=884MiB/s (927MB/s)(1024MiB/1158msec)
    slat (usec): min=4, max=4114, avg= 9.47, stdev=16.89
    clat (usec): min=413, max=9242, avg=1115.06, stdev=340.91
     lat (usec): min=431, max=9250, avg=1125.95, stdev=341.48
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  766], 10.00th=[  840], 20.00th=[  922],
     | 30.00th=[  979], 40.00th=[ 1037], 50.00th=[ 1090], 60.00th=[ 1139],
     | 70.00th=[ 1172], 80.00th=[ 1221], 90.00th=[ 1336], 95.00th=[ 1549],
     | 99.00th=[ 2180], 99.50th=[ 2704], 99.90th=[ 5014], 99.95th=[ 5145],
     | 99.99th=[ 5342]
   bw (  KiB/s): min=893952, max=931232, per=100.00%, avg=912592.00, stdev=26360.94, samples=2
   iops        : min=55872, max=58202, avg=57037.00, stdev=1647.56, samples=2
  lat (usec)   : 500=0.04%, 750=4.19%, 1000=29.40%
  lat (msec)   : 2=64.90%, 4=1.04%, 10=0.42%
  cpu          : usr=39.67%, sys=57.82%, ctx=518, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=884MiB/s (927MB/s), 884MiB/s-884MiB/s (927MB/s-927MB/s), io=1024MiB (1074MB), run=1158-1158msec

Disk stats (read/write):
  sda: ios=64954/0, merge=0/0, ticks=33763/0, in_queue=34348, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2624 MiB / 2848 MiB / 2625 MiB / 0 MiB / 16 MiB ( 2624 MiB / 2848 MiB / 2625 MiB / 0 MiB / 16 MiB )
2 min passed...2624 MiB / 3104 MiB / 2625 MiB / 0 MiB / 16 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2624 MiB / 3424 MiB / 2625 MiB / 0 MiB / 16 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2624 MiB / 3808 MiB / 2625 MiB / 0 MiB / 16 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2624 MiB / 4064 MiB / 2625 MiB / 0 MiB / 16 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2624 MiB / 4096 MiB / 2628 MiB / 0 MiB / 16 MiB ( 0 MiB / 32 MiB / 3 MiB / 0 MiB / 0 MiB )
7 min passed...2656 MiB / 4320 MiB / 2633 MiB / 0 MiB / 16 MiB ( 32 MiB / 224 MiB / 5 MiB / 0 MiB / 0 MiB )
8 min passed...2656 MiB / 4608 MiB / 2635 MiB / 0 MiB / 16 MiB ( 0 MiB / 288 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...2656 MiB / 5024 MiB / 2635 MiB / 0 MiB / 16 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2656 MiB / 5280 MiB / 2635 MiB / 0 MiB / 16 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2656 MiB / 5280 MiB / 2635 MiB / 0 MiB / 16 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2656 MiB / 5440 MiB / 2639 MiB / 0 MiB / 16 MiB ( 0 MiB / 160 MiB / 4 MiB / 0 MiB / 0 MiB )
13 min passed...2656 MiB / 5664 MiB / 2641 MiB / 0 MiB / 16 MiB ( 0 MiB / 224 MiB / 2 MiB / 0 MiB / 0 MiB )
14 min passed...2656 MiB / 6240 MiB / 2643 MiB / 0 MiB / 16 MiB ( 0 MiB / 576 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...2688 MiB / 6560 MiB / 2645 MiB / 384 MiB / 455 MiB ( 32 MiB / 320 MiB / 2 MiB / 384 MiB / 439 MiB )
16 min passed...2688 MiB / 6560 MiB / 2645 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 640 MiB / 595 MiB )
17 min passed...2688 MiB / 6624 MiB / 2650 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 64 MiB / 5 MiB / 0 MiB / 0 MiB )
18 min passed...2688 MiB / 6880 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
19 min passed...2688 MiB / 7456 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 576 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2688 MiB / 7616 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2688 MiB / 7616 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2688 MiB / 7616 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2688 MiB / 7904 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2688 MiB / 8384 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2688 MiB / 8800 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2688 MiB / 8800 MiB / 2652 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2688 MiB / 8800 MiB / 2660 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
28 min passed...2688 MiB / 8992 MiB / 2660 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2688 MiB / 9344 MiB / 2660 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2688 MiB / 9920 MiB / 2692 MiB / 1024 MiB / 1050 MiB ( 0 MiB / 576 MiB / 32 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 6: async 4M sequential read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=639: Thu Aug 12 14:08:50 2021
  write: IOPS=94, BW=379MiB/s (397MB/s)(10.0GiB/27045msec); 0 zone resets
    slat (usec): min=93, max=59910, avg=279.10, stdev=1758.66
    clat (msec): min=51, max=5118, avg=673.68, stdev=1308.87
     lat (msec): min=51, max=5118, avg=673.96, stdev=1308.87
    clat percentiles (msec):
     |  1.00th=[   63],  5.00th=[   72], 10.00th=[   77], 20.00th=[   79],
     | 30.00th=[   81], 40.00th=[   83], 50.00th=[   88], 60.00th=[  103],
     | 70.00th=[  142], 80.00th=[  388], 90.00th=[ 3239], 95.00th=[ 4463],
     | 99.00th=[ 4866], 99.50th=[ 5000], 99.90th=[ 5134], 99.95th=[ 5134],
     | 99.99th=[ 5134]
   bw (  KiB/s): min= 8175, max=3284992, per=100.00%, avg=1022404.05, stdev=1008188.99, samples=20
   iops        : min=    1, max=  802, avg=249.55, stdev=246.18, samples=20
  lat (msec)   : 100=59.10%, 250=14.22%, 500=7.77%, 750=0.78%, 1000=1.05%
  cpu          : usr=10.43%, sys=2.37%, ctx=1447, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=379MiB/s (397MB/s), 379MiB/s-379MiB/s (397MB/s-397MB/s), io=10.0GiB (10.7GB), run=27045-27045msec

Disk stats (read/write):
  sda: ios=0/10159, merge=0/104, ticks=0/4614385, in_queue=822728, util=24.39%
0 min passed...Guest Writes/Reads (Changes): 10240 MiB / 19 MiB ( 10240 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1265MiB/s][r=316 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=919: Thu Aug 12 14:23:59 2021
  read: IOPS=348, BW=1393MiB/s (1461MB/s)(10.0GiB/7349msec)
    slat (usec): min=86, max=6397, avg=215.77, stdev=274.07
    clat (msec): min=20, max=297, avg=182.64, stdev=73.42
     lat (msec): min=26, max=297, avg=182.85, stdev=73.38
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   59], 10.00th=[   70], 20.00th=[   88],
     | 30.00th=[  132], 40.00th=[  190], 50.00th=[  213], 60.00th=[  230],
     | 70.00th=[  243], 80.00th=[  247], 90.00th=[  255], 95.00th=[  262],
     | 99.00th=[  288], 99.50th=[  292], 99.90th=[  292], 99.95th=[  292],
     | 99.99th=[  296]
   bw (  MiB/s): min= 1056, max= 1528, per=99.23%, avg=1382.66, stdev=111.49, samples=14
   iops        : min=  264, max=  382, avg=345.64, stdev=27.87, samples=14
  lat (msec)   : 50=2.46%, 100=21.56%, 250=61.41%, 500=14.57%
  cpu          : usr=0.42%, sys=7.83%, ctx=1787, majf=0, minf=649
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1393MiB/s (1461MB/s), 1393MiB/s-1393MiB/s (1461MB/s-1461MB/s), io=10.0GiB (10.7GB), run=7349-7349msec

Disk stats (read/write):
  sda: ios=9959/0, merge=0/0, ticks=1337024/0, in_queue=1367948, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10241 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Script ran 1836 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...30336 MiB / 32576 MiB / 30295 MiB / 224 MiB / 184 MiB ( 30336 MiB / 32576 MiB / 30295 MiB / 224 MiB / 184 MiB )
2 min passed...30336 MiB / 32608 MiB / 30299 MiB / 224 MiB / 184 MiB ( 0 MiB / 32 MiB / 4 MiB / 0 MiB / 0 MiB )
3 min passed...30336 MiB / 32608 MiB / 30299 MiB / 224 MiB / 184 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...30336 MiB / 32864 MiB / 30304 MiB / 224 MiB / 184 MiB ( 0 MiB / 256 MiB / 5 MiB / 0 MiB / 0 MiB )
5 min passed...30336 MiB / 32960 MiB / 30304 MiB / 224 MiB / 184 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...30336 MiB / 33504 MiB / 30304 MiB / 224 MiB / 184 MiB ( 0 MiB / 544 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...30368 MiB / 33568 MiB / 30306 MiB / 224 MiB / 184 MiB ( 32 MiB / 64 MiB / 2 MiB / 0 MiB / 0 MiB )
8 min passed...30400 MiB / 33568 MiB / 30314 MiB / 224 MiB / 184 MiB ( 32 MiB / 0 MiB / 8 MiB / 0 MiB / 0 MiB )
9 min passed...30400 MiB / 33856 MiB / 30318 MiB / 224 MiB / 185 MiB ( 0 MiB / 288 MiB / 4 MiB / 0 MiB / 1 MiB )
10 min passed...30400 MiB / 34016 MiB / 30323 MiB / 224 MiB / 185 MiB ( 0 MiB / 160 MiB / 5 MiB / 0 MiB / 0 MiB )
11 min passed...30400 MiB / 34496 MiB / 30323 MiB / 224 MiB / 185 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...30400 MiB / 34560 MiB / 30323 MiB / 224 MiB / 185 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...30400 MiB / 34560 MiB / 30323 MiB / 224 MiB / 185 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...30400 MiB / 34880 MiB / 30323 MiB / 224 MiB / 185 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...30400 MiB / 35168 MiB / 30327 MiB / 224 MiB / 185 MiB ( 0 MiB / 288 MiB / 4 MiB / 0 MiB / 0 MiB )
16 min passed...30400 MiB / 35520 MiB / 30327 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 352 MiB / 0 MiB / 10592 MiB / 10645 MiB )
17 min passed...30400 MiB / 35808 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 288 MiB / 8 MiB / 0 MiB / 0 MiB )
18 min passed...30400 MiB / 35808 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...30400 MiB / 36032 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...30400 MiB / 36256 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...30400 MiB / 36608 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 352 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...30400 MiB / 36832 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...30400 MiB / 36864 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...30400 MiB / 36960 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...30400 MiB / 37152 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...30400 MiB / 37376 MiB / 30335 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...30400 MiB / 37760 MiB / 30340 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 384 MiB / 5 MiB / 0 MiB / 0 MiB )
28 min passed...30400 MiB / 37760 MiB / 30340 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...30400 MiB / 38048 MiB / 30340 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...30400 MiB / 38336 MiB / 30340 MiB / 10816 MiB / 10830 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
 
Dunuin said:
Atleast for all VMs I care about. I think 90% of all my writes are created by graylog and zabbix all they do is storing logs and metrics to elasticsearch, mongodb and MySQL DBs. Here it would be totally fine if I just restore a backup on problems because I don't care loosing some days of logs and metrics.
Click to expand...

Hi,

1.
I tested in a VM, with ext4(with no-journal), and I have copy some file/folders. Then I have start to copy a big file(ISO), and after few minutes I push reset button of the VM! After reboot, all was OK, no errors(checked with fsck).

Aug 12 14:37:21 xxxx kernel: EXT4-fs (sdb): mounted filesystem without journal. Opts: (null)

2.

On the same VM I have a mysql server(for librenms with data for around 20 devices, more than a year),
remove journal, reset....
At boot, fsck start to check the /, find some orphans inodes, an then all was OK, even if I start to broswer the devices graph!

I will try to repeat this tests.


Good luck / Bafta !
 
  • Like
Reactions: Dunuin
Round 2 - Run 4 - Test 7: async 16K random read/write but 50% compressible

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=655: Thu Aug 12 14:50:02 2021
  write: IOPS=31.3k, BW=490MiB/s (514MB/s)(1024MiB/2091msec); 0 zone resets
    slat (usec): min=7, max=327, avg=20.32, stdev=10.98
    clat (usec): min=75, max=4116, avg=2009.54, stdev=322.59
     lat (usec): min=85, max=4133, avg=2031.21, stdev=326.51
    clat percentiles (usec):
     |  1.00th=[ 1663],  5.00th=[ 1729], 10.00th=[ 1778], 20.00th=[ 1827],
     | 30.00th=[ 1860], 40.00th=[ 1893], 50.00th=[ 1926], 60.00th=[ 1942],
     | 70.00th=[ 1991], 80.00th=[ 2057], 90.00th=[ 2442], 95.00th=[ 2868],
     | 99.00th=[ 3195], 99.50th=[ 3261], 99.90th=[ 3425], 99.95th=[ 3458],
     | 99.99th=[ 3556]
   bw (  KiB/s): min=433405, max=544384, per=99.24%, avg=497679.25, stdev=46451.69, samples=4
   iops        : min=27087, max=34024, avg=31104.75, stdev=2903.61, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=72.48%, 4=27.46%, 10=0.01%
  cpu          : usr=21.53%, sys=56.36%, ctx=68710, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=1024MiB (1074MB), run=2091-2091msec

Disk stats (read/write):
  sda: ios=0/58535, merge=0/0, ticks=0/3481, in_queue=4084, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=865: Thu Aug 12 15:05:04 2021
  read: IOPS=57.5k, BW=898MiB/s (942MB/s)(1024MiB/1140msec)
    slat (nsec): min=4693, max=96526, avg=10145.06, stdev=4317.57
    clat (usec): min=224, max=3295, avg=1096.69, stdev=132.89
     lat (usec): min=240, max=3309, avg=1108.14, stdev=134.23
    clat percentiles (usec):
     |  1.00th=[  848],  5.00th=[  898], 10.00th=[  922], 20.00th=[  971],
     | 30.00th=[ 1029], 40.00th=[ 1106], 50.00th=[ 1123], 60.00th=[ 1139],
     | 70.00th=[ 1156], 80.00th=[ 1172], 90.00th=[ 1205], 95.00th=[ 1237],
     | 99.00th=[ 1336], 99.50th=[ 1778], 99.90th=[ 2212], 99.95th=[ 2409],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=858080, max=980608, per=99.95%, avg=919344.00, stdev=86640.38, samples=2
   iops        : min=53630, max=61288, avg=57459.00, stdev=5415.02, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.03%, 1000=26.33%
  lat (msec)   : 2=73.45%, 4=0.18%
  cpu          : usr=40.30%, sys=59.61%, ctx=22, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=898MiB/s (942MB/s), 898MiB/s-898MiB/s (942MB/s-942MB/s), io=1024MiB (1074MB), run=1140-1140msec

Disk stats (read/write):
  sda: ios=51130/0, merge=0/0, ticks=12031/0, in_queue=11940, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2432 MiB / 2688 MiB / 2414 MiB / 64 MiB / 20 MiB ( 2432 MiB / 2688 MiB / 2414 MiB / 64 MiB / 20 MiB )
2 min passed...2432 MiB / 2944 MiB / 2417 MiB / 64 MiB / 20 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
3 min passed...2432 MiB / 3168 MiB / 2419 MiB / 64 MiB / 20 MiB ( 0 MiB / 224 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...2432 MiB / 3360 MiB / 2421 MiB / 64 MiB / 20 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
5 min passed...2432 MiB / 3648 MiB / 2423 MiB / 64 MiB / 20 MiB ( 0 MiB / 288 MiB / 2 MiB / 0 MiB / 0 MiB )
6 min passed...2432 MiB / 3936 MiB / 2430 MiB / 64 MiB / 21 MiB ( 0 MiB / 288 MiB / 7 MiB / 0 MiB / 1 MiB )
7 min passed...2432 MiB / 4000 MiB / 2430 MiB / 64 MiB / 21 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2432 MiB / 4192 MiB / 2432 MiB / 64 MiB / 21 MiB ( 0 MiB / 192 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...2432 MiB / 4576 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 384 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...2432 MiB / 4608 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2432 MiB / 4608 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2432 MiB / 4800 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2432 MiB / 4992 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2432 MiB / 5312 MiB / 2434 MiB / 64 MiB / 21 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2432 MiB / 5536 MiB / 2436 MiB / 512 MiB / 527 MiB ( 0 MiB / 224 MiB / 2 MiB / 448 MiB / 506 MiB )
16 min passed...2432 MiB / 5568 MiB / 2446 MiB / 512 MiB / 528 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 1 MiB )
17 min passed...2432 MiB / 5824 MiB / 2449 MiB / 512 MiB / 528 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
18 min passed...2432 MiB / 6144 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 320 MiB / 1 MiB / 0 MiB / 0 MiB )
19 min passed...2432 MiB / 6400 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2432 MiB / 6592 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2432 MiB / 6720 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2432 MiB / 6848 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2432 MiB / 7328 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2432 MiB / 7648 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2432 MiB / 7712 MiB / 2450 MiB / 512 MiB / 528 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2432 MiB / 7936 MiB / 2456 MiB / 512 MiB / 528 MiB ( 0 MiB / 224 MiB / 6 MiB / 0 MiB / 0 MiB )
27 min passed...2432 MiB / 8032 MiB / 2456 MiB / 512 MiB / 528 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2432 MiB / 8416 MiB / 2462 MiB / 512 MiB / 528 MiB ( 0 MiB / 384 MiB / 6 MiB / 0 MiB / 0 MiB )
29 min passed...2432 MiB / 8608 MiB / 2462 MiB / 512 MiB / 528 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2432 MiB / 8704 MiB / 2495 MiB / 512 MiB / 528 MiB ( 0 MiB / 96 MiB / 33 MiB / 0 MiB / 0 MiB )
 
Hi,

I do not know why do you use VM(live migration?), but I can tell you that if you use CTs(so no underlying FS like ext4,xfs), and inside of CT you use mysql/percona, the corruption on the mysql/whatever is 0%.(And this is an ERP sistem, with around 150 GB in /var/lib/mysql, as SSDs show). At least we have 3-4 power problems/year(when the server have a reset, even with UPS). No problem at all.

2015-09-20.10:55:37 zfs create NODE2ssd/DB-mysql -o recordsize=16k

... one of the Intel 730 and DC S35x0/3610/3700 Series SSDs(> 6 years):

5 Reallocated_Sector_Ct 0x0032 099 099 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 52922
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 428
170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
174 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 424
175 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail Always - 4150 (300 298)
183 SATA_Downshift_Count 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
190 Temperature_Case 0x0022 076 066 000 Old_age Always - 24 (Min/Max 17/34)
192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 424
194 Temperature_Internal 0x0022 100 100 000 Old_age Always - 24
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 2193973
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 16506
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 88
228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 3175169
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 084 084 000 Old_age Always - 0
234 Thermal_Throttle 0x0032 100 100 000 Old_age Always - 0/0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 2193973
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 16760252

Good luck / Bafta!
 
I think I got an idea why I can't see any performance differences between a mirror, a striped mirror and raidz1 no matter if I use sync/async writes or multiple jobs in parallel...

If I understand it right striping won't help you at all for a single thread that is doing a write. Latency will always be the same and striping won't make your writes faster. But striping will balance multiple writes across all stripes so the throughput will be increased. But to make use of that throughput you need to do writes in parallel. My benchmarks only write to one virtual disk and with enabled "io thead" so each virtio SCSI only got 1 thread. So no matter how many parallel writes I do inside the guest, the host will only receive unparallelized writes because the virtio SCSI in between the guest and host is single threaded? Do I really need to run multiple Test VMs at the same time or do the writes directly on the host to see a performance difference?

Maybe @oguz or someone else of the staff can explain if a single virtio zvol can benefit from a striped mirror or if you really need to use multiple zvols in parallel to make use of the striping?

Lets for example look at the 16K random async writes:
write throughput: fio to guests virtual disksthroughput (reads+writes): host to SSDs
5 disk raidz1 (32K volblocksize)380 MiB/s914 MiB/s
8 disk striped mirror (16K volblocksize)441 MiB/s1144 MiB/s
2 disk mirror (4K volblocksize)450 MiB/s1898 MiB/s
So the bandwith between host and SSDs isn't the bottleneck because the 2 disk mirror is writing/reading way more data on the host compared to the striped mirror or raidz1. These are quite small random writes so in theory the 5 disk raidz1 shouldn't be much faster than the 2 disk mirror. But the 8 disk striped mirror should be 4 times faster than the 2 disk mirror. But both got the same speed. So it looks to me that striping has no effect if you just run a single VM doing the tests.
Its also "primarycache=metadata" for ZFS, "cache mode=none" for virtio and "--direct=1" for fio so I shouldn't benchmark the RAM when doing async writes.

Edit:
This was the command to test the throughput:
fio --directory=/mnt/test/ --name=async_rand_16K --rw=randwrite --bs=16K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G

Edit:
Here some updated comparison diagrams...
async_perf_comp2.png
sync_perf_comp2.pngwa_comp.png
 
Last edited:
  • Like
Reactions: guletz
guletz said:
Hi,

I do not know why do you use VM(live migration?), but I can tell you that if you use CTs(so no underlying FS like ext4,xfs), and inside of CT you use mysql/percona, the corruption on the mysql/whatever is 0%.(And this is an ERP sistem, with around 150 GB in /var/lib/mysql, as SSDs show). At least we have 3-4 power problems/year(when the server have a reset, even with UPS). No problem at all.

2015-09-20.10:55:37 zfs create NODE2ssd/DB-mysql -o recordsize=16k

... one of the Intel 730 and DC S35x0/3610/3700 Series SSDs(> 6 years):
Click to expand...
For everything that is important or is accessible from the internet I prefer using VMs because of the better isolation.
I already moved my graylog from VM to LXC but wasn't able to see a big difference in write amplfication. I also wanted to move the zabbix from VM to LXC but that would be alot of work so I never did that.

I also want to run the fio test scripts directly on the host writing to a dataset and zvol and compare that to running the same scripts inside the VM.
But that is alot of work so I think I will only do that for one pool configuration.
 
Round 2 - Run 4 - Test 8: sync 32K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=9.88MiB/s][w=316 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=670: Thu Aug 12 15:43:42 2021
  write: IOPS=316, BW=9.89MiB/s (10.4MB/s)(1024MiB/103510msec); 0 zone resets
    clat (usec): min=1582, max=42326, avg=3131.60, stdev=972.91
     lat (usec): min=1584, max=42328, avg=3134.01, stdev=974.31
    clat percentiles (usec):
     |  1.00th=[ 2311],  5.00th=[ 2540], 10.00th=[ 2606], 20.00th=[ 2737],
     | 30.00th=[ 2835], 40.00th=[ 2933], 50.00th=[ 2999], 60.00th=[ 3097],
     | 70.00th=[ 3163], 80.00th=[ 3294], 90.00th=[ 3458], 95.00th=[ 3982],
     | 99.00th=[ 5800], 99.50th=[ 8848], 99.90th=[15008], 99.95th=[18482],
     | 99.99th=[24773]
   bw (  KiB/s): min= 7744, max=12096, per=99.99%, avg=10128.72, stdev=815.62, samples=207
   iops        : min=  242, max=  378, avg=316.50, stdev=25.49, samples=207
  lat (msec)   : 2=0.13%, 4=94.94%, 10=4.57%, 20=0.31%, 50=0.04%
  cpu          : usr=1.20%, sys=4.34%, ctx=75123, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9.89MiB/s (10.4MB/s), 9.89MiB/s-9.89MiB/s (10.4MB/s-10.4MB/s), io=1024MiB (1074MB), run=103510-103510msec

Disk stats (read/write):
  sda: ios=0/98439, merge=0/66418, ticks=0/94658, in_queue=103192, util=99.72%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 1541 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=54.0MiB/s][r=1729 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=876: Thu Aug 12 15:59:00 2021
  read: IOPS=1804, BW=56.4MiB/s (59.1MB/s)(1024MiB/18163msec)
    clat (usec): min=369, max=6166, avg=543.87, stdev=175.67
     lat (usec): min=371, max=6167, avg=545.78, stdev=175.70
    clat percentiles (usec):
     |  1.00th=[  416],  5.00th=[  441], 10.00th=[  453], 20.00th=[  474],
     | 30.00th=[  490], 40.00th=[  506], 50.00th=[  523], 60.00th=[  537],
     | 70.00th=[  562], 80.00th=[  586], 90.00th=[  627], 95.00th=[  660],
     | 99.00th=[  783], 99.50th=[ 1778], 99.90th=[ 3130], 99.95th=[ 3261],
     | 99.99th=[ 3523]
   bw (  KiB/s): min=51456, max=62272, per=99.96%, avg=57705.58, stdev=2702.45, samples=36
   iops        : min= 1608, max= 1946, avg=1803.28, stdev=84.45, samples=36
  lat (usec)   : 500=35.63%, 750=63.22%, 1000=0.34%
  lat (msec)   : 2=0.36%, 4=0.45%, 10=0.01%
  cpu          : usr=3.24%, sys=9.62%, ctx=32773, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=56.4MiB/s (59.1MB/s), 56.4MiB/s-56.4MiB/s (59.1MB/s-59.1MB/s), io=1024MiB (1074MB), run=18163-18163msec

Disk stats (read/write):
  sda: ios=32739/0, merge=0/0, ticks=16523/0, in_queue=18248, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1541 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1923 seconds or 32 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...6656 MiB / 7040 MiB / 6689 MiB / 192 MiB / 168 MiB ( 6656 MiB / 7040 MiB / 6689 MiB / 192 MiB / 168 MiB )
2 min passed...11744 MiB / 12672 MiB / 11764 MiB / 256 MiB / 280 MiB ( 5088 MiB / 5632 MiB / 5075 MiB / 64 MiB / 112 MiB )
3 min passed...11744 MiB / 12768 MiB / 11766 MiB / 256 MiB / 280 MiB ( 0 MiB / 96 MiB / 2 MiB / 0 MiB / 0 MiB )
4 min passed...11744 MiB / 13056 MiB / 11773 MiB / 256 MiB / 280 MiB ( 0 MiB / 288 MiB / 7 MiB / 0 MiB / 0 MiB )
5 min passed...11744 MiB / 13056 MiB / 11773 MiB / 256 MiB / 280 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...11744 MiB / 13056 MiB / 11773 MiB / 256 MiB / 280 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11776 MiB / 13312 MiB / 11781 MiB / 256 MiB / 280 MiB ( 32 MiB / 256 MiB / 8 MiB / 0 MiB / 0 MiB )
8 min passed...11776 MiB / 13760 MiB / 11781 MiB / 256 MiB / 280 MiB ( 0 MiB / 448 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...11776 MiB / 14176 MiB / 11785 MiB / 256 MiB / 280 MiB ( 0 MiB / 416 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...11776 MiB / 14176 MiB / 11785 MiB / 256 MiB / 280 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11776 MiB / 14272 MiB / 11785 MiB / 256 MiB / 280 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...11776 MiB / 14368 MiB / 11785 MiB / 256 MiB / 280 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...11776 MiB / 14784 MiB / 11785 MiB / 256 MiB / 280 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...11776 MiB / 14880 MiB / 11792 MiB / 256 MiB / 280 MiB ( 0 MiB / 96 MiB / 7 MiB / 0 MiB / 0 MiB )
15 min passed...11776 MiB / 14880 MiB / 11792 MiB / 256 MiB / 280 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...11776 MiB / 15072 MiB / 11792 MiB / 256 MiB / 280 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...11776 MiB / 15200 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 128 MiB / 0 MiB / 1024 MiB / 1024 MiB )
18 min passed...11776 MiB / 15488 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...11776 MiB / 15488 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11776 MiB / 15520 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...11776 MiB / 15776 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...11776 MiB / 16160 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...11776 MiB / 16448 MiB / 11792 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11776 MiB / 16480 MiB / 11800 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 32 MiB / 8 MiB / 0 MiB / 0 MiB )
25 min passed...11776 MiB / 16576 MiB / 11800 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...11776 MiB / 16928 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 352 MiB / 4 MiB / 0 MiB / 0 MiB )
27 min passed...11776 MiB / 17312 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11776 MiB / 17376 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...11776 MiB / 17376 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...11776 MiB / 17568 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...11776 MiB / 17760 MiB / 11804 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...11776 MiB / 18208 MiB / 11843 MiB / 1280 MiB / 1304 MiB ( 0 MiB / 448 MiB / 39 MiB / 0 MiB / 0 MiB )
 
Round 2 - Run 4 - Test 9: sync 32K random read/write

Guest:
Code: 
root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=645: Thu Aug 12 16:40:37 2021
  write: IOPS=24.6k, BW=769MiB/s (807MB/s)(1024MiB/1331msec); 0 zone resets
    slat (usec): min=7, max=3077, avg=20.33, stdev=28.69
    clat (usec): min=158, max=18005, avg=2557.10, stdev=1208.68
     lat (usec): min=178, max=18027, avg=2578.74, stdev=1213.41
    clat percentiles (usec):
     |  1.00th=[ 1958],  5.00th=[ 2040], 10.00th=[ 2073], 20.00th=[ 2114],
     | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2245], 60.00th=[ 2278],
     | 70.00th=[ 2343], 80.00th=[ 2573], 90.00th=[ 3294], 95.00th=[ 3687],
     | 99.00th=[10159], 99.50th=[11469], 99.90th=[14615], 99.95th=[17433],
     | 99.99th=[17957]
   bw (  KiB/s): min=873472, max=891264, per=100.00%, avg=882368.00, stdev=12580.84, samples=2
   iops        : min=27296, max=27852, avg=27574.00, stdev=393.15, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=2.29%, 4=93.85%, 10=2.73%, 20=1.06%
  cpu          : usr=35.19%, sys=42.63%, ctx=32711, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=769MiB/s (807MB/s), 769MiB/s-769MiB/s (807MB/s-807MB/s), io=1024MiB (1074MB), run=1331-1331msec

Disk stats (read/write):
  sda: ios=0/29498, merge=0/0, ticks=0/4748, in_queue=5240, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=867: Thu Aug 12 16:55:39 2021
  read: IOPS=28.1k, BW=879MiB/s (922MB/s)(1024MiB/1165msec)
    slat (usec): min=5, max=951, avg=19.05, stdev=42.51
    clat (usec): min=712, max=10729, avg=2243.74, stdev=777.01
     lat (usec): min=732, max=10756, avg=2264.72, stdev=782.88
    clat percentiles (usec):
     |  1.00th=[  971],  5.00th=[ 1156], 10.00th=[ 1287], 20.00th=[ 1467],
     | 30.00th=[ 1647], 40.00th=[ 1876], 50.00th=[ 2311], 60.00th=[ 2638],
     | 70.00th=[ 2769], 80.00th=[ 2900], 90.00th=[ 3064], 95.00th=[ 3261],
     | 99.00th=[ 4424], 99.50th=[ 4948], 99.90th=[ 5866], 99.95th=[ 6063],
     | 99.99th=[ 7832]
   bw (  KiB/s): min=903104, max=925184, per=100.00%, avg=914144.00, stdev=15612.92, samples=2
   iops        : min=28222, max=28912, avg=28567.00, stdev=487.90, samples=2
  lat (usec)   : 750=0.02%, 1000=1.35%
  lat (msec)   : 2=42.25%, 4=54.74%, 10=1.64%, 20=0.01%
  cpu          : usr=29.90%, sys=60.57%, ctx=1238, majf=0, minf=520
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=879MiB/s (922MB/s), 879MiB/s-879MiB/s (922MB/s-922MB/s), io=1024MiB (1074MB), run=1165-1165msec

Disk stats (read/write):
  sda: ios=32165/0, merge=0/0, ticks=35188/0, in_queue=35256, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1804 seconds or 30 minutes.

Host:
Code: 
root@Hypervisor:~# bash /root/scripts/smartctl_8.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2592 MiB / 3104 MiB / 2575 MiB / 0 MiB / 17 MiB ( 2592 MiB / 3104 MiB / 2575 MiB / 0 MiB / 17 MiB )
2 min passed...2592 MiB / 3200 MiB / 2580 MiB / 0 MiB / 17 MiB ( 0 MiB / 96 MiB / 5 MiB / 0 MiB / 0 MiB )
3 min passed...2592 MiB / 3232 MiB / 2580 MiB / 0 MiB / 17 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2592 MiB / 3392 MiB / 2582 MiB / 0 MiB / 17 MiB ( 0 MiB / 160 MiB / 2 MiB / 0 MiB / 0 MiB )
5 min passed...2592 MiB / 3488 MiB / 2588 MiB / 0 MiB / 17 MiB ( 0 MiB / 96 MiB / 6 MiB / 0 MiB / 0 MiB )
6 min passed...2592 MiB / 4000 MiB / 2592 MiB / 0 MiB / 17 MiB ( 0 MiB / 512 MiB / 4 MiB / 0 MiB / 0 MiB )
7 min passed...2592 MiB / 4032 MiB / 2592 MiB / 0 MiB / 17 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2592 MiB / 4352 MiB / 2594 MiB / 0 MiB / 17 MiB ( 0 MiB / 320 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...2592 MiB / 4512 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 160 MiB / 2 MiB / 0 MiB / 0 MiB )
10 min passed...2592 MiB / 4672 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2592 MiB / 4960 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 288 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2592 MiB / 5088 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2592 MiB / 5216 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...2592 MiB / 5312 MiB / 2596 MiB / 0 MiB / 17 MiB ( 0 MiB / 96 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...2592 MiB / 5696 MiB / 2603 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 384 MiB / 7 MiB / 1056 MiB / 1057 MiB )
16 min passed...2592 MiB / 5760 MiB / 2609 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 64 MiB / 6 MiB / 0 MiB / 0 MiB )
17 min passed...2592 MiB / 5760 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
18 min passed...2592 MiB / 5952 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2592 MiB / 6016 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 64 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2592 MiB / 6432 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 416 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2592 MiB / 6656 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 224 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2592 MiB / 6848 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2592 MiB / 7104 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2592 MiB / 7296 MiB / 2612 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 192 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2592 MiB / 7584 MiB / 2618 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 288 MiB / 6 MiB / 0 MiB / 0 MiB )
26 min passed...2592 MiB / 7808 MiB / 2620 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 224 MiB / 2 MiB / 0 MiB / 0 MiB )
27 min passed...2592 MiB / 7936 MiB / 2620 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2592 MiB / 7968 MiB / 2620 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 32 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2592 MiB / 8352 MiB / 2620 MiB / 1056 MiB / 1074 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2624 MiB / 8384 MiB / 2658 MiB / 1056 MiB / 1074 MiB ( 32 MiB / 32 MiB / 38 MiB / 0 MiB / 0 MiB )
 
Here are some tests I will additionally run to see if striping will increase performance if I fio runs multiple instances in parallel:

Test X6: 4x async sequential read/write 32K
Code: 
fio --directory=/mnt/test/ --name=async_seq_32K --rw=write --bs=32K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=3G
fio --directory=/mnt/test/ --name=async_seq_32K --rw=read --bs=32K --direct=1 --sync=0 --numjobs=4 --ioengine=libaio --iodepth=1 --refill_buffers --size=3G

Test X7: 4x sync sequential read/write 32K
Code: 
fio --directory=/mnt/test/ --name=sync_seq_32K --rw=write --bs=32K --direct=1 --sync=1 --numjobs=4 --ioengine=psync --iodepth=1 --refill_buffers --size=512M
fio --directory=/mnt/test/ --name=sync_seq_32K --rw=read --bs=32K --direct=1 --sync=1 --numjobs=4 --ioengine=psync --iodepth=1 --refill_buffers --size=512M

They are small enough so I don't run into the chipset-cpu-bottleneck but big enough so fio writes no blocks that are smaller than the vollbocksize. They also write more data so the test should run longer, give more samples and give a better average. And they are sequential instead of random because it looked like only sequential IO is benefiting from striping. IOdepth is now 1 für async IO but numjobs is 4 instead of 1, so now theoretically even sync writes should profit from stripeing.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back