how to best benchmark SSDs?

[Dunuin]

Thanks. So its like I thought. With three mirrors I meant 3 mirrors made of 2 disks each striped together. So its your 3 stripe 2 mirror.
Will be interesting to see if a 3 stripe 2 mirror is faster with 8K or 16K volblocksize.

 

[ryba84]

4k random reads will be faster with 8k volblocksize because zfs reads only 12k from disks. With 16k volblocksize it must read 24k from disks. But this is when compression is off. You could try use 12k volblocksize with lz4 compression and format ext4 in guest with stripe-width=4 then test. If data will be compressible ZFS can write 16k guest block to 12k volblocksize. Test this scenario too.

 

[Dunuin]

4k random reads will be faster with 8k volblocksize because zfs reads only 12k from disks. With 16k volblocksize it must read 24k from disks. But this is when compression is off. You could try use 12k volblocksize with lz4 compression and format ext4 in guest with stripe-width=4 then test. If data will be compressible ZFS can write 16k guest block to 12k volblocksize. Test this scenario too.

Good idea, Thanks.

 

[Dunuin]

Any recommandations for fio tests for the next round?

I want atleast one benchmark that is doing 4K random sync writes to see what the worst case of performance would be. Like this:
start guest/host monitoring...fio --filename=/mnt/test/test.file --name=sync_randwrite_4K --rw=randwrite --bs=4K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G...wait 15 minutes...fio --filename=/mnt/test/test.file --name=sync_randread_4K --rw=randread --bs=4K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G
...wait 15 minutes...calculate write amplification if guest/host/ssd are showing no more write activity.

And I want one best case scenario. So something like a 4M sequential async read/write with a high io depth but without guest caching. So Something like this:
start guest/host monitoring...fio --filename=/mnt/test/test.file --name=async_seqwrite_4M --rw=write --bs=4M --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=32 --refill_buffers --size=10G...wait 15 minutes...fio --filename=/mnt/test/test.file --name=async_seqread_4M --rw=read --bs=4M --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=32 --refill_buffers --size=10G
...wait 15 minutes...calculate write amplification if guest/host/ssd are showing no more write activity.

Also "async 4K random write/read" and "4M sync sequential read/write" would be useful. So I would be able to see worst and best performance for async and sync writes. And this would allow to compare async vs sync writes with the same block sizes.

But I would also like to test some real world scenarios. For example:
1.) a fio test that is similar to a write heavy MySQL database would be interesting because all VMs using DBs (MariaDB, ElasticSearch MongoDB) got such a terrible write amplification. I would assume a MySQL DB would to 16K random async reads but whats about the writes? Are that 16K sync sequential writes?
2.) a fio test that will use well compressible writes would also be interesting to see but I'm not sure how to do that. By default the other fio tests should be badly compressible because should fill the write buffer with random data

Someone knows how such tests could look like?

 

[Dunuin]

Does someone know how fio handles the files and numjobs? I wan't able to find a clear explanation...

1.) If I run multiple jobs at the same time, lets say "--numjobs=4" and tell it to read/write to a file I define like "--file=/mnt/test/test.file" and set a size like "--size=10G" fio will report written/read IO is 40GiB. 40GiB written sounds fine but that would mean fio has written to the same file 4 times simultaniously? Isn't that a problem with async writes as that writes might be skipped because they would be overwritten anyway?
Or is it fine to just use a single file for multiple jobs running in parallel?

2.) I would like to run a write and read benchmark from the same batch script. So I first want to write random data to a 10GiB file to test the write performance, wait 15 minutes and then use the same file (that should be already filled with random data from the write test before) for the read benchmark. Is there any special argument I need to use to tell fio to use the same file without overwriting it? The problem is that I want to calculate the write amplification too to be able to compare different workloads later. Lets say the write benchmark will write 10GiB as sync 4K random writes.
If now the read benchmark will lay out a new 10GiB file to read from or just overwrite the 10GiB content of the old file that would end up as 20GiB of writes and maybe the read test will lay out the file using async 1M sequential write or something similar. And in that case only 50% of the writes would be what I really wanted to test for the wirte amplification and the write amplification would be wrong.

3.) I found arguments to generate compressible data. Most of my VMs got a compression factor of around 2. So I wanted to run one benchmark that will be compressible. I can set a ratio like 50%/50% where the first 50% of the file will be random data and the last 50% will be zeros. Should that be fine to test a ZFS pool with enabled compression and would that result in data that creates a compression factor of 2 on the zvol?

4.) I'm still not sure what arguments to use for the SQL like benchmark. 16K sync random writes + 16k async random reads?

 

[Dunuin]

Does someone know how fio handles the files and numjobs? I wan't able to find a clear explanation...

1.) If I run multiple jobs at the same time, lets say "--numjobs=4" and tell it to read/write to a file I define like "--file=/mnt/test/test.file" and set a size like "--size=10G" fio will report written/read IO is 40GiB. 40GiB written sounds fine but that would mean fio has written to the same file 4 times simultaniously? Isn't that a problem with async writes as that writes might be skipped because they would be overwritten anyway?
Or is it fine to just use a single file for multiple jobs running in parallel?

I now just changed it from "--filename=/mnt/test/test.file" to "--directory=/mnt/test/". In that case fio will use this as a directory and will decide on its own how many files to create and how to name them. So if I use "--numjobs=4" fio will create 4 files with the name of the job + "0.0", "1.0", "2.0" and "3.0" at the end.

2.) I would like to run a write and read benchmark from the same batch script. So I first want to write random data to a 10GiB file to test the write performance, wait 15 minutes and then use the same file (that should be already filled with random data from the write test before) for the read benchmark. Is there any special argument I need to use to tell fio to use the same file without overwriting it? The problem is that I want to calculate the write amplification too to be able to compare different workloads later. Lets say the write benchmark will write 10GiB as sync 4K random writes.
If now the read benchmark will lay out a new 10GiB file to read from or just overwrite the 10GiB content of the old file that would end up as 20GiB of writes and maybe the read test will lay out the file using async 1M sequential write or something similar. And in that case only 50% of the writes would be what I really wanted to test for the wirte amplification and the write amplification would be wrong.

I tested it and if I use the same job name for both tests with the same "--directory", same "--numjobs" and same "--size" for both then the read test will just use the files created by the write test as input without any additional writes.

3.) I found arguments to generate compressible data. Most of my VMs got a compression factor of around 2. So I wanted to run one benchmark that will be compressible. I can set a ratio like 50%/50% where the first 50% of the file will be random data and the last 50% will be zeros. Should that be fine to test a ZFS pool with enabled compression and would that result in data that creates a compression factor of 2 on the zvol?

Didn' test it but sounds like "--buffer_compress_percentage=50 --buffer_compress_chunk=0" should do the job. The "--buffer_compress_chunk=0" means that the chucks should match the "--bs" parameter. So "--buffer_compress_percentage=50 --buffer_compress_chunk=0" with a BS of 16K should result in buffer data that looks like "8K random data + 8K zeros + 8K random data + 8K zeros + ...".

4.) I'm still not sure what arguments to use for the SQL like benchmark. 16K sync random writes + 16k async random reads?

I did no special SQL benchmark but I will do 16K sync writes/reads and 16K async writes/reads. So I hope that will match what MySQL would do.

 

[Dunuin]

I will now start test round 2.

I will wait longer between tests and monitor all reads/writes on the guest, host and SSD all the time.
Intel isn't telling how much cache the S3710 got but I googled the part numbers of the ICs and I found out that there is a 4GB DDR3 Micron chip. So looks like the SSD got 4GB of cache. So if I got 5 of them there is a total of 20GB RAM caching my writes. I'm only writing 1GiB for random writes and 10GiB for sequential writes using fio so that possibly isn't creating the worst performance possible because it isn't enough to overload the cache. Atleast not if the write amplification isn't that high. But I don't want to run 7 tests each writing more then 20GB. With a write amplification of factor 20 that would write nearly 3TB per run and I don't want to waste dozens of TBs of TBW of my drives.

I got 9 benchmark scripts that look like this:

Test 1.) sync 4K random write/read

#!/usr/local/bin/bash

start=`date +%s`

TESTDEVICES="sda sdb"

guestReadStart=-1
guestWriteStart=-1
guestReadLast=0
guestWriteLast=0
guestReadNow=0
guestWriteNow=0
guestReadChange=0
guestWriteChange=0
quitted=0

function getGuestIo() {
    local guestReadTmp=$(iostat -m -H -g guest ${TESTDEVICES} | grep guest | grep -o -E '[0-9\.-]+' | tail -2 | head -1)
    if [ $guestReadStart -lt 0 ]; then
        guestReadStart=$guestReadTmp
    fi
    guestReadLast=$guestReadNow
    guestReadNow="$((guestReadTmp-guestReadStart))"
    guestReadChange="$((guestReadNow-guestReadLast))"
    
    local guestWriteTmp=$(iostat -m -H -g guest ${TESTDEVICES} | grep guest | grep -o -E '[0-9\.-]+' | tail -1)
    if [ $guestWriteStart -lt 0 ]; then
        guestWriteStart=$guestWriteTmp
    fi
    guestWriteLast=$guestWriteNow
    guestWriteNow="$((guestWriteTmp-guestWriteStart))"
    guestWriteChange="$((guestWriteNow-guestWriteLast))"
}

trap 'quitting' EXIT INT TERM
quitting(){
    if [ $quitted -eq 0 ]; then
        getGuestIo
        echo "Final...Guest Writes/Reads (Changes): ${guestWriteNow} MiB / ${guestReadNow} MiB ( ${guestWriteChange} MiB / ${guestReadChange} MiB )"
        rm /mnt/test/*
        fstrim -a
        end=`date +%s`
        runtimesec=$((end-start))
        runtimemin=$((runtimesec/60))
        echo "Script ran ${runtimesec} seconds or ${runtimemin} minutes."
        quitted=1
        exit 0
    fi
}

# init
getGuestIo

# sync randwrite 4K
fio --directory=/mnt/test/ --name=sync_rand_4K --rw=randwrite --bs=4K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G

waitmin=15
for ((i = 0 ; i <= waitmin ; i++))
do
    getGuestIo
    echo "${i} min passed...Guest Writes/Reads (Changes): ${guestWriteNow} MiB / ${guestReadNow} MiB ( ${guestWriteChange} MiB / ${guestReadChange} MiB )"
    if [ $i -ne $waitmin ]; then
        sleep 60
    fi
done

# sync randread 4K
fio --directory=/mnt/test/ --name=sync_rand_4K --rw=randread --bs=4K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G

waitmin=15
for ((i = 0 ; i <= waitmin ; i++))
do
    getGuestIo
    echo "${i} min passed...Guest Writes/Reads (Changes): ${guestWriteNow} MiB / ${guestReadNow} MiB ( ${guestWriteChange} MiB / ${guestReadChange} MiB )"
    if [ $i -ne $waitmin ]; then
        sleep 60
    fi
done

exit 0

All other benchmark scripts basically look the same except for the different two fio commands. So I will just show the changed fio commands.

Test 2: sync 16K random write/read

fio --directory=/mnt/test/ --name=sync_rand_16K --rw=randwrite --bs=16K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G
fio --directory=/mnt/test/ --name=sync_rand_16K --rw=randread --bs=16K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G

Test 3: sync 4M sequential write/read

fio --directory=/mnt/test/ --name=sync_seq_4M --rw=write --bs=4M --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=10G
fio --directory=/mnt/test/ --name=sync_seq_4M --rw=read --bs=4M --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=10G

Test 4: async 4K random write/read

fio --directory=/mnt/test/ --name=async_rand_4K --rw=randwrite --bs=4K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G
fio --directory=/mnt/test/ --name=async_rand_4K --rw=randread --bs=4K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G

Test 5: async 16K random write/read

fio --directory=/mnt/test/ --name=async_rand_16K --rw=randwrite --bs=16K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G
fio --directory=/mnt/test/ --name=async_rand_16K --rw=randread --bs=16K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G

Test 6: async 4M sequential write/read

fio --directory=/mnt/test/ --name=async_seq_4M --rw=write --bs=4M --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=10G
fio --directory=/mnt/test/ --name=async_seq_4M --rw=read --bs=4M --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=10G

Test7: async 16K random write/read but with 50% compressible data

fio --directory=/mnt/test/ --name=async_rand_16K_comp50 --rw=randwrite --bs=16K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G --buffer_compress_percentage=50 --buffer_compress_chunk=0
fio --directory=/mnt/test/ --name=async_rand_16K_comp50 --rw=randread --bs=16K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G

Test8: sync 32K random write/read

fio --directory=/mnt/test/ --name=sync_rand_32K --rw=randwrite --bs=32K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G
fio --directory=/mnt/test/ --name=sync_rand_32K --rw=randread --bs=32K --direct=1 --sync=1 --numjobs=1 --ioengine=psync --iodepth=1 --refill_buffers --size=1G

Test9: async 32K random write/read

fio --directory=/mnt/test/ --name=async_rand_32K --rw=randwrite --bs=32K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G
fio --directory=/mnt/test/ --name=async_rand_32K --rw=randread --bs=32K --direct=1 --sync=0 --numjobs=1 --ioengine=libaio --iodepth=64 --refill_buffers --size=1G

I will do several runs using the same VM but with different settings and pool configurations. For each run I will restore a fresh VM from the backup. Between each test of a run I will reboot the VM.

Run 1 will be my current setup (5 SSDs raidz1 with ashift=12 and volblocksize=32K).

After that I will do serveral runs trying out a mirror (raid1) of 2 SSDs, a striped mirror (raid10) of 4 SSDs and a striped mirror (raid10) of 6 SSDs. Maybe a striped mirror (raid10) of 8 SSDs too. I will also test serval combinations of configs like volblocksize, ashift and so on to see whats works best.

 

[Dunuin]

This is my script I run on the host to collect and log the reads/writes of the host and SSDs:

#!/bin/bash

nandwritestart=-1
nandwritenow=0
nandwritechange=0
nandwritelast=0
function getNandWrites {
    local nandwrites1=$(smartctl -a /dev/sdf | grep "243 NAND_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local nandwritesmib1=$((nandwrites1*32))
    local nandwrites2=$(smartctl -a /dev/sdg | grep "243 NAND_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local nandwritesmib2=$((nandwrites2*32))
    local nandwrites3=$(smartctl -a /dev/sdh | grep "243 NAND_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local nandwritesmib3=$((nandwrites3*32))
    local nandwrites4=$(smartctl -a /dev/sdj | grep "243 NAND_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local nandwritesmib4=$((nandwrites4*32))
    local nandwrites5=$(smartctl -a /dev/sde | grep "243 NAND_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local nandwritesmib5=$((nandwrites5*32))
    local nandwritesum=$(( nandwritesmib1 + nandwritesmib2 + nandwritesmib3 + nandwritesmib4 ))
    local nandwritesum=$(( nandwritesum * 125 ))
    local nandwritesum=$(( nandwritesum / 100 ))
    if [ $nandwritestart -lt 0 ]; then
        nandwritestart=$nandwritesum
    fi
    nandwritelast=$nandwritenow
    nandwritenow="$((nandwritesum-nandwritestart))"
    nandwritechange="$((nandwritenow-nandwritelast))"
}

hostwritestart=-1
hostwritenow=0
hostwritechange=0
hostwritelast=0
function getHostWrites {
    local hostwrites1=$(smartctl -a /dev/sdf | grep "241 Host_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostwritesmib1=$((hostwrites1*32))
    local hostwrites2=$(smartctl -a /dev/sdg | grep "241 Host_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostwritesmib2=$((hostwrites2*32))
    local hostwrites3=$(smartctl -a /dev/sdh | grep "241 Host_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostwritesmib3=$((hostwrites3*32))
    local hostwrites4=$(smartctl -a /dev/sdj | grep "241 Host_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostwritesmib4=$((hostwrites4*32))
    local hostwrites5=$(smartctl -a /dev/sde | grep "241 Host_Writes_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostwritesmib5=$((hostwrites5*32))
    local hostwritesum=$(( hostwritesmib1 + hostwritesmib2 + hostwritesmib3 + hostwritesmib4 + hostwritesmib5 ))
    if [ $hostwritestart -lt 0 ]; then
        hostwritestart=$hostwritesum
    fi
    hostwritelast=$hostwritenow
    hostwritenow="$((hostwritesum-hostwritestart))"
    hostwritechange="$((hostwritenow-hostwritelast))"
}

hostreadstart=-1
hostreadnow=0
hostreadchange=0
hostreadlast=0
function getHostReads {
    local hostreads1=$(smartctl -a /dev/sdf | grep "242 Host_Reads_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostreadsmib1=$((hostreads1*32))
    local hostreads2=$(smartctl -a /dev/sdg | grep "242 Host_Reads_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostreadsmib2=$((hostreads2*32))
    local hostreads3=$(smartctl -a /dev/sdh | grep "242 Host_Reads_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostreadsmib3=$((hostreads3*32))
    local hostreads4=$(smartctl -a /dev/sdj | grep "242 Host_Reads_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostreadsmib4=$((hostreads4*32))
    local hostreads5=$(smartctl -a /dev/sde | grep "242 Host_Reads_32MiB" | grep -o "[[:digit:]]\{1,99\}" | tail -1)
    local hostreadsmib5=$((hostreads5*32))
    local hostreadsum=$(( hostreadsmib1 + hostreadsmib2 + hostreadsmib3 + hostreadsmib4 + hostreadsmib5 ))
    if [ $hostreadstart -lt 0 ]; then
        hostreadstart=$hostreadsum
    fi
    hostreadlast=$hostreadnow
    hostreadnow="$((hostreadsum-hostreadstart))"
    hostreadchange="$((hostreadnow-hostreadlast))"
}

poolwritestart=-1
poolreadstart=-1
poolwritenow=0
poolreadnow=0
poolwritechange=0
poolreadchange=0
poolwritelast=0
poolreadlast=0
function getPoolWrites {
    local poolwrite=$(iostat -m -H -g VMpool sde sdf sdg sdh sdj | grep VMpool | grep -o -E '[0-9\.-]+' | tail -1)
    local poolread=$(iostat -m -H -g VMpool sde sdf sdg sdh sdj | grep VMpool | grep -o -E '[0-9\.-]+' | tail -2 | head -1)
    if [ $poolwritestart -lt 0 ]; then
        poolwritestart=$poolwrite
    fi
    if [ $poolreadstart -lt 0 ]; then
        poolreadstart=$poolread
    fi
    poolwritelast=$poolwritenow
    poolwritenow="$((poolwrite-poolwritestart))"
    poolwritechange="$((poolwritenow-poolwritelast))"
    poolreadlast=$poolreadnow
    poolreadnow="$((poolread-poolreadstart))"
    poolreadchange="$((poolreadnow-poolreadlast))"
}


getNandWrites
getHostWrites
getHostReads
getPoolWrites

echo "1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)"
echo "2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)"
echo "3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)"
echo "4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)"
echo "5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)"
echo "Number between brackets are changes since last minute. Same oder as above."

waitmin=60
for ((i = 0 ; i <= waitmin ; i++))
do
    getNandWrites
    getHostWrites
    getHostReads
    getPoolWrites
    echo "${i} min passed...${hostwritenow} MiB / ${nandwritenow} MiB / ${poolwritenow} MiB / ${hostreadnow} MiB / ${poolreadnow} MiB ( ${hostwritechange} MiB / ${nandwritechange} MiB / ${poolwritechange} MiB / ${hostreadchange} MiB / ${poolreadchange} MiB )"
    if [ $i -ne $waitmin ]; then
        sleep 60
    fi
done

I need to edit it each time I change the pool layout. This is for my 5 disk raidz1 pool. For NAND writes I needed to trick a bit because "/dev/sdj" is a S3700 instead of a S3710 and that model doesn't report NAND writes. So I only add up the values of the 4 S3710 drives and multiplied them by 1.25 so it get values that are similar to what 5 drives should write to the NAND.

 

[Dunuin]

One interesting observation I made is that I can watch the wear leveling working. Not sure how I should take that into account for calculation the write amplification...

This is the output of my host script running for 1 hour without any VMs, snapshots or backups running:

1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
2 min passed...0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...0 MiB / 880 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 560 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...0 MiB / 1320 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 440 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...0 MiB / 1480 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...0 MiB / 1480 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...0 MiB / 1560 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...0 MiB / 1920 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...0 MiB / 2480 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 560 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...0 MiB / 2880 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 400 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...0 MiB / 2960 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...0 MiB / 2960 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...0 MiB / 3280 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...0 MiB / 3760 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 480 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...0 MiB / 3800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...0 MiB / 3800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...0 MiB / 3800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...0 MiB / 3800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...0 MiB / 3960 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...0 MiB / 4560 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 600 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...0 MiB / 4960 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 400 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...0 MiB / 5000 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...0 MiB / 5000 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...0 MiB / 5320 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...0 MiB / 5560 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...0 MiB / 5800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...0 MiB / 6080 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...0 MiB / 6080 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...0 MiB / 6200 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 120 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...0 MiB / 6720 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 520 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...0 MiB / 7160 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 440 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...0 MiB / 7400 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...0 MiB / 7400 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...0 MiB / 7400 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...0 MiB / 7440 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...0 MiB / 7800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...0 MiB / 7840 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...0 MiB / 7840 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...0 MiB / 7840 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...0 MiB / 8040 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 200 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...0 MiB / 8640 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 600 MiB / 0 MiB / 0 MiB / 0 MiB )
42 min passed...0 MiB / 9040 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 400 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...0 MiB / 9320 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...0 MiB / 9440 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 120 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...0 MiB / 9480 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...0 MiB / 9880 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 400 MiB / 0 MiB / 0 MiB / 0 MiB )
47 min passed...0 MiB / 10160 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )
48 min passed...0 MiB / 10480 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
49 min passed...0 MiB / 10680 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 200 MiB / 0 MiB / 0 MiB / 0 MiB )
50 min passed...0 MiB / 10680 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
51 min passed...0 MiB / 10920 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
52 min passed...0 MiB / 11440 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 520 MiB / 0 MiB / 0 MiB / 0 MiB )
53 min passed...0 MiB / 11800 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 0 MiB / 0 MiB / 0 MiB )
54 min passed...0 MiB / 12080 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )
55 min passed...0 MiB / 12080 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
56 min passed...0 MiB / 12120 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
57 min passed...0 MiB / 12200 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 0 MiB / 0 MiB / 0 MiB )
58 min passed...0 MiB / 12520 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 0 MiB / 0 MiB / 0 MiB )
59 min passed...0 MiB / 12560 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )

So there was zero reads/writes at all from the host to the SSDs but the SSDs still wrote 12560 MiB to the NAND. So the SSDs will write 294 GiB per day just for wear leveling even if you don't write a single byte.

 

[Dunuin]

Here I monitored the reads/writes of the host and guest for 49 Minutes while only the (idleing) test VM was running:

Host:

1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...0 MiB / 320 MiB / 24 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 24 MiB / 0 MiB / 0 MiB )
2 min passed...32 MiB / 760 MiB / 50 MiB / 0 MiB / 0 MiB ( 32 MiB / 440 MiB / 26 MiB / 0 MiB / 0 MiB )
3 min passed...64 MiB / 800 MiB / 74 MiB / 0 MiB / 0 MiB ( 32 MiB / 40 MiB / 24 MiB / 0 MiB / 0 MiB )
4 min passed...64 MiB / 800 MiB / 98 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 24 MiB / 0 MiB / 0 MiB )
5 min passed...128 MiB / 840 MiB / 121 MiB / 0 MiB / 0 MiB ( 64 MiB / 40 MiB / 23 MiB / 0 MiB / 0 MiB )
6 min passed...128 MiB / 1040 MiB / 143 MiB / 0 MiB / 0 MiB ( 0 MiB / 200 MiB / 22 MiB / 0 MiB / 0 MiB )
7 min passed...160 MiB / 1400 MiB / 169 MiB / 0 MiB / 0 MiB ( 32 MiB / 360 MiB / 26 MiB / 0 MiB / 0 MiB )
8 min passed...160 MiB / 1840 MiB / 189 MiB / 0 MiB / 0 MiB ( 0 MiB / 440 MiB / 20 MiB / 0 MiB / 0 MiB )
9 min passed...192 MiB / 2160 MiB / 208 MiB / 0 MiB / 0 MiB ( 32 MiB / 320 MiB / 19 MiB / 0 MiB / 0 MiB )
10 min passed...192 MiB / 2200 MiB / 229 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 21 MiB / 0 MiB / 0 MiB )
11 min passed...224 MiB / 2200 MiB / 248 MiB / 0 MiB / 0 MiB ( 32 MiB / 0 MiB / 19 MiB / 0 MiB / 0 MiB )
12 min passed...288 MiB / 2240 MiB / 268 MiB / 0 MiB / 0 MiB ( 64 MiB / 40 MiB / 20 MiB / 0 MiB / 0 MiB )
13 min passed...288 MiB / 2520 MiB / 291 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 23 MiB / 0 MiB / 0 MiB )
14 min passed...288 MiB / 2800 MiB / 311 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 20 MiB / 0 MiB / 0 MiB )
15 min passed...320 MiB / 3040 MiB / 333 MiB / 0 MiB / 0 MiB ( 32 MiB / 240 MiB / 22 MiB / 0 MiB / 0 MiB )
16 min passed...320 MiB / 3080 MiB / 353 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 20 MiB / 0 MiB / 0 MiB )
17 min passed...352 MiB / 3320 MiB / 373 MiB / 0 MiB / 0 MiB ( 32 MiB / 240 MiB / 20 MiB / 0 MiB / 0 MiB )
18 min passed...384 MiB / 3720 MiB / 393 MiB / 0 MiB / 0 MiB ( 32 MiB / 400 MiB / 20 MiB / 0 MiB / 0 MiB )
19 min passed...384 MiB / 4200 MiB / 415 MiB / 0 MiB / 0 MiB ( 0 MiB / 480 MiB / 22 MiB / 0 MiB / 0 MiB )
20 min passed...448 MiB / 4560 MiB / 435 MiB / 0 MiB / 0 MiB ( 64 MiB / 360 MiB / 20 MiB / 0 MiB / 0 MiB )
21 min passed...448 MiB / 4640 MiB / 455 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 20 MiB / 0 MiB / 0 MiB )
22 min passed...480 MiB / 4680 MiB / 475 MiB / 0 MiB / 0 MiB ( 32 MiB / 40 MiB / 20 MiB / 0 MiB / 0 MiB )
23 min passed...480 MiB / 5040 MiB / 497 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 22 MiB / 0 MiB / 0 MiB )
24 min passed...512 MiB / 5720 MiB / 518 MiB / 0 MiB / 0 MiB ( 32 MiB / 680 MiB / 21 MiB / 0 MiB / 0 MiB )
25 min passed...512 MiB / 6080 MiB / 537 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 19 MiB / 0 MiB / 0 MiB )
26 min passed...544 MiB / 6280 MiB / 557 MiB / 0 MiB / 0 MiB ( 32 MiB / 200 MiB / 20 MiB / 0 MiB / 0 MiB )
27 min passed...544 MiB / 6360 MiB / 578 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 21 MiB / 0 MiB / 0 MiB )
28 min passed...608 MiB / 6560 MiB / 598 MiB / 0 MiB / 0 MiB ( 64 MiB / 200 MiB / 20 MiB / 0 MiB / 0 MiB )
29 min passed...608 MiB / 6880 MiB / 620 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 22 MiB / 0 MiB / 0 MiB )
30 min passed...640 MiB / 7480 MiB / 641 MiB / 0 MiB / 0 MiB ( 32 MiB / 600 MiB / 21 MiB / 0 MiB / 0 MiB )
31 min passed...640 MiB / 7680 MiB / 660 MiB / 0 MiB / 0 MiB ( 0 MiB / 200 MiB / 19 MiB / 0 MiB / 0 MiB )
32 min passed...672 MiB / 7720 MiB / 680 MiB / 0 MiB / 0 MiB ( 32 MiB / 40 MiB / 20 MiB / 0 MiB / 0 MiB )
33 min passed...672 MiB / 7760 MiB / 703 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 23 MiB / 0 MiB / 0 MiB )
34 min passed...704 MiB / 8160 MiB / 726 MiB / 0 MiB / 0 MiB ( 32 MiB / 400 MiB / 23 MiB / 0 MiB / 0 MiB )
35 min passed...736 MiB / 8480 MiB / 746 MiB / 0 MiB / 0 MiB ( 32 MiB / 320 MiB / 20 MiB / 0 MiB / 0 MiB )
36 min passed...768 MiB / 8560 MiB / 765 MiB / 0 MiB / 0 MiB ( 32 MiB / 80 MiB / 19 MiB / 0 MiB / 0 MiB )
37 min passed...768 MiB / 8560 MiB / 785 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )
38 min passed...800 MiB / 8560 MiB / 806 MiB / 0 MiB / 0 MiB ( 32 MiB / 0 MiB / 21 MiB / 0 MiB / 0 MiB )
39 min passed...800 MiB / 8560 MiB / 829 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 23 MiB / 0 MiB / 0 MiB )
40 min passed...832 MiB / 8760 MiB / 850 MiB / 0 MiB / 0 MiB ( 32 MiB / 200 MiB / 21 MiB / 0 MiB / 0 MiB )
41 min passed...832 MiB / 9000 MiB / 870 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 20 MiB / 0 MiB / 0 MiB )
42 min passed...864 MiB / 9320 MiB / 890 MiB / 0 MiB / 0 MiB ( 32 MiB / 320 MiB / 20 MiB / 0 MiB / 0 MiB )
43 min passed...928 MiB / 9320 MiB / 910 MiB / 0 MiB / 0 MiB ( 64 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )
44 min passed...928 MiB / 9440 MiB / 931 MiB / 0 MiB / 0 MiB ( 0 MiB / 120 MiB / 21 MiB / 0 MiB / 0 MiB )
45 min passed...960 MiB / 9800 MiB / 953 MiB / 0 MiB / 0 MiB ( 32 MiB / 360 MiB / 22 MiB / 0 MiB / 0 MiB )
46 min passed...960 MiB / 10280 MiB / 972 MiB / 0 MiB / 0 MiB ( 0 MiB / 480 MiB / 19 MiB / 0 MiB / 0 MiB )
47 min passed...960 MiB / 10600 MiB / 993 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 21 MiB / 0 MiB / 0 MiB )
48 min passed...992 MiB / 10800 MiB / 1014 MiB / 0 MiB / 0 MiB ( 32 MiB / 200 MiB / 21 MiB / 0 MiB / 0 MiB )
49 min passed...1024 MiB / 10800 MiB / 1034 MiB / 0 MiB / 0 MiB ( 32 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )

Guest:

0 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 1 MiB / 0 MiB )
16 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
17 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
18 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
19 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 1 MiB / 0 MiB )
20 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
21 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
22 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
23 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 1 MiB / 0 MiB )
24 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
25 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
26 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
27 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 1 MiB / 0 MiB )
28 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
29 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
30 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
31 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 1 MiB / 0 MiB )
32 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
33 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
34 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
35 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 1 MiB / 0 MiB )
36 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
37 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
38 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
39 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 1 MiB / 0 MiB )
40 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
41 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
42 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
43 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 1 MiB / 0 MiB )
44 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
45 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
46 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
47 min passed...Guest Writes/Reads (Changes): 12 MiB / 0 MiB ( 1 MiB / 0 MiB )
48 min passed...Guest Writes/Reads (Changes): 12 MiB / 0 MiB ( 0 MiB / 0 MiB )
49 min passed...Guest Writes/Reads (Changes): 12 MiB / 0 MiB ( 0 MiB / 0 MiB )

Someone knows why the guest only has written so less data (12 MiB according to iostat) while the host was writing so much (1034 MiB according to iostat). There were no reads at all, neither on the guest nor on the host. Is virtio/kvm maybe writing some stuff on its own?

So I think thats because I only started the scripts 4 minutes after booting the VM up so everything was already buffered in the page files so all reads gone to the RAM and not the virtual disks.

 

[Dunuin]

I did a test run and looks like read tests doesn't read from the SSDs at all. Looks like I need to use "primarycache=metadata" like in the official Proxmox ZFS NVMe benchmark so the ARC isn't caching everything.

Guest:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096                                                                                                                                                             B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=2020KiB/s][w=505 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=1521: Fri Aug  6 21:12:42 2021
  write: IOPS=538, BW=2154KiB/s (2206kB/s)(1024MiB/486748msec); 0 zone resets
    clat (usec): min=839, max=38818, avg=1842.92, stdev=540.37
     lat (usec): min=840, max=38820, avg=1845.03, stdev=540.41
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1450], 10.00th=[ 1532], 20.00th=[ 1614],
     | 30.00th=[ 1680], 40.00th=[ 1745], 50.00th=[ 1811], 60.00th=[ 1860],
     | 70.00th=[ 1926], 80.00th=[ 2008], 90.00th=[ 2114], 95.00th=[ 2212],
     | 99.00th=[ 3261], 99.50th=[ 3687], 99.90th=[ 7046], 99.95th=[11600],
     | 99.99th=[24511]
   bw (  KiB/s): min= 1704, max= 2776, per=100.00%, avg=2154.04, stdev=156.53, samples=973
   iops        : min=  426, max=  694, avg=538.49, stdev=39.13, samples=973
  lat (usec)   : 1000=0.07%
  lat (msec)   : 2=79.55%, 4=20.03%, 10=0.29%, 20=0.05%, 50=0.01%
  cpu          : usr=1.30%, sys=7.22%, ctx=572084, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2154KiB/s (2206kB/s), 2154KiB/s-2154KiB/s (2206kB/s-2206kB/s), io=1024MiB (1074MB), run=486748-486748msec

Disk stats (read/write):
  sda: ios=0/797162, merge=0/539085, ticks=0/426521, in_queue=486032, util=99.90%
0 min passed...Guest Writes/Reads (Changes): 5209 MiB / 17 MiB ( 5209 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 5210 MiB / 17 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5210 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5210 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5210 MiB / 17 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5210 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5211 MiB / 17 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5212 MiB / 17 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5212 MiB / 17 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5212 MiB / 17 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=93.0MiB/s][r=23.8k IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=1731: Fri Aug  6 21:27:57 2021
  read: IOPS=17.9k, BW=70.0MiB/s (73.4MB/s)(1024MiB/14620msec)
    clat (usec): min=24, max=683, avg=52.01, stdev=29.66
     lat (usec): min=24, max=685, avg=52.80, stdev=29.80
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   30], 10.00th=[   30], 20.00th=[   33],
     | 30.00th=[   39], 40.00th=[   40], 50.00th=[   41], 60.00th=[   43],
     | 70.00th=[   49], 80.00th=[   60], 90.00th=[  101], 95.00th=[  133],
     | 99.00th=[  145], 99.50th=[  149], 99.90th=[  165], 99.95th=[  176],
     | 99.99th=[  255]
   bw (  KiB/s): min=30728, max=98160, per=99.75%, avg=71544.31, stdev=24955.31, samples=29
   iops        : min= 7682, max=24540, avg=17886.07, stdev=6238.82, samples=29
  lat (usec)   : 50=72.26%, 100=17.57%, 250=10.16%, 500=0.01%, 750=0.01%
  cpu          : usr=8.52%, sys=21.92%, ctx=262179, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=70.0MiB/s (73.4MB/s), 70.0MiB/s-70.0MiB/s (73.4MB/s-73.4MB/s), io=1024MiB (1074MB), run=14620-14620msec

Disk stats (read/write):
  sda: ios=256861/7, merge=0/6, ticks=11079/4, in_queue=14500, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 5212 MiB / 1041 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5212 MiB / 1041 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5212 MiB / 1041 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5212 MiB / 1041 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5212 MiB / 1041 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5213 MiB / 1041 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5214 MiB / 1041 MiB ( 0 MiB / 0 MiB )
Script ran 2303 seconds or 38 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3840 MiB / 4560 MiB / 3846 MiB / 0 MiB / 0 MiB ( 3840 MiB / 4560 MiB / 3846 MiB / 0 MiB / 0 MiB )
2 min passed...7872 MiB / 9120 MiB / 7865 MiB / 0 MiB / 0 MiB ( 4032 MiB / 4560 MiB / 4019 MiB / 0 MiB / 0 MiB )
3 min passed...12064 MiB / 13600 MiB / 12061 MiB / 0 MiB / 0 MiB ( 4192 MiB / 4480 MiB / 4196 MiB / 0 MiB / 0 MiB )
4 min passed...16256 MiB / 18120 MiB / 16311 MiB / 0 MiB / 0 MiB ( 4192 MiB / 4520 MiB / 4250 MiB / 0 MiB / 0 MiB )
5 min passed...20576 MiB / 23240 MiB / 20585 MiB / 0 MiB / 0 MiB ( 4320 MiB / 5120 MiB / 4274 MiB / 0 MiB / 0 MiB )
6 min passed...25056 MiB / 28320 MiB / 25056 MiB / 0 MiB / 0 MiB ( 4480 MiB / 5080 MiB / 4471 MiB / 0 MiB / 0 MiB )
7 min passed...29568 MiB / 33600 MiB / 29577 MiB / 0 MiB / 0 MiB ( 4512 MiB / 5280 MiB / 4521 MiB / 0 MiB / 0 MiB )
8 min passed...34016 MiB / 38400 MiB / 34020 MiB / 0 MiB / 0 MiB ( 4448 MiB / 4800 MiB / 4443 MiB / 0 MiB / 0 MiB )
9 min passed...34336 MiB / 38720 MiB / 34327 MiB / 0 MiB / 0 MiB ( 320 MiB / 320 MiB / 307 MiB / 0 MiB / 0 MiB )
10 min passed...34336 MiB / 39000 MiB / 34337 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 10 MiB / 0 MiB / 0 MiB )
11 min passed...34368 MiB / 39480 MiB / 34352 MiB / 0 MiB / 0 MiB ( 32 MiB / 480 MiB / 15 MiB / 0 MiB / 0 MiB )
12 min passed...34368 MiB / 39560 MiB / 34367 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 15 MiB / 0 MiB / 0 MiB )
13 min passed...34368 MiB / 39600 MiB / 34379 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 12 MiB / 0 MiB / 0 MiB )
14 min passed...34432 MiB / 39600 MiB / 34394 MiB / 0 MiB / 0 MiB ( 64 MiB / 0 MiB / 15 MiB / 0 MiB / 0 MiB )
15 min passed...34432 MiB / 39840 MiB / 34405 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...34464 MiB / 40200 MiB / 34420 MiB / 0 MiB / 0 MiB ( 32 MiB / 360 MiB / 15 MiB / 0 MiB / 0 MiB )
17 min passed...34464 MiB / 40560 MiB / 34433 MiB / 0 MiB / 0 MiB ( 0 MiB / 360 MiB / 13 MiB / 0 MiB / 0 MiB )
18 min passed...34464 MiB / 41200 MiB / 34445 MiB / 0 MiB / 0 MiB ( 0 MiB / 640 MiB / 12 MiB / 0 MiB / 0 MiB )
19 min passed...34464 MiB / 41280 MiB / 34460 MiB / 0 MiB / 0 MiB ( 0 MiB / 80 MiB / 15 MiB / 0 MiB / 0 MiB )
20 min passed...34496 MiB / 41360 MiB / 34471 MiB / 0 MiB / 0 MiB ( 32 MiB / 80 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...34496 MiB / 41400 MiB / 34485 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
22 min passed...34496 MiB / 41720 MiB / 34498 MiB / 0 MiB / 0 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
23 min passed...34528 MiB / 42360 MiB / 34510 MiB / 0 MiB / 0 MiB ( 32 MiB / 640 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...34528 MiB / 42640 MiB / 34524 MiB / 0 MiB / 0 MiB ( 0 MiB / 280 MiB / 14 MiB / 0 MiB / 0 MiB )
25 min passed...34528 MiB / 42640 MiB / 34535 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
26 min passed...34528 MiB / 42880 MiB / 34550 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 15 MiB / 0 MiB / 0 MiB )
27 min passed...34592 MiB / 43080 MiB / 34562 MiB / 0 MiB / 0 MiB ( 64 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
28 min passed...34592 MiB / 43320 MiB / 34575 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 13 MiB / 0 MiB / 0 MiB )
29 min passed...34624 MiB / 43960 MiB / 34589 MiB / 0 MiB / 0 MiB ( 32 MiB / 640 MiB / 14 MiB / 0 MiB / 0 MiB )
30 min passed...34624 MiB / 43960 MiB / 34600 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
31 min passed...34624 MiB / 44000 MiB / 34614 MiB / 0 MiB / 0 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
32 min passed...34656 MiB / 44400 MiB / 34627 MiB / 0 MiB / 0 MiB ( 32 MiB / 400 MiB / 13 MiB / 0 MiB / 0 MiB )
33 min passed...34656 MiB / 44840 MiB / 34640 MiB / 0 MiB / 0 MiB ( 0 MiB / 440 MiB / 13 MiB / 0 MiB / 0 MiB )
34 min passed...34656 MiB / 45040 MiB / 34655 MiB / 0 MiB / 0 MiB ( 0 MiB / 200 MiB / 15 MiB / 0 MiB / 0 MiB )
35 min passed...34688 MiB / 45240 MiB / 34667 MiB / 0 MiB / 0 MiB ( 32 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
36 min passed...34688 MiB / 45240 MiB / 34680 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
37 min passed...34688 MiB / 45480 MiB / 34690 MiB / 0 MiB / 0 MiB ( 0 MiB / 240 MiB / 10 MiB / 0 MiB / 0 MiB )
38 min passed...34688 MiB / 45920 MiB / 34705 MiB / 0 MiB / 0 MiB ( 0 MiB / 440 MiB / 15 MiB / 0 MiB / 0 MiB )

 

[Dunuin]

I ran it again but this time with "primarycache=metadata":

Round 2 - Run 1: 5 disk raidz1
Host is raidz1 of 4x S3710 200GB + 1x S3700 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=32K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.

Test 1: sync 4K random read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1537KiB/s][w=384 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=653: Fri Aug  6 22:09:48 2021
  write: IOPS=299, BW=1197KiB/s (1225kB/s)(1024MiB/876304msec); 0 zone resets
    clat (usec): min=966, max=37591, avg=3328.23, stdev=1076.63
     lat (usec): min=968, max=37594, avg=3330.36, stdev=1076.65
    clat percentiles (usec):
     |  1.00th=[ 1745],  5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2671],
     | 30.00th=[ 2835], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3326],
     | 70.00th=[ 3556], 80.00th=[ 3752], 90.00th=[ 4228], 95.00th=[ 5211],
     | 99.00th=[ 6390], 99.50th=[ 7832], 99.90th=[14222], 99.95th=[17433],
     | 99.99th=[24249]
   bw (  KiB/s): min=  920, max= 1608, per=100.00%, avg=1196.43, stdev=87.46, samples=1752
   iops        : min=  230, max=  402, avg=299.10, stdev=21.87, samples=1752
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=2.55%, 4=84.67%, 10=12.50%, 20=0.25%, 50=0.03%
  cpu          : usr=0.69%, sys=4.18%, ctx=647788, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1197KiB/s (1225kB/s), 1197KiB/s-1197KiB/s (1225kB/s-1225kB/s), io=1024MiB (1074MB), run=876304-876304msec

Disk stats (read/write):
  sda: ios=25/794157, merge=0/552121, ticks=9/820390, in_queue=876352, util=99.69%
0 min passed...Guest Writes/Reads (Changes): 5254 MiB / 19 MiB ( 5254 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5255 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=6562KiB/s][r=1640 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=863: Fri Aug  6 22:27:36 2021
  read: IOPS=1568, BW=6273KiB/s (6424kB/s)(1024MiB/167150msec)
    clat (usec): min=316, max=7071, avg=626.52, stdev=257.03
     lat (usec): min=317, max=7074, avg=628.48, stdev=257.07
    clat percentiles (usec):
     |  1.00th=[  433],  5.00th=[  469], 10.00th=[  490], 20.00th=[  523],
     | 30.00th=[  545], 40.00th=[  570], 50.00th=[  594], 60.00th=[  619],
     | 70.00th=[  644], 80.00th=[  676], 90.00th=[  717], 95.00th=[  742],
     | 99.00th=[ 2114], 99.50th=[ 2802], 99.90th=[ 3326], 99.95th=[ 3458],
     | 99.99th=[ 4555]
   bw (  KiB/s): min= 5184, max= 7744, per=100.00%, avg=6272.88, stdev=515.25, samples=334
   iops        : min= 1296, max= 1936, avg=1568.20, stdev=128.81, samples=334
  lat (usec)   : 500=12.94%, 750=82.58%, 1000=2.44%
  lat (msec)   : 2=0.96%, 4=1.06%, 10=0.01%
  cpu          : usr=2.77%, sys=8.69%, ctx=262340, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6273KiB/s (6424kB/s), 6273KiB/s-6273KiB/s (6424kB/s-6424kB/s), io=1024MiB (1074MB), run=167150-167150msec

Disk stats (read/write):
  sda: ios=262143/12, merge=0/8, ticks=153889/15, in_queue=166856, util=99.83%
0 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5255 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 2845 seconds or 47 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2752 MiB / 3160 MiB / 2750 MiB / 832 MiB / 857 MiB ( 2752 MiB / 3160 MiB / 2750 MiB / 832 MiB / 857 MiB )
2 min passed...5472 MiB / 6040 MiB / 5481 MiB / 1888 MiB / 1904 MiB ( 2720 MiB / 2880 MiB / 2731 MiB / 1056 MiB / 1047 MiB )
3 min passed...8128 MiB / 9040 MiB / 8088 MiB / 3008 MiB / 3010 MiB ( 2656 MiB / 3000 MiB / 2607 MiB / 1120 MiB / 1106 MiB )
4 min passed...10720 MiB / 11960 MiB / 10713 MiB / 4160 MiB / 4177 MiB ( 2592 MiB / 2920 MiB / 2625 MiB / 1152 MiB / 1167 MiB )
5 min passed...13248 MiB / 14960 MiB / 13214 MiB / 5344 MiB / 5352 MiB ( 2528 MiB / 3000 MiB / 2501 MiB / 1184 MiB / 1175 MiB )
6 min passed...15808 MiB / 18120 MiB / 15795 MiB / 6528 MiB / 6551 MiB ( 2560 MiB / 3160 MiB / 2581 MiB / 1184 MiB / 1199 MiB )
7 min passed...18400 MiB / 20840 MiB / 18395 MiB / 7744 MiB / 7765 MiB ( 2592 MiB / 2720 MiB / 2600 MiB / 1216 MiB / 1214 MiB )
8 min passed...20992 MiB / 23760 MiB / 20998 MiB / 8960 MiB / 8988 MiB ( 2592 MiB / 2920 MiB / 2603 MiB / 1216 MiB / 1223 MiB )
9 min passed...23552 MiB / 26920 MiB / 23573 MiB / 10208 MiB / 10198 MiB ( 2560 MiB / 3160 MiB / 2575 MiB / 1248 MiB / 1210 MiB )
10 min passed...26080 MiB / 29800 MiB / 26066 MiB / 11424 MiB / 11405 MiB ( 2528 MiB / 2880 MiB / 2493 MiB / 1216 MiB / 1207 MiB )
11 min passed...28640 MiB / 33160 MiB / 28618 MiB / 12608 MiB / 12598 MiB ( 2560 MiB / 3360 MiB / 2552 MiB / 1184 MiB / 1193 MiB )
12 min passed...31232 MiB / 35880 MiB / 31226 MiB / 13824 MiB / 13802 MiB ( 2592 MiB / 2720 MiB / 2608 MiB / 1216 MiB / 1204 MiB )
13 min passed...33824 MiB / 38680 MiB / 33824 MiB / 14976 MiB / 14971 MiB ( 2592 MiB / 2800 MiB / 2598 MiB / 1152 MiB / 1169 MiB )
14 min passed...36352 MiB / 41880 MiB / 36364 MiB / 16032 MiB / 16024 MiB ( 2528 MiB / 3200 MiB / 2540 MiB / 1056 MiB / 1053 MiB )
15 min passed...37728 MiB / 43600 MiB / 37723 MiB / 16480 MiB / 16468 MiB ( 1376 MiB / 1720 MiB / 1359 MiB / 448 MiB / 444 MiB )
16 min passed...37760 MiB / 43840 MiB / 37726 MiB / 16480 MiB / 16468 MiB ( 32 MiB / 240 MiB / 3 MiB / 0 MiB / 0 MiB )
17 min passed...37760 MiB / 44040 MiB / 37726 MiB / 16480 MiB / 16468 MiB ( 0 MiB / 200 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...37760 MiB / 44040 MiB / 37726 MiB / 16480 MiB / 16468 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...37760 MiB / 44040 MiB / 37726 MiB / 16480 MiB / 16468 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...37760 MiB / 44320 MiB / 37729 MiB / 16480 MiB / 16468 MiB ( 0 MiB / 280 MiB / 3 MiB / 0 MiB / 0 MiB )
21 min passed...37760 MiB / 44640 MiB / 37734 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 320 MiB / 5 MiB / 0 MiB / 1 MiB )
22 min passed...37760 MiB / 44840 MiB / 37736 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 200 MiB / 2 MiB / 0 MiB / 0 MiB )
23 min passed...37760 MiB / 44840 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
24 min passed...37760 MiB / 44960 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 120 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...37760 MiB / 45200 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...37760 MiB / 45280 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 80 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...37760 MiB / 45920 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...37760 MiB / 45920 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...37760 MiB / 45960 MiB / 37740 MiB / 16480 MiB / 16469 MiB ( 0 MiB / 40 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...37760 MiB / 46400 MiB / 37742 MiB / 18176 MiB / 18173 MiB ( 0 MiB / 440 MiB / 2 MiB / 1696 MiB / 1704 MiB )
31 min passed...37760 MiB / 46640 MiB / 37746 MiB / 21248 MiB / 21260 MiB ( 0 MiB / 240 MiB / 4 MiB / 3072 MiB / 3087 MiB )
32 min passed...37760 MiB / 47360 MiB / 37746 MiB / 24096 MiB / 24107 MiB ( 0 MiB / 720 MiB / 0 MiB / 2848 MiB / 2847 MiB )
33 min passed...37760 MiB / 47360 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 0 MiB / 4 MiB / 576 MiB / 554 MiB )
34 min passed...37760 MiB / 47360 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...37760 MiB / 47600 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...37760 MiB / 48120 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 520 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...37760 MiB / 48320 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 200 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...37760 MiB / 48560 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 240 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...37760 MiB / 48560 MiB / 37750 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...37760 MiB / 48720 MiB / 37753 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 160 MiB / 3 MiB / 0 MiB / 0 MiB )
41 min passed...37760 MiB / 49120 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 400 MiB / 4 MiB / 0 MiB / 0 MiB )
42 min passed...37760 MiB / 49480 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 360 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...37760 MiB / 49760 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...37760 MiB / 49760 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...37760 MiB / 49760 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...37760 MiB / 49920 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 160 MiB / 0 MiB / 0 MiB / 0 MiB )
47 min passed...37760 MiB / 50200 MiB / 37757 MiB / 24672 MiB / 24661 MiB ( 0 MiB / 280 MiB / 0 MiB / 0 MiB / 0 MiB )

You can clearly see that the ARC wasn't used this time and there there is scary read amplification too. It basically read 24GiB from the SSDs to access 1GiB inside the VM.

 

[Dunuin]

Round 2 - Run 1 - Test 2: sync 16K random read/write

guest:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB,                                                                                                                                                              (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=5045KiB/s][w=315 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=652: Sat Aug  7 01:07:31 2021
  write: IOPS=352, BW=5635KiB/s (5770kB/s)(1024MiB/186099msec); 0 zone resets
    clat (usec): min=853, max=24546, avg=2819.80, stdev=995.54
     lat (usec): min=854, max=24548, avg=2822.04, stdev=995.57
    clat percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 1729], 10.00th=[ 1860], 20.00th=[ 2073],
     | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2737], 60.00th=[ 2868],
     | 70.00th=[ 3064], 80.00th=[ 3359], 90.00th=[ 3752], 95.00th=[ 4113],
     | 99.00th=[ 5932], 99.50th=[ 6521], 99.90th=[13698], 99.95th=[16909],
     | 99.99th=[20579]
   bw (  KiB/s): min= 4576, max= 7392, per=100.00%, avg=5634.31, stdev=529.64, samples=372
   iops        : min=  286, max=  462, avg=352.14, stdev=33.10, samples=372
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=16.87%, 4=77.14%, 10=5.78%, 20=0.19%, 50=0.01%
  cpu          : usr=0.99%, sys=5.02%, ctx=146100, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5635KiB/s (5770kB/s), 5635KiB/s-5635KiB/s (5770kB/s-5770kB/s), io=1024MiB (1074MB), run=186099-186099msec

Disk stats (read/write):
  sda: ios=2/196771, merge=0/132392, ticks=1/169163, in_queue=185712, util=99.88%
0 min passed...Guest Writes/Reads (Changes): 2055 MiB / 19 MiB ( 2055 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2055 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=26.0MiB/s][r=1665 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=862: Sat Aug  7 01:23:12 2021
  read: IOPS=1642, BW=25.7MiB/s (26.9MB/s)(1024MiB/39911msec)
    clat (usec): min=346, max=3680, avg=598.24, stdev=112.64
     lat (usec): min=347, max=3682, avg=600.21, stdev=112.72
    clat percentiles (usec):
     |  1.00th=[  441],  5.00th=[  474], 10.00th=[  494], 20.00th=[  523],
     | 30.00th=[  545], 40.00th=[  570], 50.00th=[  594], 60.00th=[  619],
     | 70.00th=[  644], 80.00th=[  668], 90.00th=[  701], 95.00th=[  725],
     | 99.00th=[  766], 99.50th=[  783], 99.90th=[ 2212], 99.95th=[ 2933],
     | 99.99th=[ 3359]
   bw (  KiB/s): min=22848, max=30656, per=99.95%, avg=26258.39, stdev=1856.28, samples=79
   iops        : min= 1428, max= 1916, avg=1641.14, stdev=116.02, samples=79
  lat (usec)   : 500=12.26%, 750=85.60%, 1000=1.92%
  lat (msec)   : 2=0.10%, 4=0.11%
  cpu          : usr=2.87%, sys=8.75%, ctx=65638, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=25.7MiB/s (26.9MB/s), 25.7MiB/s-25.7MiB/s (26.9MB/s-26.9MB/s), io=1024MiB (1074MB), run=39911-39911msec

Disk stats (read/write):
  sda: ios=65512/14, merge=0/12, ticks=36700/22, in_queue=40024, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 1 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2060 MiB / 1043 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2060 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2060 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 2028 seconds or 33 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3136 MiB / 3720 MiB / 3127 MiB / 320 MiB / 360 MiB ( 3136 MiB / 3720 MiB / 3127 MiB / 320 MiB / 360 MiB )
2 min passed...6080 MiB / 6840 MiB / 6089 MiB / 928 MiB / 954 MiB ( 2944 MiB / 3120 MiB / 2962 MiB / 608 MiB / 594 MiB )
3 min passed...8896 MiB / 10200 MiB / 8860 MiB / 1600 MiB / 1612 MiB ( 2816 MiB / 3360 MiB / 2771 MiB / 672 MiB / 658 MiB )
4 min passed...9152 MiB / 10520 MiB / 9132 MiB / 1632 MiB / 1667 MiB ( 256 MiB / 320 MiB / 272 MiB / 32 MiB / 55 MiB )
5 min passed...9184 MiB / 10560 MiB / 9142 MiB / 1632 MiB / 1667 MiB ( 32 MiB / 40 MiB / 10 MiB / 0 MiB / 0 MiB )
6 min passed...9216 MiB / 10560 MiB / 9155 MiB / 1632 MiB / 1668 MiB ( 32 MiB / 0 MiB / 13 MiB / 0 MiB / 1 MiB )
7 min passed...9216 MiB / 10840 MiB / 9168 MiB / 1632 MiB / 1668 MiB ( 0 MiB / 280 MiB / 13 MiB / 0 MiB / 0 MiB )
8 min passed...9216 MiB / 11320 MiB / 9180 MiB / 1632 MiB / 1668 MiB ( 0 MiB / 480 MiB / 12 MiB / 0 MiB / 0 MiB )
9 min passed...9216 MiB / 11480 MiB / 9190 MiB / 1632 MiB / 1668 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
10 min passed...9216 MiB / 11520 MiB / 9204 MiB / 1632 MiB / 1669 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 1 MiB )
11 min passed...9248 MiB / 11680 MiB / 9216 MiB / 1632 MiB / 1669 MiB ( 32 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
12 min passed...9248 MiB / 12160 MiB / 9230 MiB / 1632 MiB / 1669 MiB ( 0 MiB / 480 MiB / 14 MiB / 0 MiB / 0 MiB )
13 min passed...9280 MiB / 12560 MiB / 9244 MiB / 1632 MiB / 1669 MiB ( 32 MiB / 400 MiB / 14 MiB / 0 MiB / 0 MiB )
14 min passed...9280 MiB / 13200 MiB / 9258 MiB / 1632 MiB / 1670 MiB ( 0 MiB / 640 MiB / 14 MiB / 0 MiB / 1 MiB )
15 min passed...9280 MiB / 13240 MiB / 9270 MiB / 1632 MiB / 1670 MiB ( 0 MiB / 40 MiB / 12 MiB / 0 MiB / 0 MiB )
16 min passed...9280 MiB / 13280 MiB / 9282 MiB / 1632 MiB / 1670 MiB ( 0 MiB / 40 MiB / 12 MiB / 0 MiB / 0 MiB )
17 min passed...9312 MiB / 13720 MiB / 9294 MiB / 1632 MiB / 1670 MiB ( 32 MiB / 440 MiB / 12 MiB / 0 MiB / 0 MiB )
18 min passed...9344 MiB / 14120 MiB / 9304 MiB / 1632 MiB / 1679 MiB ( 32 MiB / 400 MiB / 10 MiB / 0 MiB / 9 MiB )
19 min passed...9376 MiB / 14160 MiB / 9315 MiB / 3680 MiB / 3719 MiB ( 32 MiB / 40 MiB / 11 MiB / 2048 MiB / 2040 MiB )
20 min passed...9376 MiB / 14200 MiB / 9325 MiB / 3680 MiB / 3719 MiB ( 0 MiB / 40 MiB / 10 MiB / 0 MiB / 0 MiB )
21 min passed...9376 MiB / 14200 MiB / 9337 MiB / 3680 MiB / 3719 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
22 min passed...9376 MiB / 14480 MiB / 9348 MiB / 3680 MiB / 3719 MiB ( 0 MiB / 280 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...9376 MiB / 15040 MiB / 9359 MiB / 3680 MiB / 3720 MiB ( 0 MiB / 560 MiB / 11 MiB / 0 MiB / 1 MiB )
24 min passed...9408 MiB / 15600 MiB / 9372 MiB / 3680 MiB / 3720 MiB ( 32 MiB / 560 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...9408 MiB / 15920 MiB / 9381 MiB / 3680 MiB / 3720 MiB ( 0 MiB / 320 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...9408 MiB / 15960 MiB / 9395 MiB / 3680 MiB / 3720 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
27 min passed...9440 MiB / 15960 MiB / 9405 MiB / 3680 MiB / 3720 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...9440 MiB / 16000 MiB / 9417 MiB / 3680 MiB / 3721 MiB ( 0 MiB / 40 MiB / 12 MiB / 0 MiB / 1 MiB )
29 min passed...9440 MiB / 16280 MiB / 9429 MiB / 3680 MiB / 3721 MiB ( 0 MiB / 280 MiB / 12 MiB / 0 MiB / 0 MiB )
30 min passed...9440 MiB / 16840 MiB / 9438 MiB / 3680 MiB / 3721 MiB ( 0 MiB / 560 MiB / 9 MiB / 0 MiB / 0 MiB )
31 min passed...9472 MiB / 16880 MiB / 9451 MiB / 3680 MiB / 3721 MiB ( 32 MiB / 40 MiB / 13 MiB / 0 MiB / 0 MiB )
32 min passed...9504 MiB / 16960 MiB / 9462 MiB / 3680 MiB / 3721 MiB ( 32 MiB / 80 MiB / 11 MiB / 0 MiB / 0 MiB )
33 min passed...9536 MiB / 17520 MiB / 9475 MiB / 3680 MiB / 3722 MiB ( 32 MiB / 560 MiB / 13 MiB / 0 MiB / 1 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 3: sync 4M sequential read/write

guest:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=204MiB/s][w=51 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=639: Sat Aug  7 01:52:24 2021
  write: IOPS=65, BW=262MiB/s (275MB/s)(10.0GiB/39039msec); 0 zone resets
    clat (usec): min=8160, max=60873, avg=13600.43, stdev=4560.02
     lat (usec): min=8162, max=60876, avg=13603.74, stdev=4560.15
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10421], 10.00th=[10814], 20.00th=[11207],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[14222], 90.00th=[20317], 95.00th=[23725],
     | 99.00th=[30540], 99.50th=[32900], 99.90th=[42730], 99.95th=[55837],
     | 99.99th=[61080]
   bw (  KiB/s): min=155648, max=335872, per=100.00%, avg=268640.28, stdev=51130.45, samples=78
   iops        : min=   38, max=   82, avg=65.56, stdev=12.47, samples=78
  lat (msec)   : 10=2.34%, 20=87.03%, 50=10.55%, 100=0.08%
  cpu          : usr=10.90%, sys=2.63%, ctx=5714, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=262MiB/s (275MB/s), 262MiB/s-262MiB/s (275MB/s-275MB/s), io=10.0GiB (10.7GB), run=39039-39039msec

Disk stats (read/write):
  sda: ios=0/15359, merge=0/5287, ticks=0/44316, in_queue=49060, util=98.51%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 10281 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10282 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10283 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=937MiB/s][r=234 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=847: Sat Aug  7 02:07:36 2021
  read: IOPS=224, BW=898MiB/s (941MB/s)(10.0GiB/11408msec)
    clat (usec): min=3723, max=10078, avg=4445.04, stdev=450.16
     lat (usec): min=3725, max=10080, avg=4446.73, stdev=450.21
    clat percentiles (usec):
     |  1.00th=[ 3884],  5.00th=[ 3982], 10.00th=[ 4015], 20.00th=[ 4113],
     | 30.00th=[ 4178], 40.00th=[ 4228], 50.00th=[ 4293], 60.00th=[ 4424],
     | 70.00th=[ 4555], 80.00th=[ 4752], 90.00th=[ 5014], 95.00th=[ 5211],
     | 99.00th=[ 5800], 99.50th=[ 6128], 99.90th=[ 8029], 99.95th=[ 8029],
     | 99.99th=[10028]
   bw (  KiB/s): min=827392, max=974848, per=99.78%, avg=917131.64, stdev=47799.08, samples=22
   iops        : min=  202, max=  238, avg=223.91, stdev=11.67, samples=22
  lat (msec)   : 4=7.46%, 10=92.50%, 20=0.04%
  cpu          : usr=0.43%, sys=5.52%, ctx=2563, majf=0, minf=523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=898MiB/s (941MB/s), 898MiB/s-898MiB/s (941MB/s-941MB/s), io=10.0GiB (10.7GB), run=11408-11408msec

Disk stats (read/write):
  sda: ios=10230/7, merge=0/6, ticks=39409/19, in_queue=40636, util=99.56%
0 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 1 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10284 MiB / 10259 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10285 MiB / 10259 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10259 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10286 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10286 MiB / 10259 MiB ( 0 MiB / 0 MiB )
Script ran 1853 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...26464 MiB / 28360 MiB / 26503 MiB / 0 MiB / 31 MiB ( 26464 MiB / 28360 MiB / 26503 MiB / 0 MiB / 31 MiB )
2 min passed...26528 MiB / 28800 MiB / 26531 MiB / 0 MiB / 31 MiB ( 64 MiB / 440 MiB / 28 MiB / 0 MiB / 0 MiB )
3 min passed...26560 MiB / 29320 MiB / 26557 MiB / 0 MiB / 32 MiB ( 32 MiB / 520 MiB / 26 MiB / 0 MiB / 1 MiB )
4 min passed...26560 MiB / 29840 MiB / 26585 MiB / 32 MiB / 32 MiB ( 0 MiB / 520 MiB / 28 MiB / 32 MiB / 0 MiB )
5 min passed...26592 MiB / 29840 MiB / 26602 MiB / 32 MiB / 33 MiB ( 32 MiB / 0 MiB / 17 MiB / 0 MiB / 1 MiB )
6 min passed...26592 MiB / 29880 MiB / 26618 MiB / 32 MiB / 33 MiB ( 0 MiB / 40 MiB / 16 MiB / 0 MiB / 0 MiB )
7 min passed...26624 MiB / 29920 MiB / 26634 MiB / 32 MiB / 33 MiB ( 32 MiB / 40 MiB / 16 MiB / 0 MiB / 0 MiB )
8 min passed...26624 MiB / 30080 MiB / 26650 MiB / 32 MiB / 33 MiB ( 0 MiB / 160 MiB / 16 MiB / 0 MiB / 0 MiB )
9 min passed...26624 MiB / 30760 MiB / 26666 MiB / 32 MiB / 33 MiB ( 0 MiB / 680 MiB / 16 MiB / 0 MiB / 0 MiB )
10 min passed...26688 MiB / 30800 MiB / 26684 MiB / 32 MiB / 34 MiB ( 64 MiB / 40 MiB / 18 MiB / 0 MiB / 1 MiB )
11 min passed...26720 MiB / 30880 MiB / 26698 MiB / 32 MiB / 34 MiB ( 32 MiB / 80 MiB / 14 MiB / 0 MiB / 0 MiB )
12 min passed...26720 MiB / 30960 MiB / 26715 MiB / 32 MiB / 34 MiB ( 0 MiB / 80 MiB / 17 MiB / 0 MiB / 0 MiB )
13 min passed...26720 MiB / 31040 MiB / 26728 MiB / 32 MiB / 34 MiB ( 0 MiB / 80 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...26720 MiB / 31080 MiB / 26748 MiB / 32 MiB / 34 MiB ( 0 MiB / 40 MiB / 20 MiB / 0 MiB / 0 MiB )
15 min passed...26752 MiB / 31120 MiB / 26763 MiB / 32 MiB / 35 MiB ( 32 MiB / 40 MiB / 15 MiB / 0 MiB / 1 MiB )
16 min passed...26752 MiB / 31160 MiB / 26776 MiB / 11520 MiB / 11569 MiB ( 0 MiB / 40 MiB / 13 MiB / 11488 MiB / 11534 MiB )
17 min passed...26784 MiB / 31320 MiB / 26789 MiB / 11520 MiB / 11570 MiB ( 32 MiB / 160 MiB / 13 MiB / 0 MiB / 1 MiB )
18 min passed...26784 MiB / 31640 MiB / 26802 MiB / 11520 MiB / 11570 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
19 min passed...26784 MiB / 32320 MiB / 26817 MiB / 11520 MiB / 11570 MiB ( 0 MiB / 680 MiB / 15 MiB / 0 MiB / 0 MiB )
20 min passed...26784 MiB / 32440 MiB / 26831 MiB / 11520 MiB / 11570 MiB ( 0 MiB / 120 MiB / 14 MiB / 0 MiB / 0 MiB )
21 min passed...26848 MiB / 32440 MiB / 26844 MiB / 11520 MiB / 11570 MiB ( 64 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...26880 MiB / 32480 MiB / 26856 MiB / 11520 MiB / 11570 MiB ( 32 MiB / 40 MiB / 12 MiB / 0 MiB / 0 MiB )
23 min passed...26880 MiB / 32480 MiB / 26871 MiB / 11520 MiB / 11570 MiB ( 0 MiB / 0 MiB / 15 MiB / 0 MiB / 0 MiB )
24 min passed...26880 MiB / 32760 MiB / 26886 MiB / 11520 MiB / 11571 MiB ( 0 MiB / 280 MiB / 15 MiB / 0 MiB / 1 MiB )
25 min passed...26880 MiB / 32960 MiB / 26900 MiB / 11520 MiB / 11571 MiB ( 0 MiB / 200 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...26912 MiB / 32960 MiB / 26914 MiB / 11520 MiB / 11571 MiB ( 32 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
27 min passed...26912 MiB / 33000 MiB / 26929 MiB / 11520 MiB / 11572 MiB ( 0 MiB / 40 MiB / 15 MiB / 0 MiB / 1 MiB )
28 min passed...26912 MiB / 33040 MiB / 26944 MiB / 11520 MiB / 11572 MiB ( 0 MiB / 40 MiB / 15 MiB / 0 MiB / 0 MiB )
29 min passed...26944 MiB / 33320 MiB / 26962 MiB / 11520 MiB / 11572 MiB ( 32 MiB / 280 MiB / 18 MiB / 0 MiB / 0 MiB )
30 min passed...26944 MiB / 33480 MiB / 26976 MiB / 11520 MiB / 11572 MiB ( 0 MiB / 160 MiB / 14 MiB / 0 MiB / 0 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 4: async 4K random read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][92.3%][w=62.4MiB/s][w=15.0k IOPS][eta 00m:01s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=660: Sat Aug  7 02:26:22 2021
  write: IOPS=20.5k, BW=80.1MiB/s (83.0MB/s)(1024MiB/12786msec); 0 zone resets
    slat (usec): min=7, max=17125, avg=27.46, stdev=47.58
    clat (usec): min=571, max=45043, avg=3085.81, stdev=3238.17
     lat (usec): min=590, max=45088, avg=3114.85, stdev=3241.12
    clat percentiles (usec):
     |  1.00th=[ 1156],  5.00th=[ 1565], 10.00th=[ 1680], 20.00th=[ 1795],
     | 30.00th=[ 1893], 40.00th=[ 2008], 50.00th=[ 2343], 60.00th=[ 2540],
     | 70.00th=[ 2737], 80.00th=[ 3130], 90.00th=[ 4113], 95.00th=[ 6652],
     | 99.00th=[21365], 99.50th=[23725], 99.90th=[27395], 99.95th=[28443],
     | 99.99th=[33424]
   bw (  KiB/s): min=16456, max=128816, per=98.35%, avg=80652.80, stdev=35420.61, samples=25
   iops        : min= 4114, max=32204, avg=20163.20, stdev=8855.15, samples=25
  lat (usec)   : 750=0.04%, 1000=0.36%
  lat (msec)   : 2=38.98%, 4=50.26%, 10=6.87%, 20=2.14%, 50=1.34%
  cpu          : usr=15.60%, sys=44.40%, ctx=196523, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=80.1MiB/s (83.0MB/s), 80.1MiB/s-80.1MiB/s (83.0MB/s-83.0MB/s), io=1024MiB (1074MB), run=12786-12786msec

Disk stats (read/write):
  sda: ios=1/256867, merge=0/1177, ticks=2/365870, in_queue=369596, util=99.99%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 1029 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1035 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1036 MiB / 19 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1036 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1036 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=144MiB/s][r=36.9k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=975: Sat Aug  7 02:41:30 2021
  read: IOPS=35.0k, BW=137MiB/s (143MB/s)(1024MiB/7484msec)
    slat (usec): min=4, max=2570, avg=11.45, stdev=24.10
    clat (usec): min=454, max=17970, avg=1809.37, stdev=420.35
     lat (usec): min=460, max=17981, avg=1822.22, stdev=423.03
    clat percentiles (usec):
     |  1.00th=[ 1090],  5.00th=[ 1336], 10.00th=[ 1450], 20.00th=[ 1565],
     | 30.00th=[ 1631], 40.00th=[ 1680], 50.00th=[ 1729], 60.00th=[ 1778],
     | 70.00th=[ 1844], 80.00th=[ 1926], 90.00th=[ 2409], 95.00th=[ 2737],
     | 99.00th=[ 3097], 99.50th=[ 3326], 99.90th=[ 4146], 99.95th=[ 4621],
     | 99.99th=[ 9372]
   bw (  KiB/s): min=104208, max=148128, per=98.89%, avg=138556.57, stdev=12651.04, samples=14
   iops        : min=26052, max=37032, avg=34639.14, stdev=3162.76, samples=14
  lat (usec)   : 500=0.01%, 750=0.07%, 1000=0.56%
  lat (msec)   : 2=83.35%, 4=15.87%, 10=0.14%, 20=0.01%
  cpu          : usr=28.96%, sys=48.68%, ctx=36056, majf=0, minf=74
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=1024MiB (1074MB), run=7484-7484msec

Disk stats (read/write):
  sda: ios=257822/2, merge=0/3, ticks=404638/6, in_queue=404964, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1036 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1037 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1038 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1822 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5184 MiB / 5520 MiB / 5223 MiB / 7168 MiB / 7145 MiB ( 5184 MiB / 5520 MiB / 5223 MiB / 7168 MiB / 7145 MiB )
2 min passed...5248 MiB / 5520 MiB / 5239 MiB / 7168 MiB / 7145 MiB ( 64 MiB / 0 MiB / 16 MiB / 0 MiB / 0 MiB )
3 min passed...5280 MiB / 5600 MiB / 5252 MiB / 7168 MiB / 7145 MiB ( 32 MiB / 80 MiB / 13 MiB / 0 MiB / 0 MiB )
4 min passed...5280 MiB / 5920 MiB / 5265 MiB / 7168 MiB / 7145 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...5280 MiB / 6040 MiB / 5279 MiB / 7168 MiB / 7145 MiB ( 0 MiB / 120 MiB / 14 MiB / 0 MiB / 0 MiB )
6 min passed...5280 MiB / 6040 MiB / 5295 MiB / 7168 MiB / 7146 MiB ( 0 MiB / 0 MiB / 16 MiB / 0 MiB / 1 MiB )
7 min passed...5312 MiB / 6040 MiB / 5312 MiB / 7168 MiB / 7146 MiB ( 32 MiB / 0 MiB / 17 MiB / 0 MiB / 0 MiB )
8 min passed...5312 MiB / 6200 MiB / 5331 MiB / 7168 MiB / 7146 MiB ( 0 MiB / 160 MiB / 19 MiB / 0 MiB / 0 MiB )
9 min passed...5344 MiB / 6560 MiB / 5346 MiB / 7168 MiB / 7146 MiB ( 32 MiB / 360 MiB / 15 MiB / 0 MiB / 0 MiB )
10 min passed...5344 MiB / 6760 MiB / 5362 MiB / 7168 MiB / 7146 MiB ( 0 MiB / 200 MiB / 16 MiB / 0 MiB / 0 MiB )
11 min passed...5344 MiB / 6920 MiB / 5375 MiB / 7168 MiB / 7146 MiB ( 0 MiB / 160 MiB / 13 MiB / 0 MiB / 0 MiB )
12 min passed...5408 MiB / 6960 MiB / 5391 MiB / 7168 MiB / 7147 MiB ( 64 MiB / 40 MiB / 16 MiB / 0 MiB / 1 MiB )
13 min passed...5440 MiB / 7040 MiB / 5406 MiB / 7168 MiB / 7147 MiB ( 32 MiB / 80 MiB / 15 MiB / 0 MiB / 0 MiB )
14 min passed...5440 MiB / 7600 MiB / 5418 MiB / 7168 MiB / 7147 MiB ( 0 MiB / 560 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...5440 MiB / 7840 MiB / 5437 MiB / 7168 MiB / 7147 MiB ( 0 MiB / 240 MiB / 19 MiB / 0 MiB / 0 MiB )
16 min passed...5440 MiB / 7880 MiB / 5451 MiB / 15296 MiB / 15290 MiB ( 0 MiB / 40 MiB / 14 MiB / 8128 MiB / 8143 MiB )
17 min passed...5472 MiB / 7920 MiB / 5465 MiB / 15296 MiB / 15290 MiB ( 32 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
18 min passed...5472 MiB / 7960 MiB / 5479 MiB / 15296 MiB / 15290 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
19 min passed...5472 MiB / 8160 MiB / 5491 MiB / 15296 MiB / 15290 MiB ( 0 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...5504 MiB / 8320 MiB / 5506 MiB / 15296 MiB / 15290 MiB ( 32 MiB / 160 MiB / 15 MiB / 0 MiB / 0 MiB )
21 min passed...5504 MiB / 8720 MiB / 5519 MiB / 15296 MiB / 15291 MiB ( 0 MiB / 400 MiB / 13 MiB / 0 MiB / 1 MiB )
22 min passed...5504 MiB / 8760 MiB / 5532 MiB / 15296 MiB / 15291 MiB ( 0 MiB / 40 MiB / 13 MiB / 0 MiB / 0 MiB )
23 min passed...5504 MiB / 8840 MiB / 5543 MiB / 15296 MiB / 15291 MiB ( 0 MiB / 80 MiB / 11 MiB / 0 MiB / 0 MiB )
24 min passed...5568 MiB / 9200 MiB / 5557 MiB / 15296 MiB / 15291 MiB ( 64 MiB / 360 MiB / 14 MiB / 0 MiB / 0 MiB )
25 min passed...5600 MiB / 9200 MiB / 5571 MiB / 15296 MiB / 15291 MiB ( 32 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...5600 MiB / 9840 MiB / 5585 MiB / 15296 MiB / 15291 MiB ( 0 MiB / 640 MiB / 14 MiB / 0 MiB / 0 MiB )
27 min passed...5600 MiB / 10000 MiB / 5599 MiB / 15296 MiB / 15292 MiB ( 0 MiB / 160 MiB / 14 MiB / 0 MiB / 1 MiB )
28 min passed...5600 MiB / 10000 MiB / 5611 MiB / 15296 MiB / 15292 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
29 min passed...5632 MiB / 10480 MiB / 5622 MiB / 15296 MiB / 15292 MiB ( 32 MiB / 480 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...5632 MiB / 10840 MiB / 5633 MiB / 15296 MiB / 15292 MiB ( 0 MiB / 360 MiB / 11 MiB / 0 MiB / 0 MiB )

It's interesting how much reads async writes will produce. Never noticed this with enabled ARC.

 

[Dunuin]

Round 2 - Run 1 - Test 5: async 16K random read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=640: Sat Aug  7 03:14:53 2021
  write: IOPS=24.3k, BW=380MiB/s (399MB/s)(1024MiB/2694msec); 0 zone resets
    slat (usec): min=7, max=3344, avg=22.25, stdev=34.31
    clat (usec): min=361, max=22505, avg=2595.08, stdev=2051.89
     lat (usec): min=467, max=22517, avg=2618.71, stdev=2053.33
    clat percentiles (usec):
     |  1.00th=[ 1254],  5.00th=[ 1663], 10.00th=[ 1795], 20.00th=[ 1893],
     | 30.00th=[ 1942], 40.00th=[ 1991], 50.00th=[ 2040], 60.00th=[ 2114],
     | 70.00th=[ 2245], 80.00th=[ 2507], 90.00th=[ 3097], 95.00th=[ 5735],
     | 99.00th=[13304], 99.50th=[15270], 99.90th=[18482], 99.95th=[19792],
     | 99.99th=[22152]
   bw (  KiB/s): min=325504, max=480064, per=100.00%, avg=407232.00, stdev=72374.02, samples=5
   iops        : min=20344, max=30004, avg=25452.00, stdev=4523.38, samples=5
  lat (usec)   : 500=0.01%, 750=0.04%, 1000=0.17%
  lat (msec)   : 2=41.98%, 4=50.78%, 10=4.49%, 20=2.50%, 50=0.04%
  cpu          : usr=22.24%, sys=48.05%, ctx=47569, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=1024MiB (1074MB), run=2694-2694msec

Disk stats (read/write):
  sda: ios=0/65018, merge=0/0, ticks=0/46210, in_queue=46056, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=891: Sat Aug  7 03:29:56 2021
  read: IOPS=34.3k, BW=536MiB/s (562MB/s)(1024MiB/1911msec)
    slat (usec): min=4, max=1183, avg=12.01, stdev=21.04
    clat (usec): min=698, max=17643, avg=1846.15, stdev=489.44
     lat (usec): min=722, max=17654, avg=1859.60, stdev=491.72
    clat percentiles (usec):
     |  1.00th=[ 1172],  5.00th=[ 1385], 10.00th=[ 1483], 20.00th=[ 1565],
     | 30.00th=[ 1631], 40.00th=[ 1680], 50.00th=[ 1729], 60.00th=[ 1778],
     | 70.00th=[ 1844], 80.00th=[ 1942], 90.00th=[ 2474], 95.00th=[ 2900],
     | 99.00th=[ 3654], 99.50th=[ 4080], 99.90th=[ 4883], 99.95th=[ 5211],
     | 99.99th=[10421]
   bw (  KiB/s): min=532128, max=595104, per=100.00%, avg=569312.00, stdev=32997.38, samples=3
   iops        : min=33258, max=37194, avg=35582.00, stdev=2062.34, samples=3
  lat (usec)   : 750=0.01%, 1000=0.33%
  lat (msec)   : 2=82.29%, 4=16.81%, 10=0.55%, 20=0.01%
  cpu          : usr=28.27%, sys=48.95%, ctx=8788, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=1024MiB (1074MB), run=1911-1911msec

Disk stats (read/write):
  sda: ios=65010/0, merge=0/0, ticks=105221/0, in_queue=105164, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1807 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1888 MiB / 2160 MiB / 1841 MiB / 416 MiB / 426 MiB ( 1888 MiB / 2160 MiB / 1841 MiB / 416 MiB / 426 MiB )
2 min passed...1888 MiB / 2800 MiB / 1854 MiB / 416 MiB / 427 MiB ( 0 MiB / 640 MiB / 13 MiB / 0 MiB / 1 MiB )
3 min passed...1888 MiB / 3040 MiB / 1868 MiB / 416 MiB / 427 MiB ( 0 MiB / 240 MiB / 14 MiB / 0 MiB / 0 MiB )
4 min passed...1888 MiB / 3360 MiB / 1881 MiB / 416 MiB / 427 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...1920 MiB / 3520 MiB / 1892 MiB / 416 MiB / 428 MiB ( 32 MiB / 160 MiB / 11 MiB / 0 MiB / 1 MiB )
6 min passed...1920 MiB / 3640 MiB / 1906 MiB / 416 MiB / 428 MiB ( 0 MiB / 120 MiB / 14 MiB / 0 MiB / 0 MiB )
7 min passed...1920 MiB / 3960 MiB / 1918 MiB / 416 MiB / 428 MiB ( 0 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
8 min passed...1984 MiB / 4000 MiB / 1930 MiB / 416 MiB / 429 MiB ( 64 MiB / 40 MiB / 12 MiB / 0 MiB / 1 MiB )
9 min passed...1984 MiB / 4280 MiB / 1941 MiB / 416 MiB / 429 MiB ( 0 MiB / 280 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...2016 MiB / 4320 MiB / 1955 MiB / 416 MiB / 429 MiB ( 32 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
11 min passed...2016 MiB / 4400 MiB / 1969 MiB / 416 MiB / 429 MiB ( 0 MiB / 80 MiB / 14 MiB / 0 MiB / 0 MiB )
12 min passed...2016 MiB / 5080 MiB / 1980 MiB / 416 MiB / 430 MiB ( 0 MiB / 680 MiB / 11 MiB / 0 MiB / 1 MiB )
13 min passed...2016 MiB / 5360 MiB / 1993 MiB / 416 MiB / 430 MiB ( 0 MiB / 280 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...2048 MiB / 5720 MiB / 2005 MiB / 416 MiB / 430 MiB ( 32 MiB / 360 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...2048 MiB / 5720 MiB / 2017 MiB / 2368 MiB / 2445 MiB ( 0 MiB / 0 MiB / 12 MiB / 1952 MiB / 2015 MiB )
16 min passed...2048 MiB / 5760 MiB / 2028 MiB / 2464 MiB / 2479 MiB ( 0 MiB / 40 MiB / 11 MiB / 96 MiB / 34 MiB )
17 min passed...2048 MiB / 6080 MiB / 2038 MiB / 2464 MiB / 2480 MiB ( 0 MiB / 320 MiB / 10 MiB / 0 MiB / 1 MiB )
18 min passed...2080 MiB / 6160 MiB / 2051 MiB / 2464 MiB / 2480 MiB ( 32 MiB / 80 MiB / 13 MiB / 0 MiB / 0 MiB )
19 min passed...2080 MiB / 6560 MiB / 2064 MiB / 2464 MiB / 2481 MiB ( 0 MiB / 400 MiB / 13 MiB / 0 MiB / 1 MiB )
20 min passed...2080 MiB / 6960 MiB / 2075 MiB / 2464 MiB / 2481 MiB ( 0 MiB / 400 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...2112 MiB / 6960 MiB / 2085 MiB / 2464 MiB / 2481 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
22 min passed...2144 MiB / 7200 MiB / 2097 MiB / 2464 MiB / 2482 MiB ( 32 MiB / 240 MiB / 12 MiB / 0 MiB / 1 MiB )
23 min passed...2144 MiB / 7280 MiB / 2109 MiB / 2464 MiB / 2482 MiB ( 0 MiB / 80 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...2176 MiB / 7520 MiB / 2122 MiB / 2464 MiB / 2482 MiB ( 32 MiB / 240 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...2176 MiB / 7880 MiB / 2133 MiB / 2464 MiB / 2483 MiB ( 0 MiB / 360 MiB / 11 MiB / 0 MiB / 1 MiB )
26 min passed...2176 MiB / 7960 MiB / 2145 MiB / 2464 MiB / 2483 MiB ( 0 MiB / 80 MiB / 12 MiB / 0 MiB / 0 MiB )
27 min passed...2208 MiB / 8160 MiB / 2156 MiB / 2464 MiB / 2483 MiB ( 32 MiB / 200 MiB / 11 MiB / 0 MiB / 0 MiB )
28 min passed...2208 MiB / 8200 MiB / 2165 MiB / 2496 MiB / 2484 MiB ( 0 MiB / 40 MiB / 9 MiB / 32 MiB / 1 MiB )
29 min passed...2208 MiB / 8200 MiB / 2177 MiB / 2496 MiB / 2484 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
30 min passed...2208 MiB / 8360 MiB / 2201 MiB / 2496 MiB / 2485 MiB ( 0 MiB / 160 MiB / 24 MiB / 0 MiB / 1 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 6: async 4M sequential read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=646: Sat Aug  7 03:52:44 2021
  write: IOPS=234, BW=938MiB/s (983MB/s)(10.0GiB/10918msec); 0 zone resets
    slat (usec): min=96, max=1377, avg=207.99, stdev=95.50
    clat (msec): min=44, max=1783, avg=270.64, stdev=470.93
     lat (msec): min=44, max=1783, avg=270.85, stdev=470.93
    clat percentiles (msec):
     |  1.00th=[   70],  5.00th=[   74], 10.00th=[   77], 20.00th=[   80],
     | 30.00th=[   82], 40.00th=[   83], 50.00th=[   85], 60.00th=[   88],
     | 70.00th=[   92], 80.00th=[  103], 90.00th=[ 1284], 95.00th=[ 1519],
     | 99.00th=[ 1754], 99.50th=[ 1770], 99.90th=[ 1787], 99.95th=[ 1787],
     | 99.99th=[ 1787]
   bw (  MiB/s): min=   24, max= 2992, per=100.00%, avg=1534.14, stdev=1048.37, samples=13
   iops        : min=    6, max=  748, avg=383.46, stdev=262.20, samples=13
  lat (msec)   : 50=0.08%, 100=78.91%, 250=6.02%, 500=2.46%, 750=0.04%
  cpu          : usr=25.08%, sys=4.56%, ctx=2797, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=938MiB/s (983MB/s), 938MiB/s-938MiB/s (983MB/s-983MB/s), io=10.0GiB (10.7GB), run=10918-10918msec

Disk stats (read/write):
  sda: ios=0/9994, merge=0/100, ticks=0/1951039, in_queue=669024, util=42.33%
0 min passed...Guest Writes/Reads (Changes): 10240 MiB / 19 MiB ( 10240 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10242 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10243 MiB / 19 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10243 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10243 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10243 MiB / 20 MiB ( 0 MiB / 1 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1476MiB/s][r=369 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=868: Sat Aug  7 04:07:52 2021
  read: IOPS=348, BW=1393MiB/s (1460MB/s)(10.0GiB/7352msec)
    slat (usec): min=86, max=2044, avg=185.28, stdev=151.67
    clat (msec): min=37, max=349, avg=182.86, stdev=69.75
     lat (msec): min=38, max=349, avg=183.05, stdev=69.73
    clat percentiles (msec):
     |  1.00th=[   49],  5.00th=[   65], 10.00th=[   78], 20.00th=[  103],
     | 30.00th=[  144], 40.00th=[  180], 50.00th=[  199], 60.00th=[  215],
     | 70.00th=[  226], 80.00th=[  241], 90.00th=[  262], 95.00th=[  279],
     | 99.00th=[  338], 99.50th=[  347], 99.90th=[  351], 99.95th=[  351],
     | 99.99th=[  351]
   bw (  MiB/s): min=  984, max= 1624, per=99.05%, avg=1379.60, stdev=151.51, samples=14
   iops        : min=  246, max=  406, avg=344.86, stdev=37.87, samples=14
  lat (msec)   : 50=1.29%, 100=17.97%, 250=67.34%, 500=13.40%
  cpu          : usr=0.53%, sys=6.56%, ctx=2334, majf=0, minf=648
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1393MiB/s (1460MB/s), 1393MiB/s-1393MiB/s (1460MB/s-1460MB/s), io=10.0GiB (10.7GB), run=7352-7352msec

Disk stats (read/write):
  sda: ios=9929/2, merge=0/5, ticks=1343702/307, in_queue=1351004, util=99.97%
0 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10260 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10260 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10260 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10243 MiB / 10260 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10244 MiB / 10260 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10245 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Script ran 1820 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...13472 MiB / 14680 MiB / 13479 MiB / 288 MiB / 297 MiB ( 13472 MiB / 14680 MiB / 13479 MiB / 288 MiB / 297 MiB )
2 min passed...13472 MiB / 15000 MiB / 13492 MiB / 288 MiB / 298 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 1 MiB )
3 min passed...13472 MiB / 15200 MiB / 13504 MiB / 288 MiB / 298 MiB ( 0 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
4 min passed...13504 MiB / 15240 MiB / 13517 MiB / 288 MiB / 298 MiB ( 32 MiB / 40 MiB / 13 MiB / 0 MiB / 0 MiB )
5 min passed...13504 MiB / 15400 MiB / 13529 MiB / 288 MiB / 298 MiB ( 0 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
6 min passed...13504 MiB / 16000 MiB / 13543 MiB / 288 MiB / 298 MiB ( 0 MiB / 600 MiB / 14 MiB / 0 MiB / 0 MiB )
7 min passed...13536 MiB / 16080 MiB / 13557 MiB / 288 MiB / 299 MiB ( 32 MiB / 80 MiB / 14 MiB / 0 MiB / 1 MiB )
8 min passed...13568 MiB / 16120 MiB / 13572 MiB / 288 MiB / 299 MiB ( 32 MiB / 40 MiB / 15 MiB / 0 MiB / 0 MiB )
9 min passed...13600 MiB / 16120 MiB / 13583 MiB / 288 MiB / 299 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...13600 MiB / 16200 MiB / 13597 MiB / 288 MiB / 299 MiB ( 0 MiB / 80 MiB / 14 MiB / 0 MiB / 0 MiB )
11 min passed...13600 MiB / 16520 MiB / 13606 MiB / 288 MiB / 299 MiB ( 0 MiB / 320 MiB / 9 MiB / 0 MiB / 0 MiB )
12 min passed...13600 MiB / 16640 MiB / 13618 MiB / 288 MiB / 299 MiB ( 0 MiB / 120 MiB / 12 MiB / 0 MiB / 0 MiB )
13 min passed...13632 MiB / 16680 MiB / 13630 MiB / 288 MiB / 300 MiB ( 32 MiB / 40 MiB / 12 MiB / 0 MiB / 1 MiB )
14 min passed...13632 MiB / 16680 MiB / 13643 MiB / 288 MiB / 300 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
15 min passed...13632 MiB / 16680 MiB / 13655 MiB / 288 MiB / 300 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
16 min passed...13632 MiB / 17280 MiB / 13670 MiB / 11264 MiB / 11262 MiB ( 0 MiB / 600 MiB / 15 MiB / 10976 MiB / 10962 MiB )
17 min passed...13664 MiB / 17720 MiB / 13682 MiB / 11264 MiB / 11262 MiB ( 32 MiB / 440 MiB / 12 MiB / 0 MiB / 0 MiB )
18 min passed...13664 MiB / 17920 MiB / 13693 MiB / 11264 MiB / 11262 MiB ( 0 MiB / 200 MiB / 11 MiB / 0 MiB / 0 MiB )
19 min passed...13664 MiB / 18120 MiB / 13704 MiB / 11264 MiB / 11262 MiB ( 0 MiB / 200 MiB / 11 MiB / 0 MiB / 0 MiB )
20 min passed...13696 MiB / 18120 MiB / 13716 MiB / 11264 MiB / 11262 MiB ( 32 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
21 min passed...13728 MiB / 18360 MiB / 13728 MiB / 11264 MiB / 11262 MiB ( 32 MiB / 240 MiB / 12 MiB / 0 MiB / 0 MiB )
22 min passed...13728 MiB / 18520 MiB / 13739 MiB / 11264 MiB / 11263 MiB ( 0 MiB / 160 MiB / 11 MiB / 0 MiB / 1 MiB )
23 min passed...13760 MiB / 18800 MiB / 13755 MiB / 11264 MiB / 11263 MiB ( 32 MiB / 280 MiB / 16 MiB / 0 MiB / 0 MiB )
24 min passed...13760 MiB / 19120 MiB / 13766 MiB / 11264 MiB / 11263 MiB ( 0 MiB / 320 MiB / 11 MiB / 0 MiB / 0 MiB )
25 min passed...13760 MiB / 19120 MiB / 13779 MiB / 11264 MiB / 11263 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
26 min passed...13792 MiB / 19720 MiB / 13792 MiB / 11264 MiB / 11263 MiB ( 32 MiB / 600 MiB / 13 MiB / 0 MiB / 0 MiB )
27 min passed...13792 MiB / 19920 MiB / 13803 MiB / 11264 MiB / 11263 MiB ( 0 MiB / 200 MiB / 11 MiB / 0 MiB / 0 MiB )
28 min passed...13792 MiB / 19920 MiB / 13813 MiB / 11264 MiB / 11263 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
29 min passed...13792 MiB / 20000 MiB / 13824 MiB / 11264 MiB / 11264 MiB ( 0 MiB / 80 MiB / 11 MiB / 0 MiB / 1 MiB )
30 min passed...13824 MiB / 20040 MiB / 13836 MiB / 11264 MiB / 11264 MiB ( 32 MiB / 40 MiB / 12 MiB / 0 MiB / 0 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 7: async 16K random read/write but 50% compressible

Guest:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=227MiB/s][w=14.5k IOPS][eta 00m:00s]
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=649: Sat Aug  7 04:25:40 2021
  write: IOPS=14.9k, BW=234MiB/s (245MB/s)(1024MiB/4384msec); 0 zone resets
    slat (usec): min=7, max=7499, avg=31.11, stdev=87.91
    clat (usec): min=674, max=30518, avg=4236.01, stdev=4071.64
     lat (usec): min=687, max=30545, avg=4268.87, stdev=4080.15
    clat percentiles (usec):
     |  1.00th=[ 1057],  5.00th=[ 1352], 10.00th=[ 1598], 20.00th=[ 1958],
     | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3032], 60.00th=[ 3294],
     | 70.00th=[ 3589], 80.00th=[ 4293], 90.00th=[ 9241], 95.00th=[14615],
     | 99.00th=[20579], 99.50th=[22676], 99.90th=[26870], 99.95th=[28181],
     | 99.99th=[30016]
   bw (  KiB/s): min=70003, max=397568, per=100.00%, avg=244346.38, stdev=122462.78, samples=8
   iops        : min= 4375, max=24848, avg=15271.62, stdev=7653.96, samples=8
  lat (usec)   : 750=0.03%, 1000=0.62%
  lat (msec)   : 2=20.42%, 4=56.41%, 10=13.15%, 20=8.13%, 50=1.23%
  cpu          : usr=18.75%, sys=37.12%, ctx=40138, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=1024MiB (1074MB), run=4384-4384msec

Disk stats (read/write):
  sda: ios=1/61786, merge=0/158, ticks=3/188291, in_queue=189960, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1025 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 1 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1028 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=940: Sat Aug  7 04:40:43 2021
  read: IOPS=40.6k, BW=635MiB/s (666MB/s)(1024MiB/1613msec)
    slat (usec): min=4, max=809, avg=10.54, stdev= 7.96
    clat (usec): min=494, max=11861, avg=1558.07, stdev=244.31
     lat (usec): min=500, max=11883, avg=1569.85, stdev=244.23
    clat percentiles (usec):
     |  1.00th=[ 1057],  5.00th=[ 1237], 10.00th=[ 1319], 20.00th=[ 1401],
     | 30.00th=[ 1450], 40.00th=[ 1500], 50.00th=[ 1549], 60.00th=[ 1582],
     | 70.00th=[ 1631], 80.00th=[ 1696], 90.00th=[ 1795], 95.00th=[ 1926],
     | 99.00th=[ 2343], 99.50th=[ 2540], 99.90th=[ 3425], 99.95th=[ 3916],
     | 99.99th=[ 5276]
   bw (  KiB/s): min=635264, max=656384, per=99.67%, avg=647904.00, stdev=11157.63, samples=3
   iops        : min=39704, max=41024, avg=40494.00, stdev=697.35, samples=3
  lat (usec)   : 500=0.01%, 750=0.04%, 1000=0.54%
  lat (msec)   : 2=95.83%, 4=3.55%, 10=0.04%, 20=0.01%
  cpu          : usr=29.84%, sys=50.81%, ctx=8236, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=635MiB/s (666MB/s), 635MiB/s-635MiB/s (666MB/s-666MB/s), io=1024MiB (1074MB), run=1613-1613msec

Disk stats (read/write):
  sda: ios=56485/0, merge=0/0, ticks=85401/0, in_queue=85424, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1030 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1808 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1888 MiB / 2320 MiB / 1909 MiB / 2816 MiB / 2775 MiB ( 1888 MiB / 2320 MiB / 1909 MiB / 2816 MiB / 2775 MiB )
2 min passed...1920 MiB / 2480 MiB / 1922 MiB / 2816 MiB / 2776 MiB ( 32 MiB / 160 MiB / 13 MiB / 0 MiB / 1 MiB )
3 min passed...1920 MiB / 2560 MiB / 1935 MiB / 2816 MiB / 2776 MiB ( 0 MiB / 80 MiB / 13 MiB / 0 MiB / 0 MiB )
4 min passed...1952 MiB / 2600 MiB / 1949 MiB / 2816 MiB / 2776 MiB ( 32 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
5 min passed...1952 MiB / 2960 MiB / 1958 MiB / 2816 MiB / 2776 MiB ( 0 MiB / 360 MiB / 9 MiB / 0 MiB / 0 MiB )
6 min passed...1952 MiB / 3240 MiB / 1970 MiB / 2816 MiB / 2776 MiB ( 0 MiB / 280 MiB / 12 MiB / 0 MiB / 0 MiB )
7 min passed...1952 MiB / 3480 MiB / 1980 MiB / 2816 MiB / 2776 MiB ( 0 MiB / 240 MiB / 10 MiB / 0 MiB / 0 MiB )
8 min passed...1984 MiB / 3480 MiB / 1996 MiB / 2816 MiB / 2777 MiB ( 32 MiB / 0 MiB / 16 MiB / 0 MiB / 1 MiB )
9 min passed...1984 MiB / 4200 MiB / 2007 MiB / 2816 MiB / 2777 MiB ( 0 MiB / 720 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...1984 MiB / 4600 MiB / 2018 MiB / 2816 MiB / 2777 MiB ( 0 MiB / 400 MiB / 11 MiB / 0 MiB / 0 MiB )
11 min passed...1984 MiB / 4800 MiB / 2029 MiB / 2816 MiB / 2777 MiB ( 0 MiB / 200 MiB / 11 MiB / 0 MiB / 0 MiB )
12 min passed...2016 MiB / 5120 MiB / 2040 MiB / 2816 MiB / 2777 MiB ( 32 MiB / 320 MiB / 11 MiB / 0 MiB / 0 MiB )
13 min passed...2016 MiB / 5120 MiB / 2053 MiB / 2816 MiB / 2777 MiB ( 0 MiB / 0 MiB / 13 MiB / 0 MiB / 0 MiB )
14 min passed...2016 MiB / 5720 MiB / 2063 MiB / 2816 MiB / 2778 MiB ( 0 MiB / 600 MiB / 10 MiB / 0 MiB / 1 MiB )
15 min passed...2080 MiB / 5960 MiB / 2075 MiB / 2816 MiB / 2778 MiB ( 64 MiB / 240 MiB / 12 MiB / 0 MiB / 0 MiB )
16 min passed...2080 MiB / 6040 MiB / 2086 MiB / 4096 MiB / 4068 MiB ( 0 MiB / 80 MiB / 11 MiB / 1280 MiB / 1290 MiB )
17 min passed...2080 MiB / 6360 MiB / 2099 MiB / 4096 MiB / 4068 MiB ( 0 MiB / 320 MiB / 13 MiB / 0 MiB / 0 MiB )
18 min passed...2112 MiB / 6360 MiB / 2109 MiB / 4096 MiB / 4068 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...2112 MiB / 6920 MiB / 2121 MiB / 4096 MiB / 4068 MiB ( 0 MiB / 560 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...2112 MiB / 7200 MiB / 2134 MiB / 4096 MiB / 4068 MiB ( 0 MiB / 280 MiB / 13 MiB / 0 MiB / 0 MiB )
21 min passed...2144 MiB / 7480 MiB / 2146 MiB / 4096 MiB / 4068 MiB ( 32 MiB / 280 MiB / 12 MiB / 0 MiB / 0 MiB )
22 min passed...2144 MiB / 7520 MiB / 2160 MiB / 4096 MiB / 4068 MiB ( 0 MiB / 40 MiB / 14 MiB / 0 MiB / 0 MiB )
23 min passed...2144 MiB / 7560 MiB / 2172 MiB / 4096 MiB / 4069 MiB ( 0 MiB / 40 MiB / 12 MiB / 0 MiB / 1 MiB )
24 min passed...2144 MiB / 7640 MiB / 2186 MiB / 4096 MiB / 4069 MiB ( 0 MiB / 80 MiB / 14 MiB / 0 MiB / 0 MiB )
25 min passed...2176 MiB / 7840 MiB / 2199 MiB / 4096 MiB / 4069 MiB ( 32 MiB / 200 MiB / 13 MiB / 0 MiB / 0 MiB )
26 min passed...2176 MiB / 8120 MiB / 2211 MiB / 4096 MiB / 4069 MiB ( 0 MiB / 280 MiB / 12 MiB / 0 MiB / 0 MiB )
27 min passed...2208 MiB / 8400 MiB / 2226 MiB / 4096 MiB / 4069 MiB ( 32 MiB / 280 MiB / 15 MiB / 0 MiB / 0 MiB )
28 min passed...2240 MiB / 8400 MiB / 2238 MiB / 4096 MiB / 4069 MiB ( 32 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
29 min passed...2240 MiB / 8800 MiB / 2254 MiB / 4096 MiB / 4070 MiB ( 0 MiB / 400 MiB / 16 MiB / 0 MiB / 1 MiB )
30 min passed...2272 MiB / 9320 MiB / 2271 MiB / 4096 MiB / 4070 MiB ( 32 MiB / 520 MiB / 17 MiB / 0 MiB / 0 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 8: sync 32K random read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=11.8MiB/s][w=376 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=658: Sat Aug  7 06:05:17 2021
  write: IOPS=397, BW=12.4MiB/s (13.0MB/s)(1024MiB/82350msec); 0 zone resets
    clat (usec): min=1170, max=23761, avg=2486.18, stdev=809.50
     lat (usec): min=1172, max=23764, avg=2488.50, stdev=809.51
    clat percentiles (usec):
     |  1.00th=[ 1631],  5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 1975],
     | 30.00th=[ 2089], 40.00th=[ 2212], 50.00th=[ 2376], 60.00th=[ 2573],
     | 70.00th=[ 2737], 80.00th=[ 2868], 90.00th=[ 3064], 95.00th=[ 3261],
     | 99.00th=[ 5014], 99.50th=[ 5735], 99.90th=[12518], 99.95th=[14746],
     | 99.99th=[21890]
   bw (  KiB/s): min=10112, max=14464, per=99.95%, avg=12726.55, stdev=770.16, samples=164
   iops        : min=  316, max=  452, avg=397.70, stdev=24.07, samples=164
  lat (msec)   : 2=21.70%, 4=75.64%, 10=2.46%, 20=0.18%, 50=0.02%
  cpu          : usr=1.53%, sys=5.68%, ctx=68044, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=12.4MiB/s (13.0MB/s), 12.4MiB/s-12.4MiB/s (13.0MB/s-13.0MB/s), io=1024MiB (1074MB), run=82350-82350msec

Disk stats (read/write):
  sda: ios=0/98177, merge=0/66129, ticks=0/73443, in_queue=82192, util=99.72%
0 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 1540 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1540 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 1 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1541 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1542 MiB / 19 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1542 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1542 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][95.5%][r=54.2MiB/s][r=1736 IOPS][eta 00m:01s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=868: Sat Aug  7 06:20:39 2021
  read: IOPS=1569, BW=49.0MiB/s (51.4MB/s)(1024MiB/20884msec)
    clat (usec): min=269, max=3555, avg=626.24, stdev=182.72
     lat (usec): min=271, max=3557, avg=628.19, stdev=182.77
    clat percentiles (usec):
     |  1.00th=[  449],  5.00th=[  482], 10.00th=[  506], 20.00th=[  537],
     | 30.00th=[  562], 40.00th=[  586], 50.00th=[  611], 60.00th=[  635],
     | 70.00th=[  660], 80.00th=[  693], 90.00th=[  725], 95.00th=[  750],
     | 99.00th=[  971], 99.50th=[ 2057], 99.90th=[ 3130], 99.95th=[ 3228],
     | 99.99th=[ 3392]
   bw (  KiB/s): min=44352, max=57536, per=99.75%, avg=50085.07, stdev=3308.15, samples=41
   iops        : min= 1386, max= 1798, avg=1565.15, stdev=103.36, samples=41
  lat (usec)   : 500=8.76%, 750=86.22%, 1000=4.05%
  lat (msec)   : 2=0.45%, 4=0.52%
  cpu          : usr=2.93%, sys=9.21%, ctx=32775, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=49.0MiB/s (51.4MB/s), 49.0MiB/s-49.0MiB/s (51.4MB/s-51.4MB/s), io=1024MiB (1074MB), run=20884-20884msec

Disk stats (read/write):
  sda: ios=32331/7, merge=0/6, ticks=18914/8, in_queue=20760, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1542 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1045 MiB ( 1 MiB / 2 MiB )
6 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1045 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1544 MiB / 1045 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1545 MiB / 1045 MiB ( 1 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1545 MiB / 1045 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1545 MiB / 1045 MiB ( 0 MiB / 0 MiB )
Script ran 1905 seconds or 31 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5024 MiB / 5800 MiB / 5056 MiB / 192 MiB / 180 MiB ( 5024 MiB / 5800 MiB / 5056 MiB / 192 MiB / 180 MiB )
2 min passed...7232 MiB / 8120 MiB / 7248 MiB / 256 MiB / 252 MiB ( 2208 MiB / 2320 MiB / 2192 MiB / 64 MiB / 72 MiB )
3 min passed...7264 MiB / 8400 MiB / 7263 MiB / 256 MiB / 252 MiB ( 32 MiB / 280 MiB / 15 MiB / 0 MiB / 0 MiB )
4 min passed...7264 MiB / 9000 MiB / 7272 MiB / 256 MiB / 252 MiB ( 0 MiB / 600 MiB / 9 MiB / 0 MiB / 0 MiB )
5 min passed...7264 MiB / 9440 MiB / 7284 MiB / 256 MiB / 252 MiB ( 0 MiB / 440 MiB / 12 MiB / 0 MiB / 0 MiB )
6 min passed...7264 MiB / 9680 MiB / 7296 MiB / 256 MiB / 252 MiB ( 0 MiB / 240 MiB / 12 MiB / 0 MiB / 0 MiB )
7 min passed...7296 MiB / 9760 MiB / 7310 MiB / 256 MiB / 252 MiB ( 32 MiB / 80 MiB / 14 MiB / 0 MiB / 0 MiB )
8 min passed...7296 MiB / 9920 MiB / 7320 MiB / 288 MiB / 253 MiB ( 0 MiB / 160 MiB / 10 MiB / 32 MiB / 1 MiB )
9 min passed...7296 MiB / 10640 MiB / 7329 MiB / 288 MiB / 253 MiB ( 0 MiB / 720 MiB / 9 MiB / 0 MiB / 0 MiB )
10 min passed...7328 MiB / 10760 MiB / 7342 MiB / 288 MiB / 253 MiB ( 32 MiB / 120 MiB / 13 MiB / 0 MiB / 0 MiB )
11 min passed...7360 MiB / 10920 MiB / 7354 MiB / 288 MiB / 253 MiB ( 32 MiB / 160 MiB / 12 MiB / 0 MiB / 0 MiB )
12 min passed...7360 MiB / 10920 MiB / 7368 MiB / 288 MiB / 253 MiB ( 0 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
13 min passed...7392 MiB / 10960 MiB / 7378 MiB / 288 MiB / 253 MiB ( 32 MiB / 40 MiB / 10 MiB / 0 MiB / 0 MiB )
14 min passed...7392 MiB / 11160 MiB / 7391 MiB / 288 MiB / 254 MiB ( 0 MiB / 200 MiB / 13 MiB / 0 MiB / 1 MiB )
15 min passed...7392 MiB / 11480 MiB / 7403 MiB / 288 MiB / 254 MiB ( 0 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
16 min passed...7392 MiB / 11760 MiB / 7416 MiB / 288 MiB / 254 MiB ( 0 MiB / 280 MiB / 13 MiB / 0 MiB / 0 MiB )
17 min passed...7424 MiB / 11960 MiB / 7428 MiB / 1312 MiB / 1278 MiB ( 32 MiB / 200 MiB / 12 MiB / 1024 MiB / 1024 MiB )
18 min passed...7424 MiB / 12000 MiB / 7439 MiB / 1312 MiB / 1278 MiB ( 0 MiB / 40 MiB / 11 MiB / 0 MiB / 0 MiB )
19 min passed...7424 MiB / 12520 MiB / 7451 MiB / 1312 MiB / 1278 MiB ( 0 MiB / 520 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...7424 MiB / 13000 MiB / 7462 MiB / 1312 MiB / 1279 MiB ( 0 MiB / 480 MiB / 11 MiB / 0 MiB / 1 MiB )
21 min passed...7456 MiB / 13320 MiB / 7474 MiB / 1312 MiB / 1280 MiB ( 32 MiB / 320 MiB / 12 MiB / 0 MiB / 1 MiB )
22 min passed...7456 MiB / 13640 MiB / 7488 MiB / 1312 MiB / 1281 MiB ( 0 MiB / 320 MiB / 14 MiB / 0 MiB / 1 MiB )
23 min passed...7488 MiB / 13680 MiB / 7501 MiB / 1312 MiB / 1281 MiB ( 32 MiB / 40 MiB / 13 MiB / 0 MiB / 0 MiB )
24 min passed...7520 MiB / 14240 MiB / 7514 MiB / 1312 MiB / 1281 MiB ( 32 MiB / 560 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...7520 MiB / 14720 MiB / 7526 MiB / 1312 MiB / 1282 MiB ( 0 MiB / 480 MiB / 12 MiB / 0 MiB / 1 MiB )
26 min passed...7552 MiB / 15000 MiB / 7536 MiB / 1312 MiB / 1282 MiB ( 32 MiB / 280 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...7552 MiB / 15360 MiB / 7547 MiB / 1312 MiB / 1282 MiB ( 0 MiB / 360 MiB / 11 MiB / 0 MiB / 0 MiB )
28 min passed...7552 MiB / 15400 MiB / 7558 MiB / 1312 MiB / 1282 MiB ( 0 MiB / 40 MiB / 11 MiB / 0 MiB / 0 MiB )
29 min passed...7552 MiB / 15640 MiB / 7571 MiB / 1312 MiB / 1282 MiB ( 0 MiB / 240 MiB / 13 MiB / 0 MiB / 0 MiB )
30 min passed...7584 MiB / 15760 MiB / 7582 MiB / 1312 MiB / 1282 MiB ( 32 MiB / 120 MiB / 11 MiB / 0 MiB / 0 MiB )
31 min passed...7584 MiB / 15760 MiB / 7596 MiB / 1312 MiB / 1283 MiB ( 0 MiB / 0 MiB / 14 MiB / 0 MiB / 1 MiB )

 

[Dunuin]

Round 2 - Run 1 - Test 9: async 32K random read/write

Guest:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=674: Sat Aug  7 06:40:36 2021
  write: IOPS=23.5k, BW=733MiB/s (769MB/s)(1024MiB/1397msec); 0 zone resets
    slat (usec): min=9, max=987, avg=21.75, stdev=13.87
    clat (usec): min=109, max=10903, avg=2683.42, stdev=520.51
     lat (usec): min=126, max=10915, avg=2706.48, stdev=522.27
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2376], 10.00th=[ 2409], 20.00th=[ 2442],
     | 30.00th=[ 2474], 40.00th=[ 2507], 50.00th=[ 2540], 60.00th=[ 2573],
     | 70.00th=[ 2638], 80.00th=[ 2737], 90.00th=[ 3064], 95.00th=[ 3556],
     | 99.00th=[ 4883], 99.50th=[ 5735], 99.90th=[ 7701], 99.95th=[ 7898],
     | 99.99th=[ 8979]
   bw (  KiB/s): min=787840, max=794432, per=100.00%, avg=791136.00, stdev=4661.25, samples=2
   iops        : min=24620, max=24826, avg=24723.00, stdev=145.66, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.19%, 4=96.67%, 10=3.07%, 20=0.01%
  cpu          : usr=27.01%, sys=53.87%, ctx=28936, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=733MiB/s (769MB/s), 733MiB/s-733MiB/s (769MB/s-769MB/s), io=1024MiB (1074MB), run=1397-1397msec

Disk stats (read/write):
  sda: ios=0/32536, merge=0/0, ticks=0/5871, in_queue=6232, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 17 MiB ( 1024 MiB / 17 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 17 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 17 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 17 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 17 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1027 MiB / 17 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1028 MiB / 18 MiB ( 1 MiB / 1 MiB )
13 min passed...Guest Writes/Reads (Changes): 1028 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1028 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1028 MiB / 18 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=891: Sat Aug  7 06:55:38 2021
  read: IOPS=33.8k, BW=1057MiB/s (1108MB/s)(1024MiB/969msec)
    slat (usec): min=4, max=503, avg=12.41, stdev=16.57
    clat (usec): min=839, max=17710, avg=1870.74, stdev=549.43
     lat (usec): min=846, max=17732, avg=1884.41, stdev=550.10
    clat percentiles (usec):
     |  1.00th=[ 1237],  5.00th=[ 1434], 10.00th=[ 1549], 20.00th=[ 1631],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1778], 60.00th=[ 1827],
     | 70.00th=[ 1893], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2671],
     | 99.00th=[ 3195], 99.50th=[ 3621], 99.90th=[10814], 99.95th=[10945],
     | 99.99th=[11076]
   bw (  KiB/s): min=1022144, max=1022144, per=94.46%, avg=1022144.00, stdev= 0.00, samples=1
   iops        : min=31942, max=31942, avg=31942.00, stdev= 0.00, samples=1
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=81.24%, 4=18.32%, 10=0.19%, 20=0.20%
  cpu          : usr=31.51%, sys=44.11%, ctx=4163, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1057MiB/s (1108MB/s), 1057MiB/s-1057MiB/s (1108MB/s-1108MB/s), io=1024MiB (1074MB), run=969-969msec

Disk stats (read/write):
  sda: ios=30233/2, merge=0/3, ticks=51547/2, in_queue=52224, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1028 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1029 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1030 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1031 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 1804 seconds or 30 minutes.

Host:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...1568 MiB / 2000 MiB / 1577 MiB / 0 MiB / 14 MiB ( 1568 MiB / 2000 MiB / 1577 MiB / 0 MiB / 14 MiB )
2 min passed...1568 MiB / 2000 MiB / 1597 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 20 MiB / 0 MiB / 0 MiB )
3 min passed...1600 MiB / 2520 MiB / 1618 MiB / 0 MiB / 15 MiB ( 32 MiB / 520 MiB / 21 MiB / 0 MiB / 1 MiB )
4 min passed...1600 MiB / 3080 MiB / 1637 MiB / 0 MiB / 16 MiB ( 0 MiB / 560 MiB / 19 MiB / 0 MiB / 1 MiB )
5 min passed...1632 MiB / 3320 MiB / 1658 MiB / 0 MiB / 16 MiB ( 32 MiB / 240 MiB / 21 MiB / 0 MiB / 0 MiB )
6 min passed...1632 MiB / 3400 MiB / 1681 MiB / 0 MiB / 17 MiB ( 0 MiB / 80 MiB / 23 MiB / 0 MiB / 1 MiB )
7 min passed...1664 MiB / 3400 MiB / 1699 MiB / 0 MiB / 17 MiB ( 32 MiB / 0 MiB / 18 MiB / 0 MiB / 0 MiB )
8 min passed...1664 MiB / 3920 MiB / 1720 MiB / 32 MiB / 18 MiB ( 0 MiB / 520 MiB / 21 MiB / 32 MiB / 1 MiB )
9 min passed...1728 MiB / 4440 MiB / 1742 MiB / 32 MiB / 18 MiB ( 64 MiB / 520 MiB / 22 MiB / 0 MiB / 0 MiB )
10 min passed...1760 MiB / 4600 MiB / 1761 MiB / 32 MiB / 19 MiB ( 32 MiB / 160 MiB / 19 MiB / 0 MiB / 1 MiB )
11 min passed...1760 MiB / 4960 MiB / 1782 MiB / 32 MiB / 19 MiB ( 0 MiB / 360 MiB / 21 MiB / 0 MiB / 0 MiB )
12 min passed...1760 MiB / 5040 MiB / 1801 MiB / 32 MiB / 20 MiB ( 0 MiB / 80 MiB / 19 MiB / 0 MiB / 1 MiB )
13 min passed...1792 MiB / 5400 MiB / 1820 MiB / 32 MiB / 20 MiB ( 32 MiB / 360 MiB / 19 MiB / 0 MiB / 0 MiB )
14 min passed...1792 MiB / 5760 MiB / 1840 MiB / 32 MiB / 21 MiB ( 0 MiB / 360 MiB / 20 MiB / 0 MiB / 1 MiB )
15 min passed...1824 MiB / 5800 MiB / 1856 MiB / 1024 MiB / 1056 MiB ( 32 MiB / 40 MiB / 16 MiB / 992 MiB / 1035 MiB )
16 min passed...1824 MiB / 6160 MiB / 1868 MiB / 1024 MiB / 1056 MiB ( 0 MiB / 360 MiB / 12 MiB / 0 MiB / 0 MiB )
17 min passed...1824 MiB / 6240 MiB / 1879 MiB / 1024 MiB / 1056 MiB ( 0 MiB / 80 MiB / 11 MiB / 0 MiB / 0 MiB )
18 min passed...1856 MiB / 6480 MiB / 1889 MiB / 1024 MiB / 1056 MiB ( 32 MiB / 240 MiB / 10 MiB / 0 MiB / 0 MiB )
19 min passed...1888 MiB / 6920 MiB / 1901 MiB / 1024 MiB / 1056 MiB ( 32 MiB / 440 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...1888 MiB / 7240 MiB / 1915 MiB / 1024 MiB / 1056 MiB ( 0 MiB / 320 MiB / 14 MiB / 0 MiB / 0 MiB )
21 min passed...1920 MiB / 7480 MiB / 1927 MiB / 1024 MiB / 1057 MiB ( 32 MiB / 240 MiB / 12 MiB / 0 MiB / 1 MiB )
22 min passed...1920 MiB / 7680 MiB / 1939 MiB / 1024 MiB / 1057 MiB ( 0 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
23 min passed...1920 MiB / 7880 MiB / 1951 MiB / 1024 MiB / 1057 MiB ( 0 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
24 min passed...1920 MiB / 8560 MiB / 1963 MiB / 1024 MiB / 1057 MiB ( 0 MiB / 680 MiB / 12 MiB / 0 MiB / 0 MiB )
25 min passed...1952 MiB / 8960 MiB / 1977 MiB / 1024 MiB / 1057 MiB ( 32 MiB / 400 MiB / 14 MiB / 0 MiB / 0 MiB )
26 min passed...1952 MiB / 9120 MiB / 1987 MiB / 1024 MiB / 1057 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
27 min passed...1952 MiB / 9120 MiB / 1999 MiB / 1024 MiB / 1058 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
28 min passed...1984 MiB / 9240 MiB / 2011 MiB / 1024 MiB / 1058 MiB ( 32 MiB / 120 MiB / 12 MiB / 0 MiB / 0 MiB )
29 min passed...1984 MiB / 9600 MiB / 2023 MiB / 1024 MiB / 1058 MiB ( 0 MiB / 360 MiB / 12 MiB / 0 MiB / 0 MiB )
30 min passed...2016 MiB / 9840 MiB / 2048 MiB / 1024 MiB / 1059 MiB ( 32 MiB / 240 MiB / 25 MiB / 0 MiB / 1 MiB )