how to best benchmark SSDs?

Dunuin · Aug 7, 2021

Round 2 - Run 1 - Test 0: just idleing for 1 hour

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_0.sh
0 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 1 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 1 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
16 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
17 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
18 min passed...Guest Writes/Reads (Changes): 3 MiB / 0 MiB ( 0 MiB / 0 MiB )
19 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 1 MiB / 0 MiB )
20 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
21 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
22 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
23 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
24 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
25 min passed...Guest Writes/Reads (Changes): 4 MiB / 0 MiB ( 0 MiB / 0 MiB )
26 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 1 MiB / 0 MiB )
27 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
28 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
29 min passed...Guest Writes/Reads (Changes): 5 MiB / 0 MiB ( 0 MiB / 0 MiB )
30 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 1 MiB / 0 MiB )
31 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
32 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
33 min passed...Guest Writes/Reads (Changes): 6 MiB / 0 MiB ( 0 MiB / 0 MiB )
34 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 1 MiB / 0 MiB )
35 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
36 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
37 min passed...Guest Writes/Reads (Changes): 7 MiB / 0 MiB ( 0 MiB / 0 MiB )
38 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 1 MiB / 0 MiB )
39 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
40 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
41 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
42 min passed...Guest Writes/Reads (Changes): 8 MiB / 0 MiB ( 0 MiB / 0 MiB )
43 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 1 MiB / 0 MiB )
44 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
45 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
46 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
47 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
48 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
49 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
50 min passed...Guest Writes/Reads (Changes): 9 MiB / 0 MiB ( 0 MiB / 0 MiB )
51 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 1 MiB / 0 MiB )
52 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
53 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
54 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
55 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
56 min passed...Guest Writes/Reads (Changes): 10 MiB / 0 MiB ( 0 MiB / 0 MiB )
57 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 1 MiB / 0 MiB )
58 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
59 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
60 min passed...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 11 MiB / 0 MiB ( 0 MiB / 0 MiB )
Script ran 3601 seconds or 60 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_5.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...32 MiB / 360 MiB / 21 MiB / 32 MiB / 1 MiB ( 32 MiB / 360 MiB / 21 MiB / 32 MiB / 1 MiB )
2 min passed...32 MiB / 640 MiB / 41 MiB / 32 MiB / 1 MiB ( 0 MiB / 280 MiB / 20 MiB / 0 MiB / 0 MiB )
3 min passed...32 MiB / 1000 MiB / 61 MiB / 32 MiB / 2 MiB ( 0 MiB / 360 MiB / 20 MiB / 0 MiB / 1 MiB )
4 min passed...64 MiB / 1120 MiB / 80 MiB / 32 MiB / 2 MiB ( 32 MiB / 120 MiB / 19 MiB / 0 MiB / 0 MiB )
5 min passed...96 MiB / 1480 MiB / 100 MiB / 32 MiB / 3 MiB ( 32 MiB / 360 MiB / 20 MiB / 0 MiB / 1 MiB )
6 min passed...128 MiB / 2080 MiB / 121 MiB / 32 MiB / 3 MiB ( 32 MiB / 600 MiB / 21 MiB / 0 MiB / 0 MiB )
7 min passed...160 MiB / 2440 MiB / 141 MiB / 32 MiB / 4 MiB ( 32 MiB / 360 MiB / 20 MiB / 0 MiB / 1 MiB )
8 min passed...160 MiB / 2800 MiB / 159 MiB / 32 MiB / 4 MiB ( 0 MiB / 360 MiB / 18 MiB / 0 MiB / 0 MiB )
9 min passed...192 MiB / 2920 MiB / 178 MiB / 32 MiB / 4 MiB ( 32 MiB / 120 MiB / 19 MiB / 0 MiB / 0 MiB )
10 min passed...192 MiB / 3120 MiB / 197 MiB / 32 MiB / 5 MiB ( 0 MiB / 200 MiB / 19 MiB / 0 MiB / 1 MiB )
11 min passed...192 MiB / 3840 MiB / 216 MiB / 32 MiB / 5 MiB ( 0 MiB / 720 MiB / 19 MiB / 0 MiB / 0 MiB )
12 min passed...224 MiB / 4200 MiB / 234 MiB / 32 MiB / 6 MiB ( 32 MiB / 360 MiB / 18 MiB / 0 MiB / 1 MiB )
13 min passed...224 MiB / 4360 MiB / 243 MiB / 32 MiB / 6 MiB ( 0 MiB / 160 MiB / 9 MiB / 0 MiB / 0 MiB )
14 min passed...224 MiB / 4360 MiB / 253 MiB / 32 MiB / 6 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
15 min passed...288 MiB / 4560 MiB / 267 MiB / 32 MiB / 6 MiB ( 64 MiB / 200 MiB / 14 MiB / 0 MiB / 0 MiB )
16 min passed...288 MiB / 4880 MiB / 279 MiB / 32 MiB / 6 MiB ( 0 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
17 min passed...320 MiB / 5000 MiB / 291 MiB / 32 MiB / 6 MiB ( 32 MiB / 120 MiB / 12 MiB / 0 MiB / 0 MiB )
18 min passed...320 MiB / 5280 MiB / 300 MiB / 32 MiB / 7 MiB ( 0 MiB / 280 MiB / 9 MiB / 0 MiB / 1 MiB )
19 min passed...320 MiB / 5480 MiB / 312 MiB / 32 MiB / 7 MiB ( 0 MiB / 200 MiB / 12 MiB / 0 MiB / 0 MiB )
20 min passed...320 MiB / 5560 MiB / 324 MiB / 32 MiB / 7 MiB ( 0 MiB / 80 MiB / 12 MiB / 0 MiB / 0 MiB )
21 min passed...320 MiB / 6040 MiB / 337 MiB / 32 MiB / 7 MiB ( 0 MiB / 480 MiB / 13 MiB / 0 MiB / 0 MiB )
22 min passed...352 MiB / 6400 MiB / 351 MiB / 32 MiB / 7 MiB ( 32 MiB / 360 MiB / 14 MiB / 0 MiB / 0 MiB )
23 min passed...352 MiB / 6600 MiB / 364 MiB / 32 MiB / 7 MiB ( 0 MiB / 200 MiB / 13 MiB / 0 MiB / 0 MiB )
24 min passed...352 MiB / 6880 MiB / 376 MiB / 32 MiB / 7 MiB ( 0 MiB / 280 MiB / 12 MiB / 0 MiB / 0 MiB )
25 min passed...384 MiB / 6920 MiB / 393 MiB / 32 MiB / 8 MiB ( 32 MiB / 40 MiB / 17 MiB / 0 MiB / 1 MiB )
26 min passed...384 MiB / 7640 MiB / 414 MiB / 32 MiB / 8 MiB ( 0 MiB / 720 MiB / 21 MiB / 0 MiB / 0 MiB )
27 min passed...448 MiB / 8080 MiB / 433 MiB / 32 MiB / 8 MiB ( 64 MiB / 440 MiB / 19 MiB / 0 MiB / 0 MiB )
28 min passed...480 MiB / 8360 MiB / 454 MiB / 32 MiB / 9 MiB ( 32 MiB / 280 MiB / 21 MiB / 0 MiB / 1 MiB )
29 min passed...480 MiB / 8640 MiB / 472 MiB / 32 MiB / 9 MiB ( 0 MiB / 280 MiB / 18 MiB / 0 MiB / 0 MiB )
30 min passed...480 MiB / 8640 MiB / 491 MiB / 32 MiB / 10 MiB ( 0 MiB / 0 MiB / 19 MiB / 0 MiB / 1 MiB )
31 min passed...512 MiB / 9360 MiB / 510 MiB / 32 MiB / 10 MiB ( 32 MiB / 720 MiB / 19 MiB / 0 MiB / 0 MiB )
32 min passed...512 MiB / 9720 MiB / 530 MiB / 32 MiB / 11 MiB ( 0 MiB / 360 MiB / 20 MiB / 0 MiB / 1 MiB )
33 min passed...544 MiB / 9800 MiB / 550 MiB / 32 MiB / 11 MiB ( 32 MiB / 80 MiB / 20 MiB / 0 MiB / 0 MiB )
34 min passed...544 MiB / 9840 MiB / 568 MiB / 32 MiB / 11 MiB ( 0 MiB / 40 MiB / 18 MiB / 0 MiB / 0 MiB )
35 min passed...608 MiB / 9840 MiB / 587 MiB / 32 MiB / 12 MiB ( 64 MiB / 0 MiB / 19 MiB / 0 MiB / 1 MiB )
36 min passed...608 MiB / 10080 MiB / 606 MiB / 32 MiB / 12 MiB ( 0 MiB / 240 MiB / 19 MiB / 0 MiB / 0 MiB )
37 min passed...640 MiB / 10280 MiB / 625 MiB / 32 MiB / 13 MiB ( 32 MiB / 200 MiB / 19 MiB / 0 MiB / 1 MiB )
38 min passed...640 MiB / 10600 MiB / 644 MiB / 32 MiB / 13 MiB ( 0 MiB / 320 MiB / 19 MiB / 0 MiB / 0 MiB )
39 min passed...672 MiB / 11040 MiB / 664 MiB / 32 MiB / 14 MiB ( 32 MiB / 440 MiB / 20 MiB / 0 MiB / 1 MiB )
40 min passed...672 MiB / 11040 MiB / 678 MiB / 32 MiB / 14 MiB ( 0 MiB / 0 MiB / 14 MiB / 0 MiB / 0 MiB )
41 min passed...672 MiB / 11480 MiB / 690 MiB / 32 MiB / 14 MiB ( 0 MiB / 440 MiB / 12 MiB / 0 MiB / 0 MiB )
42 min passed...672 MiB / 12000 MiB / 703 MiB / 32 MiB / 14 MiB ( 0 MiB / 520 MiB / 13 MiB / 0 MiB / 0 MiB )
43 min passed...704 MiB / 12280 MiB / 715 MiB / 32 MiB / 15 MiB ( 32 MiB / 280 MiB / 12 MiB / 0 MiB / 1 MiB )
44 min passed...704 MiB / 12680 MiB / 725 MiB / 32 MiB / 15 MiB ( 0 MiB / 400 MiB / 10 MiB / 0 MiB / 0 MiB )
45 min passed...736 MiB / 12760 MiB / 738 MiB / 32 MiB / 15 MiB ( 32 MiB / 80 MiB / 13 MiB / 0 MiB / 0 MiB )
46 min passed...768 MiB / 13160 MiB / 746 MiB / 32 MiB / 15 MiB ( 32 MiB / 400 MiB / 8 MiB / 0 MiB / 0 MiB )
47 min passed...768 MiB / 13680 MiB / 759 MiB / 32 MiB / 15 MiB ( 0 MiB / 520 MiB / 13 MiB / 0 MiB / 0 MiB )
48 min passed...768 MiB / 14000 MiB / 769 MiB / 32 MiB / 15 MiB ( 0 MiB / 320 MiB / 10 MiB / 0 MiB / 0 MiB )
49 min passed...800 MiB / 14280 MiB / 783 MiB / 32 MiB / 15 MiB ( 32 MiB / 280 MiB / 14 MiB / 0 MiB / 0 MiB )
50 min passed...800 MiB / 14480 MiB / 796 MiB / 32 MiB / 16 MiB ( 0 MiB / 200 MiB / 13 MiB / 0 MiB / 1 MiB )
51 min passed...800 MiB / 14760 MiB / 812 MiB / 32 MiB / 16 MiB ( 0 MiB / 280 MiB / 16 MiB / 0 MiB / 0 MiB )
52 min passed...832 MiB / 15040 MiB / 826 MiB / 32 MiB / 16 MiB ( 32 MiB / 280 MiB / 14 MiB / 0 MiB / 0 MiB )
53 min passed...832 MiB / 15160 MiB / 838 MiB / 32 MiB / 16 MiB ( 0 MiB / 120 MiB / 12 MiB / 0 MiB / 0 MiB )
54 min passed...832 MiB / 15280 MiB / 850 MiB / 32 MiB / 16 MiB ( 0 MiB / 120 MiB / 12 MiB / 0 MiB / 0 MiB )
55 min passed...832 MiB / 15280 MiB / 860 MiB / 32 MiB / 16 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
56 min passed...864 MiB / 15520 MiB / 874 MiB / 32 MiB / 17 MiB ( 32 MiB / 240 MiB / 14 MiB / 0 MiB / 1 MiB )
57 min passed...864 MiB / 15880 MiB / 886 MiB / 32 MiB / 17 MiB ( 0 MiB / 360 MiB / 12 MiB / 0 MiB / 0 MiB )
58 min passed...896 MiB / 16200 MiB / 898 MiB / 32 MiB / 17 MiB ( 32 MiB / 320 MiB / 12 MiB / 0 MiB / 0 MiB )
59 min passed...928 MiB / 16560 MiB / 910 MiB / 32 MiB / 17 MiB ( 32 MiB / 360 MiB / 12 MiB / 0 MiB / 0 MiB )
60 min passed...960 MiB / 16760 MiB / 930 MiB / 32 MiB / 18 MiB ( 32 MiB / 200 MiB / 20 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 7, 2021

Here is the Total Write Amplification, Read Amplification and Read Overhead for Round 2 Run 1 Test 1-9:

Diagram explanation:
The total write amplification is measured from the write fio did to what the NAND of the SSDs actually wrote. So if fio wrote 1 GiB and 45 GiB was written to the NAND of the SSDs thats a total write amplification of factor 45.

The read overhead shows how much additional reads fios write benchmark produced. If fio wanted to just write 1 GiB but that also caused 16 GiB of reads from the SSDs, that is a read overhead of factor 16.

The total read amplification is measured from the read that fios read benchmark did to what was actually read from the SSDs. So if fio read 1 GiB and 8 GiB was read from the SSDs thats a total read amplification of factor 8.

Table:

	Total Write Ampliifcation	Read Overhead	Total Read Amplification
R1T1: sync 4K rand	44,88	16,09	8,00
R1T2: sync 16K rand	13,79	1,59	2,00
R1T3: sync 4M seq	3,04	0,00	1,12
R1T4: async 4K rand	7,66	7,00	7,94
R1T5: async 16K rand	5,59	0,41	2,03
R1T6: async 4M seq	1,63	0,03	1,07
R1T7: async 16K rand 50% cmp	5,82	2,75	1,25
R1T8: sync 32K rand	11,48	0,28	1,00
R1T9: async 32K rand	5,63	0,03	0,97

Conclusion:
Sync writes/reads got a terrible amplification compared to async writes/reads. And the smaller the block size the bigger the amplification.
Interesting is that async 32K writes cause a total write amplification of 5.625 even through the zvols are also using a 32K volblocksize.
And the amplification of test 7 (async 16K random 50% compressible) is a little bit strange compared to test 5 (what is basically the same just with 100% random data that should be less compressible). LZ4 is used everywhere but if there actually is compressible data available the read overhead will be much higher but the read amplification lower.

Dunuin · Aug 7, 2021

And here the performance diagram of round 2 run 1:

Because it is hard to see any sync writes at all here a table:

	Write Throughput (MiB/s)	Read Throughput (MiB/s)
R1T1: sync 4K rand	1,17	6,13
R1T2: sync 16K rand	5,50	25,70
R1T3: sync 4M seq	262,00	898,00
R1T4: async 4K rand	80,10	137,00
R1T5: async 16K rand	380,00	536,00
R1T6: async 4M seq	938,00	1.393,00
R1T7: async 16K rand 50% cmp	234,00	635,00
R1T8: sync 32K rand	12,40	49,00
R1T9: async 32K rand	733,00	1.057,00

Most interesting conclusion: Compressible data makes the writes slower but reads faster (see test 5 vs test 7).
Async performance should be so much higher because of the iodepth of 64 compared to iodepth of 1 for sync writes. So SSDs are good at doing much stuff in parallel but terrible at doing single small operations.

Dunuin · Aug 7, 2021

Here is the write amplification more detailed:

Diagram explanation:
"W.A. Fio -> Guest" is how much data fio wrote running the benchmark vs amount of data that the guests virtual disks actually have written. So if fio wrote 1 GiB but the guest reported 5 GiB were written to the virtual disks thats a write amplification of factor 5.

"W.A. Guest -> Host" is how much data the guest has written to the virtual disks vs how much data the host has written to the physical disks. So if the guest wrote 5 GiB but the host reported 35 GiB were written to the physical disks thats a write amplification of factor 7.

"W.A. Host -> NAND" is how much data the host has written to the physical disks vs how much data the SSDs have reported to be written to the NAND flash cells. So if the host wrote 35 GiB but the SSDs reported 70 GiB were written to the NAND thats a write amplification of factor 2.

Table:

	Write Amplification Fio -> Guest	Write Amplification Guest -> Host	Write Amplification Host -> NAND
R1T1: sync 4K rand	5,13	7,18	1,22
R1T2: sync 16K rand	2,01	4,52	1,51
R1T3: sync 4M seq	1,00	2,60	1,16
R1T4: async 4K rand	1,01	5,25	1,44
R1T5: async 16K rand	1,00	1,95	2,79
R1T6: async 4M seq	1,00	1,33	1,22
R1T7: async 16K rand 50% cmp	1,00	2,02	2,87
R1T8: sync 32K rand	1,51	4,81	1,59
R1T9: async 32K rand	1,00	1,79	3,21

Conclusion:
The biggest problem isn't the write amplification inside the SSD but the write amplification that virtio and/or ZFS are producing.
What I don't understand is the write amplification from fio to guest. If fio writes 1 GiB as sync 4K blocks, why is ext4 (also used 4K blocksize) writing over 5 GiB? Ext4 should write stuff in the journal first and then later again to its final location but that shouldn't result in a write amplification of over factor 5.
Would be great if someone got an idea how to explain this.

Edit:
By the way...there is no detailed read amplification because all read amplification is caused by virtio/zfs between the guest and host. And it isn't hard to guess whats causing this:
read 4K block from 32K zvol = 32K/4K = factor 8 read amplification
read 16K block from 32K zvol = 32K/16K = factor 2 read amplification
read 32K block from 32K zvol = 32K/32K = no read amplification
So you really don't want to run a MySQL DB with 16K blocks on a raidz1/2/3 because a raidz forces you to increase you volblocksize over 16K and this will cause the more read amplification the bigger your volblocksize is. With a raidz you bascially always will need a volblocksize of 32K or higher except for a raidz1 consisting of 3 disks.

Dunuin · Aug 8, 2021

This was the Pool used in Round 2 Run 1:

root@Hypervisor:~# zpool status

Code:

  pool: VMpool7
 state: ONLINE
  scan: scrub repaired 0B in 00:04:12 with 0 errors on Sun Jul 25 08:04:13 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool7                                         ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G3_BTTV334505YT200GGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN  ONLINE       0     0     0

errors: No known data errors

zpool get all

Code:

NAME     PROPERTY                       VALUE                          SOURCE
VMpool7  size                           928G                           -
VMpool7  capacity                       27%                            -
VMpool7  altroot                        -                              default
VMpool7  health                         ONLINE                         -
VMpool7  guid                           8935882122565705413            -
VMpool7  version                        -                              default
VMpool7  bootfs                         -                              default
VMpool7  delegation                     on                             default
VMpool7  autoreplace                    off                            default
VMpool7  cachefile                      -                              default
VMpool7  failmode                       wait                           default
VMpool7  listsnapshots                  off                            default
VMpool7  autoexpand                     off                            default
VMpool7  dedupratio                     1.00x                          -
VMpool7  free                           676G                           -
VMpool7  allocated                      252G                           -
VMpool7  readonly                       off                            -
VMpool7  ashift                         12                             local
VMpool7  comment                        -                              default
VMpool7  expandsize                     -                              -
VMpool7  freeing                        0                              -
VMpool7  fragmentation                  44%                            -
VMpool7  leaked                         0                              -
VMpool7  multihost                      off                            default
VMpool7  checkpoint                     -                              -
VMpool7  load_guid                      6555050462514402137            -
VMpool7  autotrim                       off                            default
VMpool7  feature@async_destroy          enabled                        local
VMpool7  feature@empty_bpobj            active                         local
VMpool7  feature@lz4_compress           active                         local
VMpool7  feature@multi_vdev_crash_dump  enabled                        local
VMpool7  feature@spacemap_histogram     active                         local
VMpool7  feature@enabled_txg            active                         local
VMpool7  feature@hole_birth             active                         local
VMpool7  feature@extensible_dataset     active                         local
VMpool7  feature@embedded_data          active                         local
VMpool7  feature@bookmarks              enabled                        local
VMpool7  feature@filesystem_limits      enabled                        local
VMpool7  feature@large_blocks           enabled                        local
VMpool7  feature@large_dnode            enabled                        local
VMpool7  feature@sha512                 enabled                        local
VMpool7  feature@skein                  enabled                        local
VMpool7  feature@edonr                  enabled                        local
VMpool7  feature@userobj_accounting     active                         local
VMpool7  feature@encryption             active                         local
VMpool7  feature@project_quota          active                         local
VMpool7  feature@device_removal         enabled                        local
VMpool7  feature@obsolete_counts        enabled                        local
VMpool7  feature@zpool_checkpoint       enabled                        local
VMpool7  feature@spacemap_v2            active                         local
VMpool7  feature@allocation_classes     enabled                        local
VMpool7  feature@resilver_defer         enabled                        local
VMpool7  feature@bookmark_v2            enabled                        local
VMpool7  feature@redaction_bookmarks    enabled                        local
VMpool7  feature@redacted_datasets      enabled                        local
VMpool7  feature@bookmark_written       enabled                        local
VMpool7  feature@log_spacemap           active                         local
VMpool7  feature@livelist               enabled                        local
VMpool7  feature@device_rebuild         enabled                        local
VMpool7  feature@zstd_compress          enabled                        local

zfs get all VMpool7/VLT/VM

Code:

NAME            PROPERTY              VALUE                  SOURCE
VMpool7/VLT/VM  type                  filesystem             -
VMpool7/VLT/VM  creation              Sun Sep 13 13:26 2020  -
VMpool7/VLT/VM  used                  188G                   -
VMpool7/VLT/VM  available             373G                   -
VMpool7/VLT/VM  referenced            409K                   -
VMpool7/VLT/VM  compressratio         1.47x                  -
VMpool7/VLT/VM  mounted               yes                    -
VMpool7/VLT/VM  quota                 none                   default
VMpool7/VLT/VM  reservation           none                   default
VMpool7/VLT/VM  recordsize            128K                   default
VMpool7/VLT/VM  mountpoint            /VMpool7/VLT/VM        default
VMpool7/VLT/VM  sharenfs              off                    default
VMpool7/VLT/VM  checksum              on                     default
VMpool7/VLT/VM  compression           lz4                    inherited from VMpool7
VMpool7/VLT/VM  atime                 off                    inherited from VMpool7
VMpool7/VLT/VM  devices               on                     default
VMpool7/VLT/VM  exec                  on                     default
VMpool7/VLT/VM  setuid                on                     default
VMpool7/VLT/VM  readonly              off                    default
VMpool7/VLT/VM  zoned                 off                    default
VMpool7/VLT/VM  snapdir               hidden                 default
VMpool7/VLT/VM  aclmode               discard                default
VMpool7/VLT/VM  aclinherit            restricted             default
VMpool7/VLT/VM  createtxg             30                     -
VMpool7/VLT/VM  canmount              on                     local
VMpool7/VLT/VM  xattr                 on                     default
VMpool7/VLT/VM  copies                1                      default
VMpool7/VLT/VM  version               5                      -
VMpool7/VLT/VM  utf8only              off                    -
VMpool7/VLT/VM  normalization         none                   -
VMpool7/VLT/VM  casesensitivity       sensitive              -
VMpool7/VLT/VM  vscan                 off                    default
VMpool7/VLT/VM  nbmand                off                    default
VMpool7/VLT/VM  sharesmb              off                    default
VMpool7/VLT/VM  refquota              none                   default
VMpool7/VLT/VM  refreservation        none                   default
VMpool7/VLT/VM  guid                  2364380994997490815    -
VMpool7/VLT/VM  primarycache          all                    default
VMpool7/VLT/VM  secondarycache        all                    default
VMpool7/VLT/VM  usedbysnapshots       0B                     -
VMpool7/VLT/VM  usedbydataset         409K                   -
VMpool7/VLT/VM  usedbychildren        188G                   -
VMpool7/VLT/VM  usedbyrefreservation  0B                     -
VMpool7/VLT/VM  logbias               latency                default
VMpool7/VLT/VM  objsetid              387                    -
VMpool7/VLT/VM  dedup                 off                    default
VMpool7/VLT/VM  mlslabel              none                   default
VMpool7/VLT/VM  sync                  standard               inherited from VMpool7
VMpool7/VLT/VM  dnodesize             legacy                 default
VMpool7/VLT/VM  refcompressratio      1.00x                  -
VMpool7/VLT/VM  written               409K                   -
VMpool7/VLT/VM  logicalused           251G                   -
VMpool7/VLT/VM  logicalreferenced     103K                   -
VMpool7/VLT/VM  volmode               default                default
VMpool7/VLT/VM  filesystem_limit      none                   default
VMpool7/VLT/VM  snapshot_limit        none                   default
VMpool7/VLT/VM  filesystem_count      none                   default
VMpool7/VLT/VM  snapshot_count        none                   default
VMpool7/VLT/VM  snapdev               hidden                 default
VMpool7/VLT/VM  acltype               off                    default
VMpool7/VLT/VM  context               none                   default
VMpool7/VLT/VM  fscontext             none                   default
VMpool7/VLT/VM  defcontext            none                   default
VMpool7/VLT/VM  rootcontext           none                   default
VMpool7/VLT/VM  relatime              off                    default
VMpool7/VLT/VM  redundant_metadata    all                    default
VMpool7/VLT/VM  overlay               on                     default
VMpool7/VLT/VM  encryption            aes-256-gcm            -
VMpool7/VLT/VM  keylocation           none                   default
VMpool7/VLT/VM  keyformat             passphrase             -
VMpool7/VLT/VM  pbkdf2iters           350000                 -
VMpool7/VLT/VM  encryptionroot        VMpool7/VLT            -
VMpool7/VLT/VM  keystatus             available              -
VMpool7/VLT/VM  special_small_blocks  0                      default

Dunuin · Aug 8, 2021

This is the pool and VM that will be used in Round 2 Run 2:

Host is striped mirror of 6x S3710 200GB + 2x S3700 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=16K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 1 are bold)

root@Hypervisor:~# zpool status

Code:

  pool: VMpool9
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool9                                         ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G3_BTTV334405PG200GGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G3_BTTV334505YT200GGN  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN  ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN  ONLINE       0     0     0

errors: No known data errors

root@Hypervisor:~# zpool get all

Code:

NAME     PROPERTY                       VALUE                          SOURCE
VMpool9  size                           744G                           -
VMpool9  capacity                       0%                             -
VMpool9  altroot                        -                              default
VMpool9  health                         ONLINE                         -
VMpool9  guid                           5940430627130778622            -
VMpool9  version                        -                              default
VMpool9  bootfs                         -                              default
VMpool9  delegation                     on                             default
VMpool9  autoreplace                    off                            default
VMpool9  cachefile                      -                              default
VMpool9  failmode                       wait                           default
VMpool9  listsnapshots                  off                            default
VMpool9  autoexpand                     off                            default
VMpool9  dedupratio                     1.00x                          -
VMpool9  free                           743G                           -
VMpool9  allocated                      876M                           -
VMpool9  readonly                       off                            -
VMpool9  ashift                         12                             local
VMpool9  comment                        -                              default
VMpool9  expandsize                     -                              -
VMpool9  freeing                        0                              -
VMpool9  fragmentation                  0%                             -
VMpool9  leaked                         0                              -
VMpool9  multihost                      off                            default
VMpool9  checkpoint                     -                              -
VMpool9  load_guid                      6029569908885019619            -
VMpool9  autotrim                       off                            default
VMpool9  feature@async_destroy          enabled                        local
VMpool9  feature@empty_bpobj            active                         local
VMpool9  feature@lz4_compress           active                         local
VMpool9  feature@multi_vdev_crash_dump  enabled                        local
VMpool9  feature@spacemap_histogram     active                         local
VMpool9  feature@enabled_txg            active                         local
VMpool9  feature@hole_birth             active                         local
VMpool9  feature@extensible_dataset     active                         local
VMpool9  feature@embedded_data          active                         local
VMpool9  feature@bookmarks              enabled                        local
VMpool9  feature@filesystem_limits      enabled                        local
VMpool9  feature@large_blocks           enabled                        local
VMpool9  feature@large_dnode            enabled                        local
VMpool9  feature@sha512                 enabled                        local
VMpool9  feature@skein                  enabled                        local
VMpool9  feature@edonr                  enabled                        local
VMpool9  feature@userobj_accounting     active                         local
VMpool9  feature@encryption             active                         local
VMpool9  feature@project_quota          active                         local
VMpool9  feature@device_removal         enabled                        local
VMpool9  feature@obsolete_counts        enabled                        local
VMpool9  feature@zpool_checkpoint       enabled                        local
VMpool9  feature@spacemap_v2            active                         local
VMpool9  feature@allocation_classes     enabled                        local
VMpool9  feature@resilver_defer         enabled                        local
VMpool9  feature@bookmark_v2            enabled                        local
VMpool9  feature@redaction_bookmarks    enabled                        local
VMpool9  feature@redacted_datasets      enabled                        local
VMpool9  feature@bookmark_written       enabled                        local
VMpool9  feature@log_spacemap           active                         local
VMpool9  feature@livelist               enabled                        local
VMpool9  feature@device_rebuild         enabled                        local
VMpool9  feature@zstd_compress          enabled                        local

root@Hypervisor:~# zfs get all VMpool9/VLT/VM

Code:

NAME            PROPERTY              VALUE                  SOURCE
VMpool9/VLT/VM  type                  filesystem             -
VMpool9/VLT/VM  creation              Sun Aug  8 16:31 2021  -
VMpool9/VLT/VM  used                  875M                   -
VMpool9/VLT/VM  available             575G                   -
VMpool9/VLT/VM  referenced            192K                   -
VMpool9/VLT/VM  compressratio         1.72x                  -
VMpool9/VLT/VM  mounted               yes                    -
VMpool9/VLT/VM  quota                 none                   default
VMpool9/VLT/VM  reservation           none                   default
VMpool9/VLT/VM  recordsize            128K                   default
VMpool9/VLT/VM  mountpoint            /VMpool9/VLT/VM        default
VMpool9/VLT/VM  sharenfs              off                    default
VMpool9/VLT/VM  checksum              on                     default
VMpool9/VLT/VM  compression           lz4                    inherited from VMpool9
VMpool9/VLT/VM  atime                 off                    inherited from VMpool9
VMpool9/VLT/VM  devices               on                     default
VMpool9/VLT/VM  exec                  on                     default
VMpool9/VLT/VM  setuid                on                     default
VMpool9/VLT/VM  readonly              off                    default
VMpool9/VLT/VM  zoned                 off                    default
VMpool9/VLT/VM  snapdir               hidden                 default
VMpool9/VLT/VM  aclmode               discard                default
VMpool9/VLT/VM  aclinherit            restricted             default
VMpool9/VLT/VM  createtxg             104                    -
VMpool9/VLT/VM  canmount              on                     local
VMpool9/VLT/VM  xattr                 on                     default
VMpool9/VLT/VM  copies                1                      default
VMpool9/VLT/VM  version               5                      -
VMpool9/VLT/VM  utf8only              off                    -
VMpool9/VLT/VM  normalization         none                   -
VMpool9/VLT/VM  casesensitivity       sensitive              -
VMpool9/VLT/VM  vscan                 off                    default
VMpool9/VLT/VM  nbmand                off                    default
VMpool9/VLT/VM  sharesmb              off                    default
VMpool9/VLT/VM  refquota              none                   default
VMpool9/VLT/VM  refreservation        none                   default
VMpool9/VLT/VM  guid                  6946211764074389943    -
VMpool9/VLT/VM  primarycache          metadata               inherited from VMpool9
VMpool9/VLT/VM  secondarycache        all                    default
VMpool9/VLT/VM  usedbysnapshots       0B                     -
VMpool9/VLT/VM  usedbydataset         192K                   -
VMpool9/VLT/VM  usedbychildren        874M                   -
VMpool9/VLT/VM  usedbyrefreservation  0B                     -
VMpool9/VLT/VM  logbias               latency                default
VMpool9/VLT/VM  objsetid              285                    -
VMpool9/VLT/VM  dedup                 off                    default
VMpool9/VLT/VM  mlslabel              none                   default
VMpool9/VLT/VM  sync                  standard               inherited from VMpool9
VMpool9/VLT/VM  dnodesize             legacy                 default
VMpool9/VLT/VM  refcompressratio      1.00x                  -
VMpool9/VLT/VM  written               192K                   -
VMpool9/VLT/VM  logicalused           1.47G                  -
VMpool9/VLT/VM  logicalreferenced     69K                    -
VMpool9/VLT/VM  volmode               default                default
VMpool9/VLT/VM  filesystem_limit      none                   default
VMpool9/VLT/VM  snapshot_limit        none                   default
VMpool9/VLT/VM  filesystem_count      none                   default
VMpool9/VLT/VM  snapshot_count        none                   default
VMpool9/VLT/VM  snapdev               hidden                 default
VMpool9/VLT/VM  acltype               off                    default
VMpool9/VLT/VM  context               none                   default
VMpool9/VLT/VM  fscontext             none                   default
VMpool9/VLT/VM  defcontext            none                   default
VMpool9/VLT/VM  rootcontext           none                   default
VMpool9/VLT/VM  relatime              off                    default
VMpool9/VLT/VM  redundant_metadata    all                    default
VMpool9/VLT/VM  overlay               on                     default
VMpool9/VLT/VM  encryption            aes-256-gcm            -
VMpool9/VLT/VM  keylocation           none                   default
VMpool9/VLT/VM  keyformat             passphrase             -
VMpool9/VLT/VM  pbkdf2iters           350000                 -
VMpool9/VLT/VM  encryptionroot        VMpool9/VLT            -
VMpool9/VLT/VM  keystatus             available              -
VMpool9/VLT/VM  special_small_blocks  0                      default

Commands used to create the pool:

Code:

# find names of disks:
ls -l /dev/disk/by-id/
# create striped mirror of 8 disks (6x Intel S3710 200GB + 2x Intel S3700 200GB):
zpool create -f -o ashift=12 VMpool9 mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G3_BTTV334405PG200GGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G3_BTTV334505YT200GGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636205M3200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208K6200MGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV511505AE200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV636208L0200MGN mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN
# set ZFS options
zpool set autoexpand=off VMpool9
zpool set feature@encryption=enabled VMpool9
zfs set compression=lz4 VMpool9
zfs set sync=standard VMpool9
zfs set canmount=off VMpool9
zfs set atime=off VMpool9
# set quota to 80% so pool can't get full/slow:
zfs set quota=576G VMpool9
# create encrypted dataset:
zfs create -o encryption=on -o keyformat=passphrase VMpool9/VLT
zfs set canmount=off VMpool9/VLT
# create dataset for encrypted VMs
zfs create -o encryption=on VMpool9/VLT/VM
zfs set canmount=on VMpool9/VLT/VM
# add dataset to PVE:
pvesm add zfspool VMpool9_VLT_VM -pool VMpool9/VLT/VM --content images,rootdir --sparse 1 --mountpoint /VMpool9/VLT/VM
# disable ARC caching:
zfs set primarycache=metadata VMpool9
# don't forget to set block size from 8K to 16K using WebUI

Dunuin · Aug 8, 2021

All tests are done on a Supermicro X10SRM-F with a Intel E5-2620v4 and 4x 16GB DDR4 2400 ECC RAM. BIOS, IPMI and SSDs firmware is flashed to latest version.
The VM is set to "host", got 8 vCPUs and 16GiB RAM.

Round 2 - Run 2 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1164KiB/s][w=291 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=787: Sun Aug  8 17:38:35 2021
  write: IOPS=291, BW=1165KiB/s (1193kB/s)(1024MiB/899806msec); 0 zone resets
    clat (usec): min=1278, max=37823, avg=3417.65, stdev=965.78
     lat (usec): min=1279, max=37825, avg=3419.85, stdev=965.82
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2474], 10.00th=[ 2638], 20.00th=[ 2868],
     | 30.00th=[ 3064], 40.00th=[ 3261], 50.00th=[ 3392], 60.00th=[ 3523],
     | 70.00th=[ 3654], 80.00th=[ 3785], 90.00th=[ 3949], 95.00th=[ 4146],
     | 99.00th=[ 6063], 99.50th=[ 6521], 99.90th=[16057], 99.95th=[22152],
     | 99.99th=[30540]
   bw (  KiB/s): min=  920, max= 1704, per=100.00%, avg=1165.20, stdev=104.76, samples=1799
   iops        : min=  230, max=  426, avg=291.29, stdev=26.19, samples=1799
  lat (msec)   : 2=0.27%, 4=92.08%, 10=7.43%, 20=0.17%, 50=0.06%
  cpu          : usr=0.71%, sys=4.10%, ctx=631791, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1165KiB/s (1193kB/s), 1165KiB/s-1165KiB/s (1193kB/s-1193kB/s), io=1024MiB (1074MB), run=899806-899806msec

Disk stats (read/write):
  sda: ios=2/797488, merge=0/552624, ticks=1/850284, in_queue=903812, util=99.63%
0 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 5270 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5270 MiB / 18 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=6718KiB/s][r=1679 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=996: Sun Aug  8 17:56:11 2021
  read: IOPS=1694, BW=6778KiB/s (6941kB/s)(1024MiB/154695msec)
    clat (usec): min=205, max=6276, avg=578.86, stdev=190.66
     lat (usec): min=207, max=6278, avg=580.88, stdev=190.70
    clat percentiles (usec):
     |  1.00th=[  367],  5.00th=[  412], 10.00th=[  441], 20.00th=[  482],
     | 30.00th=[  515], 40.00th=[  545], 50.00th=[  570], 60.00th=[  594],
     | 70.00th=[  627], 80.00th=[  652], 90.00th=[  676], 95.00th=[  701],
     | 99.00th=[  898], 99.50th=[ 1991], 99.90th=[ 3163], 99.95th=[ 3294],
     | 99.99th=[ 3687]
   bw (  KiB/s): min= 5784, max= 8856, per=99.99%, avg=6777.59, stdev=629.11, samples=309
   iops        : min= 1446, max= 2214, avg=1694.37, stdev=157.29, samples=309
  lat (usec)   : 250=0.01%, 500=24.97%, 750=73.81%, 1000=0.29%
  lat (msec)   : 2=0.43%, 4=0.49%, 10=0.01%
  cpu          : usr=3.10%, sys=9.39%, ctx=262276, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6778KiB/s (6941kB/s), 6778KiB/s-6778KiB/s (6941kB/s-6941kB/s), io=1024MiB (1074MB), run=154695-154695msec

Disk stats (read/write):
  sda: ios=262064/10, merge=0/7, ticks=141281/12, in_queue=154736, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5270 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5271 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 2856 seconds or 47 minutes.

Host:
CPU utilization jumped from 1 to 6.5%. IO Delay from 0 to 0,5%.

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3456 MiB / 4011 MiB / 3475 MiB / 256 MiB / 283 MiB ( 3456 MiB / 4011 MiB / 3475 MiB / 256 MiB / 283 MiB )
2 min passed...6912 MiB / 7765 MiB / 6982 MiB / 672 MiB / 635 MiB ( 3456 MiB / 3754 MiB / 3507 MiB / 416 MiB / 352 MiB )
3 min passed...10400 MiB / 11392 MiB / 10429 MiB / 1056 MiB / 1038 MiB ( 3488 MiB / 3627 MiB / 3447 MiB / 384 MiB / 403 MiB )
4 min passed...13856 MiB / 15530 MiB / 13866 MiB / 1504 MiB / 1483 MiB ( 3456 MiB / 4138 MiB / 3437 MiB / 448 MiB / 445 MiB )
5 min passed...17216 MiB / 19498 MiB / 17249 MiB / 1984 MiB / 1962 MiB ( 3360 MiB / 3968 MiB / 3383 MiB / 480 MiB / 479 MiB )
6 min passed...20576 MiB / 24319 MiB / 20592 MiB / 2496 MiB / 2465 MiB ( 3360 MiB / 4821 MiB / 3343 MiB / 512 MiB / 503 MiB )
7 min passed...23968 MiB / 27818 MiB / 23978 MiB / 3040 MiB / 3002 MiB ( 3392 MiB / 3499 MiB / 3386 MiB / 544 MiB / 537 MiB )
8 min passed...27232 MiB / 31359 MiB / 27244 MiB / 3584 MiB / 3555 MiB ( 3264 MiB / 3541 MiB / 3266 MiB / 544 MiB / 553 MiB )
9 min passed...30624 MiB / 34858 MiB / 30625 MiB / 4160 MiB / 4124 MiB ( 3392 MiB / 3499 MiB / 3381 MiB / 576 MiB / 569 MiB )
10 min passed...33952 MiB / 38740 MiB / 33943 MiB / 4640 MiB / 4678 MiB ( 3328 MiB / 3882 MiB / 3318 MiB / 480 MiB / 554 MiB )
11 min passed...37152 MiB / 43604 MiB / 37170 MiB / 5216 MiB / 5212 MiB ( 3200 MiB / 4864 MiB / 3227 MiB / 576 MiB / 534 MiB )
12 min passed...40512 MiB / 47444 MiB / 40494 MiB / 5760 MiB / 5768 MiB ( 3360 MiB / 3840 MiB / 3324 MiB / 544 MiB / 556 MiB )
13 min passed...43776 MiB / 50900 MiB / 43820 MiB / 6272 MiB / 6329 MiB ( 3264 MiB / 3456 MiB / 3326 MiB / 512 MiB / 561 MiB )
14 min passed...47104 MiB / 54399 MiB / 47111 MiB / 6912 MiB / 6874 MiB ( 3328 MiB / 3499 MiB / 3291 MiB / 640 MiB / 545 MiB )
15 min passed...49920 MiB / 57940 MiB / 49932 MiB / 7296 MiB / 7303 MiB ( 2816 MiB / 3541 MiB / 2821 MiB / 384 MiB / 429 MiB )
16 min passed...49920 MiB / 58623 MiB / 49935 MiB / 7296 MiB / 7304 MiB ( 0 MiB / 683 MiB / 3 MiB / 0 MiB / 1 MiB )
17 min passed...49920 MiB / 58623 MiB / 49935 MiB / 7296 MiB / 7304 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...49920 MiB / 59135 MiB / 49935 MiB / 7296 MiB / 7304 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...49920 MiB / 59476 MiB / 49935 MiB / 7296 MiB / 7304 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...49920 MiB / 59647 MiB / 49935 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 6 MiB )
21 min passed...49920 MiB / 60286 MiB / 49935 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 639 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...49920 MiB / 60286 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
23 min passed...49920 MiB / 60286 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...49920 MiB / 60286 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...49920 MiB / 61481 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 1195 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...49920 MiB / 62462 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 981 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...49920 MiB / 62462 MiB / 49939 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...49920 MiB / 62462 MiB / 49941 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
29 min passed...49920 MiB / 62462 MiB / 49942 MiB / 7296 MiB / 7310 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
30 min passed...49920 MiB / 62676 MiB / 49942 MiB / 7808 MiB / 7798 MiB ( 0 MiB / 214 MiB / 0 MiB / 512 MiB / 488 MiB )
31 min passed...49920 MiB / 63316 MiB / 49942 MiB / 9376 MiB / 9393 MiB ( 0 MiB / 640 MiB / 0 MiB / 1568 MiB / 1595 MiB )
32 min passed...49920 MiB / 64126 MiB / 49947 MiB / 10976 MiB / 10999 MiB ( 0 MiB / 810 MiB / 5 MiB / 1600 MiB / 1606 MiB )
33 min passed...49920 MiB / 64254 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 128 MiB / 0 MiB / 416 MiB / 407 MiB )
34 min passed...49920 MiB / 64254 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...49920 MiB / 65022 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
36 min passed...49920 MiB / 66004 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 982 MiB / 0 MiB / 0 MiB / 0 MiB )
37 min passed...49920 MiB / 66004 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...49920 MiB / 66004 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...49920 MiB / 66772 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...49920 MiB / 67497 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...49920 MiB / 67838 MiB / 49947 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
42 min passed...49920 MiB / 67838 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
43 min passed...49920 MiB / 67838 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...49920 MiB / 67838 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...49920 MiB / 68180 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 342 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...49920 MiB / 69630 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 1450 MiB / 0 MiB / 0 MiB / 0 MiB )
47 min passed...49920 MiB / 69673 MiB / 49952 MiB / 11392 MiB / 11406 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )

Conclusion:
The write amplification is even more terrible. Its a total write amplification of factor 61 (and raidz1 "only" was factor 45).

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB,                                                                                                                                                              (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=6064KiB/s][w=379 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=629: Sun Aug  8 18:21:02 2021
  write: IOPS=343, BW=5495KiB/s (5627kB/s)(1024MiB/190826msec); 0 zone resets
    clat (usec): min=1615, max=33989, avg=2891.67, stdev=834.65
     lat (usec): min=1616, max=33992, avg=2893.88, stdev=834.65
    clat percentiles (usec):
     |  1.00th=[ 2089],  5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2573],
     | 30.00th=[ 2671], 40.00th=[ 2769], 50.00th=[ 2835], 60.00th=[ 2900],
     | 70.00th=[ 2999], 80.00th=[ 3097], 90.00th=[ 3228], 95.00th=[ 3359],
     | 99.00th=[ 4948], 99.50th=[ 5473], 99.90th=[13698], 99.95th=[19530],
     | 99.99th=[30540]
   bw (  KiB/s): min= 4576, max= 6912, per=100.00%, avg=5494.86, stdev=404.99, s                                                                                                                                                             amples=381
   iops        : min=  286, max=  432, avg=343.42, stdev=25.32, samples=381
  lat (msec)   : 2=0.50%, 4=97.50%, 10=1.76%, 20=0.20%, 50=0.05%
  cpu          : usr=1.03%, sys=4.84%, ctx=144508, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5495KiB/s (5627kB/s), 5495KiB/s-5495KiB/s (5627kB/s-5627kB/s), io=10                                                                                                                                                             24MiB (1074MB), run=190826-190826msec

Disk stats (read/write):
  sda: ios=0/197320, merge=0/133688, ticks=0/173779, in_queue=190060, util=99.68                                                                                                                                                             %
0 min passed...Guest Writes/Reads (Changes): 2062 MiB / 19 MiB ( 2062 MiB / 19 M                                                                                                                                                             iB )
1 min passed...Guest Writes/Reads (Changes): 2062 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2062 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2062 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2062 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 1 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2063 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][97.4%][r=29.8MiB/s][r=1909 IOPS][eta 00m:01s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=835: Sun Aug  8 18:36:39 2021
  read: IOPS=1777, BW=27.8MiB/s (29.1MB/s)(1024MiB/36871msec)
    clat (usec): min=205, max=5619, avg=551.42, stdev=128.29
     lat (usec): min=207, max=5621, avg=553.43, stdev=128.35
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  412], 10.00th=[  437], 20.00th=[  465],
     | 30.00th=[  494], 40.00th=[  519], 50.00th=[  545], 60.00th=[  570],
     | 70.00th=[  594], 80.00th=[  635], 90.00th=[  668], 95.00th=[  693],
     | 99.00th=[  734], 99.50th=[  750], 99.90th=[ 2540], 99.95th=[ 3032],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=24032, max=33472, per=99.87%, avg=28402.84, stdev=1941.81, samples=73
   iops        : min= 1502, max= 2092, avg=1775.15, stdev=121.38, samples=73
  lat (usec)   : 250=0.01%, 500=33.01%, 750=66.53%, 1000=0.16%
  lat (msec)   : 2=0.16%, 4=0.14%, 10=0.01%
  cpu          : usr=3.39%, sys=9.61%, ctx=65565, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=27.8MiB/s (29.1MB/s), 27.8MiB/s-27.8MiB/s (29.1MB/s-29.1MB/s), io=1024MiB (1074MB), run=36871-36871msec

Disk stats (read/write):
  sda: ios=65090/0, merge=0/0, ticks=33375/0, in_queue=36688, util=99.92%
0 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2063 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 2030 seconds or 33 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...3936 MiB / 4863 MiB / 3969 MiB / 256 MiB / 238 MiB ( 3936 MiB / 4863 MiB / 3969 MiB / 256 MiB / 238 MiB )
2 min passed...8128 MiB / 9941 MiB / 8148 MiB / 512 MiB / 497 MiB ( 4192 MiB / 5078 MiB / 4179 MiB / 256 MiB / 259 MiB )
3 min passed...12128 MiB / 14079 MiB / 12232 MiB / 672 MiB / 688 MiB ( 4000 MiB / 4138 MiB / 4084 MiB / 160 MiB / 191 MiB )
4 min passed...13056 MiB / 15060 MiB / 13028 MiB / 768 MiB / 723 MiB ( 928 MiB / 981 MiB / 796 MiB / 96 MiB / 35 MiB )
5 min passed...13056 MiB / 15060 MiB / 13028 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...13056 MiB / 16340 MiB / 13028 MiB / 768 MiB / 723 MiB ( 0 MiB / 1280 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...13056 MiB / 16980 MiB / 13028 MiB / 768 MiB / 723 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...13056 MiB / 16980 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 7 MiB / 0 MiB / 0 MiB )
9 min passed...13056 MiB / 16980 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...13056 MiB / 16980 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...13056 MiB / 17706 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 726 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...13056 MiB / 18858 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 1152 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...13056 MiB / 19540 MiB / 13035 MiB / 768 MiB / 723 MiB ( 0 MiB / 682 MiB / 0 MiB / 0 MiB / 0 MiB )
14 min passed...13056 MiB / 19540 MiB / 13039 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
15 min passed...13056 MiB / 19540 MiB / 13039 MiB / 768 MiB / 723 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...13056 MiB / 19924 MiB / 13042 MiB / 768 MiB / 723 MiB ( 0 MiB / 384 MiB / 3 MiB / 0 MiB / 0 MiB )
17 min passed...13056 MiB / 20607 MiB / 13042 MiB / 768 MiB / 723 MiB ( 0 MiB / 683 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...13056 MiB / 20607 MiB / 13046 MiB / 768 MiB / 724 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 1 MiB )
19 min passed...13056 MiB / 21034 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 427 MiB / 0 MiB / 960 MiB / 1024 MiB )
20 min passed...13056 MiB / 21546 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...13056 MiB / 22399 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 853 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...13056 MiB / 23124 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...13056 MiB / 23124 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...13056 MiB / 23124 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...13056 MiB / 23124 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...13056 MiB / 24362 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 1238 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...13056 MiB / 25044 MiB / 13046 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 682 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...13056 MiB / 25087 MiB / 13049 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 43 MiB / 3 MiB / 0 MiB / 0 MiB )
29 min passed...13056 MiB / 25087 MiB / 13049 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...13056 MiB / 25130 MiB / 13049 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...13056 MiB / 26068 MiB / 13049 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 938 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...13056 MiB / 26580 MiB / 13049 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...13056 MiB / 27348 MiB / 13052 MiB / 1728 MiB / 1748 MiB ( 0 MiB / 768 MiB / 3 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_3.sh
sync_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][98.0%][w=164MiB/s][w=41 IOPS][eta 00m:01s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=692: Sun Aug  8 18:57:52 2021
  write: IOPS=52, BW=210MiB/s (220MB/s)(10.0GiB/48775msec); 0 zone resets
    clat (usec): min=11749, max=89702, avg=17292.33, stdev=7976.30
     lat (usec): min=11752, max=89705, avg=17295.86, stdev=7976.53
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[12649], 10.00th=[12780], 20.00th=[12911],
     | 30.00th=[13042], 40.00th=[13173], 50.00th=[13304], 60.00th=[13435],
     | 70.00th=[13829], 80.00th=[26870], 90.00th=[29230], 95.00th=[30802],
     | 99.00th=[44827], 99.50th=[53216], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[89654]
   bw (  KiB/s): min=114688, max=286720, per=99.81%, avg=214582.45, stdev=63267.16, samples=97
   iops        : min=   28, max=   70, avg=52.36, stdev=15.44, samples=97
  lat (msec)   : 20=75.90%, 50=23.36%, 100=0.74%
  cpu          : usr=9.23%, sys=2.23%, ctx=6179, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=10.0GiB (10.7GB), run=48775-48775msec

Disk stats (read/write):
  sda: ios=0/15317, merge=0/5268, ticks=0/58628, in_queue=63668, util=98.01%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 10281 MiB / 20 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 20 MiB ( 0 MiB / 0 MiB )
sync_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=985MiB/s][r=246 IOPS][eta 00m:00s]
sync_seq_4M: (groupid=0, jobs=1): err= 0: pid=895: Sun Aug  8 19:13:03 2021
  read: IOPS=246, BW=986MiB/s (1034MB/s)(10.0GiB/10386msec)
    clat (usec): min=3388, max=10367, avg=4046.43, stdev=613.46
     lat (usec): min=3390, max=10369, avg=4048.03, stdev=613.49
    clat percentiles (usec):
     |  1.00th=[ 3589],  5.00th=[ 3654], 10.00th=[ 3687], 20.00th=[ 3752],
     | 30.00th=[ 3785], 40.00th=[ 3818], 50.00th=[ 3851], 60.00th=[ 3916],
     | 70.00th=[ 3982], 80.00th=[ 4080], 90.00th=[ 4490], 95.00th=[ 5473],
     | 99.00th=[ 6849], 99.50th=[ 7177], 99.90th=[ 7963], 99.95th=[ 8094],
     | 99.99th=[10421]
   bw (  KiB/s): min=964726, max=1024000, per=99.91%, avg=1008648.55, stdev=17392.09, samples=20
   iops        : min=  235, max=  250, avg=246.20, stdev= 4.34, samples=20
  lat (msec)   : 4=73.48%, 10=26.48%, 20=0.04%
  cpu          : usr=0.44%, sys=5.79%, ctx=2570, majf=0, minf=523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=986MiB/s (1034MB/s), 986MiB/s-986MiB/s (1034MB/s-1034MB/s), io=10.0GiB (10.7GB), run=10386-10386msec

Disk stats (read/write):
  sda: ios=10000/0, merge=0/0, ticks=34305/0, in_queue=35220, util=99.43%
0 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10281 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Script ran 1861 seconds or 31 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...42464 MiB / 45482 MiB / 42442 MiB / 64 MiB / 34 MiB ( 42464 MiB / 45482 MiB / 42442 MiB / 64 MiB / 34 MiB )
2 min passed...42464 MiB / 46378 MiB / 42443 MiB / 64 MiB / 34 MiB ( 0 MiB / 896 MiB / 1 MiB / 0 MiB / 0 MiB )
3 min passed...42464 MiB / 46932 MiB / 42443 MiB / 64 MiB / 34 MiB ( 0 MiB / 554 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...42464 MiB / 46932 MiB / 42443 MiB / 64 MiB / 34 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...42464 MiB / 46932 MiB / 42447 MiB / 64 MiB / 35 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 1 MiB )
6 min passed...42464 MiB / 47018 MiB / 42447 MiB / 64 MiB / 35 MiB ( 0 MiB / 86 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...42464 MiB / 48127 MiB / 42447 MiB / 64 MiB / 35 MiB ( 0 MiB / 1109 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...42464 MiB / 49236 MiB / 42447 MiB / 64 MiB / 35 MiB ( 0 MiB / 1109 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...42464 MiB / 49236 MiB / 42451 MiB / 64 MiB / 35 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...42464 MiB / 49236 MiB / 42451 MiB / 64 MiB / 35 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...42464 MiB / 49364 MiB / 42451 MiB / 64 MiB / 35 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...42464 MiB / 50004 MiB / 42454 MiB / 64 MiB / 35 MiB ( 0 MiB / 640 MiB / 3 MiB / 0 MiB / 0 MiB )
13 min passed...42464 MiB / 50516 MiB / 42456 MiB / 64 MiB / 35 MiB ( 0 MiB / 512 MiB / 2 MiB / 0 MiB / 0 MiB )
14 min passed...42496 MiB / 51242 MiB / 42458 MiB / 64 MiB / 35 MiB ( 32 MiB / 726 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...42496 MiB / 51412 MiB / 42458 MiB / 64 MiB / 35 MiB ( 0 MiB / 170 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...42496 MiB / 51711 MiB / 42458 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 299 MiB / 0 MiB / 10432 MiB / 10424 MiB )
17 min passed...42496 MiB / 52436 MiB / 42458 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...42496 MiB / 52649 MiB / 42460 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 213 MiB / 2 MiB / 0 MiB / 0 MiB )
19 min passed...42496 MiB / 52649 MiB / 42462 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
20 min passed...42496 MiB / 52863 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 214 MiB / 3 MiB / 0 MiB / 0 MiB )
21 min passed...42496 MiB / 53801 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 938 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...42496 MiB / 54569 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...42496 MiB / 54740 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...42496 MiB / 54740 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...42496 MiB / 54740 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...42496 MiB / 55124 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...42496 MiB / 56319 MiB / 42465 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 1195 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...42528 MiB / 57044 MiB / 42470 MiB / 10496 MiB / 10459 MiB ( 32 MiB / 725 MiB / 5 MiB / 0 MiB / 0 MiB )
29 min passed...42528 MiB / 57044 MiB / 42470 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...42528 MiB / 57044 MiB / 42473 MiB / 10496 MiB / 10459 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
31 min passed...42560 MiB / 57471 MiB / 42501 MiB / 10496 MiB / 10460 MiB ( 32 MiB / 427 MiB / 28 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 4: async 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_4.sh
async_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=113MiB/s][w=29.0k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=632: Sun Aug  8 19:32:25 2021
  write: IOPS=23.9k, BW=93.2MiB/s (97.8MB/s)(1024MiB/10982msec); 0 zone resets
    slat (usec): min=7, max=34822, avg=25.71, stdev=71.14
    clat (usec): min=643, max=48417, avg=2647.50, stdev=2207.26
     lat (usec): min=655, max=48440, avg=2674.63, stdev=2210.30
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1614], 10.00th=[ 1696], 20.00th=[ 1795],
     | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2278],
     | 70.00th=[ 2376], 80.00th=[ 2638], 90.00th=[ 3130], 95.00th=[ 6128],
     | 99.00th=[11469], 99.50th=[12649], 99.90th=[36963], 99.95th=[41681],
     | 99.99th=[45876]
   bw (  KiB/s): min=32664, max=132032, per=98.10%, avg=93666.29, stdev=32922.39, samples=21
   iops        : min= 8166, max=33008, avg=23416.67, stdev=8230.64, samples=21
  lat (usec)   : 750=0.01%, 1000=0.17%
  lat (msec)   : 2=32.76%, 4=60.02%, 10=4.95%, 20=1.95%, 50=0.14%
  cpu          : usr=15.80%, sys=49.82%, ctx=203492, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=93.2MiB/s (97.8MB/s), 93.2MiB/s-93.2MiB/s (97.8MB/s-97.8MB/s), io=1024MiB (1074MB), run=10982-10982msec

Disk stats (read/write):
  sda: ios=0/259000, merge=0/1309, ticks=0/240018, in_queue=244804, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1029 MiB / 19 MiB ( 1029 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 5 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1034 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=264MiB/s][r=67.6k IOPS][eta 00m:00s]
async_rand_4K: (groupid=0, jobs=1): err= 0: pid=885: Sun Aug  8 19:47:30 2021
  read: IOPS=66.6k, BW=260MiB/s (273MB/s)(1024MiB/3934msec)
    slat (usec): min=4, max=1247, avg= 8.04, stdev= 4.61
    clat (usec): min=253, max=5012, avg=947.32, stdev=132.17
     lat (usec): min=263, max=5021, avg=956.55, stdev=132.46
    clat percentiles (usec):
     |  1.00th=[  799],  5.00th=[  848], 10.00th=[  865], 20.00th=[  889],
     | 30.00th=[  906], 40.00th=[  922], 50.00th=[  938], 60.00th=[  955],
     | 70.00th=[  963], 80.00th=[  988], 90.00th=[ 1012], 95.00th=[ 1045],
     | 99.00th=[ 1221], 99.50th=[ 1663], 99.90th=[ 3130], 99.95th=[ 3458],
     | 99.99th=[ 3752]
   bw (  KiB/s): min=258136, max=269352, per=99.68%, avg=265692.57, stdev=3846.19, samples=7
   iops        : min=64534, max=67338, avg=66423.14, stdev=961.55, samples=7
  lat (usec)   : 500=0.01%, 750=0.18%, 1000=85.40%
  lat (msec)   : 2=14.10%, 4=0.31%, 10=0.01%
  cpu          : usr=40.17%, sys=59.78%, ctx=30, majf=0, minf=73
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=260MiB/s (273MB/s), 260MiB/s-260MiB/s (273MB/s-273MB/s), io=1024MiB (1074MB), run=3934-3934msec

Disk stats (read/write):
  sda: ios=259604/0, merge=0/0, ticks=73067/0, in_queue=73684, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1034 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 1 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1035 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1816 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...4992 MiB / 5546 MiB / 5018 MiB / 4128 MiB / 4130 MiB ( 4992 MiB / 5546 MiB / 5018 MiB / 4128 MiB / 4130 MiB )
2 min passed...4992 MiB / 6272 MiB / 5021 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 726 MiB / 3 MiB / 0 MiB / 0 MiB )
3 min passed...4992 MiB / 6570 MiB / 5024 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 298 MiB / 3 MiB / 0 MiB / 0 MiB )
4 min passed...4992 MiB / 6570 MiB / 5028 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
5 min passed...4992 MiB / 6783 MiB / 5028 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 213 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...4992 MiB / 7850 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 1067 MiB / 4 MiB / 0 MiB / 0 MiB )
7 min passed...4992 MiB / 8874 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 1024 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...4992 MiB / 9130 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...4992 MiB / 9130 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...4992 MiB / 9258 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...4992 MiB / 9599 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...4992 MiB / 10922 MiB / 5032 MiB / 4128 MiB / 4130 MiB ( 0 MiB / 1323 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...4992 MiB / 11519 MiB / 5041 MiB / 4128 MiB / 4131 MiB ( 0 MiB / 597 MiB / 9 MiB / 0 MiB / 1 MiB )
14 min passed...4992 MiB / 11519 MiB / 5041 MiB / 4128 MiB / 4131 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...4992 MiB / 11775 MiB / 5045 MiB / 4128 MiB / 4131 MiB ( 0 MiB / 256 MiB / 4 MiB / 0 MiB / 0 MiB )
16 min passed...4992 MiB / 12245 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 470 MiB / 0 MiB / 4064 MiB / 4077 MiB )
17 min passed...4992 MiB / 13055 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 810 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...4992 MiB / 13269 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 214 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...4992 MiB / 14079 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 810 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...4992 MiB / 14250 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...4992 MiB / 14463 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 213 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...4992 MiB / 15018 MiB / 5045 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 555 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...4992 MiB / 15061 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 43 MiB / 4 MiB / 0 MiB / 0 MiB )
24 min passed...4992 MiB / 15061 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...4992 MiB / 15701 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...4992 MiB / 16639 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 938 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...4992 MiB / 17194 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 555 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...4992 MiB / 17279 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...4992 MiB / 17279 MiB / 5049 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...4992 MiB / 17322 MiB / 5071 MiB / 8192 MiB / 8208 MiB ( 0 MiB / 43 MiB / 22 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 5: async 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_5.sh
async_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=728: Sun Aug  8 20:07:16 2021
  write: IOPS=28.2k, BW=441MiB/s (462MB/s)(1024MiB/2323msec); 0 zone resets
    slat (usec): min=7, max=8443, avg=19.82, stdev=36.42
    clat (usec): min=86, max=28278, avg=2233.44, stdev=913.81
     lat (usec): min=114, max=28304, avg=2254.45, stdev=916.53
    clat percentiles (usec):
     |  1.00th=[ 1811],  5.00th=[ 1893], 10.00th=[ 1926], 20.00th=[ 1991],
     | 30.00th=[ 2024], 40.00th=[ 2073], 50.00th=[ 2114], 60.00th=[ 2180],
     | 70.00th=[ 2212], 80.00th=[ 2278], 90.00th=[ 2442], 95.00th=[ 2999],
     | 99.00th=[ 4228], 99.50th=[ 5080], 99.90th=[12649], 99.95th=[25822],
     | 99.99th=[27919]
   bw (  KiB/s): min=379872, max=492384, per=100.00%, avg=453968.00, stdev=50389.60, samples=4
   iops        : min=23742, max=30774, avg=28373.00, stdev=3149.35, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=23.73%, 4=74.97%, 10=1.07%, 20=0.10%, 50=0.10%
  cpu          : usr=25.93%, sys=51.21%, ctx=65724, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=441MiB/s (462MB/s), 441MiB/s-441MiB/s (462MB/s-462MB/s), io=1024MiB (1074MB), run=2323-2323msec

Disk stats (read/write):
  sda: ios=0/60505, merge=0/0, ticks=0/4917, in_queue=5700, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 18 MiB ( 1024 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 18 MiB ( 0 MiB / 0 MiB )
async_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K: (groupid=0, jobs=1): err= 0: pid=941: Sun Aug  8 20:22:18 2021
  read: IOPS=60.0k, BW=953MiB/s (999MB/s)(1024MiB/1075msec)
    slat (usec): min=4, max=209, avg= 9.06, stdev= 4.55
    clat (usec): min=341, max=6859, avg=1033.70, stdev=192.85
     lat (usec): min=351, max=6874, avg=1043.96, stdev=193.69
    clat percentiles (usec):
     |  1.00th=[  840],  5.00th=[  889], 10.00th=[  914], 20.00th=[  938],
     | 30.00th=[  963], 40.00th=[  979], 50.00th=[  996], 60.00th=[ 1020],
     | 70.00th=[ 1045], 80.00th=[ 1090], 90.00th=[ 1188], 95.00th=[ 1237],
     | 99.00th=[ 1745], 99.50th=[ 2311], 99.90th=[ 3425], 99.95th=[ 3818],
     | 99.99th=[ 4359]
   bw (  KiB/s): min=974976, max=982464, per=100.00%, avg=978720.00, stdev=5294.82, samples=2
   iops        : min=60936, max=61404, avg=61170.00, stdev=330.93, samples=2
  lat (usec)   : 500=0.03%, 750=0.25%, 1000=51.70%
  lat (msec)   : 2=47.33%, 4=0.67%, 10=0.03%
  cpu          : usr=40.69%, sys=58.85%, ctx=44, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=953MiB/s (999MB/s), 953MiB/s-953MiB/s (999MB/s-999MB/s), io=1024MiB (1074MB), run=1075-1075msec

Disk stats (read/write):
  sda: ios=54653/0, merge=0/0, ticks=16639/0, in_queue=16688, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 1806 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
1 min passed...2624 MiB / 3157 MiB / 2586 MiB / 0 MiB / 14 MiB ( 2624 MiB / 3157 MiB / 2585 MiB / 0 MiB / 14 MiB )
2 min passed...2624 MiB / 4053 MiB / 2588 MiB / 0 MiB / 14 MiB ( 0 MiB / 896 MiB / 2 MiB / 0 MiB / 0 MiB )
3 min passed...2624 MiB / 4309 MiB / 2588 MiB / 0 MiB / 14 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...2624 MiB / 4992 MiB / 2588 MiB / 0 MiB / 14 MiB ( 0 MiB / 683 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2624 MiB / 5248 MiB / 2588 MiB / 0 MiB / 14 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2624 MiB / 5504 MiB / 2591 MiB / 0 MiB / 14 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
7 min passed...2624 MiB / 6314 MiB / 2591 MiB / 0 MiB / 14 MiB ( 0 MiB / 810 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2624 MiB / 6613 MiB / 2593 MiB / 0 MiB / 14 MiB ( 0 MiB / 299 MiB / 2 MiB / 0 MiB / 0 MiB )
9 min passed...2624 MiB / 6613 MiB / 2597 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
10 min passed...2624 MiB / 7040 MiB / 2599 MiB / 0 MiB / 14 MiB ( 0 MiB / 427 MiB / 2 MiB / 0 MiB / 0 MiB )
11 min passed...2624 MiB / 8149 MiB / 2604 MiB / 0 MiB / 14 MiB ( 0 MiB / 1109 MiB / 5 MiB / 0 MiB / 0 MiB )
12 min passed...2624 MiB / 8490 MiB / 2606 MiB / 0 MiB / 14 MiB ( 0 MiB / 341 MiB / 2 MiB / 0 MiB / 0 MiB )
13 min passed...2656 MiB / 8490 MiB / 2611 MiB / 0 MiB / 14 MiB ( 32 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
14 min passed...2656 MiB / 8490 MiB / 2613 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
15 min passed...2656 MiB / 8704 MiB / 2614 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 214 MiB / 1 MiB / 1024 MiB / 1024 MiB )
16 min passed...2656 MiB / 9173 MiB / 2614 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 469 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2656 MiB / 10240 MiB / 2614 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 1067 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2656 MiB / 10752 MiB / 2615 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 512 MiB / 1 MiB / 0 MiB / 0 MiB )
19 min passed...2656 MiB / 10752 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
20 min passed...2656 MiB / 11050 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 298 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2656 MiB / 11648 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 598 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2656 MiB / 12202 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 554 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2656 MiB / 12288 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 86 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2656 MiB / 13098 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 810 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2656 MiB / 13141 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2656 MiB / 13354 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 213 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2656 MiB / 13952 MiB / 2618 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 598 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2656 MiB / 14037 MiB / 2620 MiB / 1024 MiB / 1038 MiB ( 0 MiB / 85 MiB / 2 MiB / 0 MiB / 0 MiB )
29 min passed...2688 MiB / 14037 MiB / 2624 MiB / 1024 MiB / 1038 MiB ( 32 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
30 min passed...2752 MiB / 14719 MiB / 2649 MiB / 1024 MiB / 1039 MiB ( 64 MiB / 682 MiB / 25 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 3: sync 4M sequential read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_6.sh
async_seq_4M: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_seq_4M: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1508MiB/s][w=377 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=647: Sun Aug  8 20:43:10 2021
  write: IOPS=117, BW=469MiB/s (492MB/s)(10.0GiB/21841msec); 0 zone resets
    slat (usec): min=89, max=20638, avg=230.89, stdev=474.25
    clat (usec): min=1545, max=7017.3k, avg=543587.86, stdev=1207290.70
     lat (usec): min=1709, max=7017.5k, avg=543821.02, stdev=1207268.61
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   72], 10.00th=[   78], 20.00th=[   79],
     | 30.00th=[   82], 40.00th=[   84], 50.00th=[   88], 60.00th=[  103],
     | 70.00th=[  157], 80.00th=[  215], 90.00th=[ 1754], 95.00th=[ 3775],
     | 99.00th=[ 5537], 99.50th=[ 5537], 99.90th=[ 7013], 99.95th=[ 7013],
     | 99.99th=[ 7013]
   bw (  MiB/s): min=    8, max= 3168, per=100.00%, avg=1182.00, stdev=998.01, samples=16
   iops        : min=    2, max=  792, avg=295.50, stdev=249.50, samples=16
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.20%, 20=0.12%, 50=0.70%
  lat (msec)   : 100=57.58%, 250=22.62%, 500=1.99%, 750=2.19%, 1000=1.13%
  cpu          : usr=13.87%, sys=2.77%, ctx=2030, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=469MiB/s (492MB/s), 469MiB/s-469MiB/s (492MB/s-492MB/s), io=10.0GiB (10.7GB), run=21841-21841msec

Disk stats (read/write):
  sda: ios=0/10161, merge=0/179, ticks=0/4304572, in_queue=588896, util=26.98%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 10241 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 20 MiB ( 0 MiB / 1 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 20 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 20 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 20 MiB ( 0 MiB / 0 MiB )
async_seq_4M: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1791MiB/s][r=447 IOPS][eta 00m:00s]
async_seq_4M: (groupid=0, jobs=1): err= 0: pid=919: Sun Aug  8 20:58:16 2021
  read: IOPS=452, BW=1810MiB/s (1898MB/s)(10.0GiB/5658msec)
    slat (usec): min=77, max=19858, avg=224.41, stdev=580.90
    clat (msec): min=16, max=307, avg=140.43, stdev=68.32
     lat (msec): min=16, max=307, avg=140.66, stdev=68.26
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   39], 10.00th=[   46], 20.00th=[   65],
     | 30.00th=[   80], 40.00th=[  116], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  199], 80.00th=[  211], 90.00th=[  215], 95.00th=[  222],
     | 99.00th=[  255], 99.50th=[  300], 99.90th=[  309], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  MiB/s): min= 1360, max= 3808, per=99.72%, avg=1804.84, stdev=684.47, samples=11
   iops        : min=  340, max=  952, avg=451.18, stdev=171.12, samples=11
  lat (msec)   : 20=1.25%, 50=10.51%, 100=24.69%, 250=62.38%, 500=1.17%
  cpu          : usr=0.41%, sys=10.43%, ctx=1726, majf=0, minf=649
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1810MiB/s (1898MB/s), 1810MiB/s-1810MiB/s (1898MB/s-1898MB/s), io=10.0GiB (10.7GB), run=5658-5658msec

Disk stats (read/write):
  sda: ios=10209/2, merge=0/1, ticks=1049938/152, in_queue=1051588, util=99.58%
0 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 10240 MiB )
1 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 10241 MiB / 10260 MiB ( 0 MiB / 0 MiB )
Script ran 1830 seconds or 30 minutes.

Host:
CPU usage: 1% to 8,5%; IO delay: 0% to 9,5%

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...31712 MiB / 34560 MiB / 31731 MiB / 128 MiB / 137 MiB ( 31712 MiB / 34560 MiB / 31731 MiB / 128 MiB / 137 MiB )
2 min passed...31712 MiB / 35242 MiB / 31732 MiB / 128 MiB / 137 MiB ( 0 MiB / 682 MiB / 1 MiB / 0 MiB / 0 MiB )
3 min passed...31744 MiB / 35285 MiB / 31739 MiB / 128 MiB / 137 MiB ( 32 MiB / 43 MiB / 7 MiB / 0 MiB / 0 MiB )
4 min passed...31744 MiB / 35413 MiB / 31739 MiB / 128 MiB / 137 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...31744 MiB / 35797 MiB / 31739 MiB / 128 MiB / 137 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...31744 MiB / 36650 MiB / 31739 MiB / 128 MiB / 137 MiB ( 0 MiB / 853 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...31744 MiB / 36992 MiB / 31741 MiB / 128 MiB / 137 MiB ( 0 MiB / 342 MiB / 2 MiB / 0 MiB / 0 MiB )
8 min passed...31744 MiB / 37674 MiB / 31742 MiB / 128 MiB / 137 MiB ( 0 MiB / 682 MiB / 1 MiB / 0 MiB / 0 MiB )
9 min passed...31744 MiB / 37973 MiB / 31742 MiB / 128 MiB / 137 MiB ( 0 MiB / 299 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...31744 MiB / 38058 MiB / 31742 MiB / 128 MiB / 137 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...31744 MiB / 38655 MiB / 31742 MiB / 128 MiB / 137 MiB ( 0 MiB / 597 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...31744 MiB / 38783 MiB / 31742 MiB / 128 MiB / 137 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...31744 MiB / 38783 MiB / 31748 MiB / 128 MiB / 137 MiB ( 0 MiB / 0 MiB / 6 MiB / 0 MiB / 0 MiB )
14 min passed...31744 MiB / 39423 MiB / 31748 MiB / 128 MiB / 137 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...31744 MiB / 40405 MiB / 31748 MiB / 128 MiB / 137 MiB ( 0 MiB / 982 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...31744 MiB / 40917 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 512 MiB / 4 MiB / 7936 MiB / 7913 MiB )
17 min passed...31744 MiB / 41045 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...31744 MiB / 41045 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...31744 MiB / 41429 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...31744 MiB / 42026 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 597 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...31744 MiB / 43050 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 1024 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...31744 MiB / 43135 MiB / 31752 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...31744 MiB / 43221 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 86 MiB / 3 MiB / 0 MiB / 0 MiB )
24 min passed...31744 MiB / 43605 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...31744 MiB / 44330 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...31744 MiB / 44671 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...31744 MiB / 45141 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 470 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...31744 MiB / 45610 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 469 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...31744 MiB / 45695 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...31744 MiB / 46207 MiB / 31755 MiB / 8064 MiB / 8050 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )

Comment:
Its strange that the write performance is so low.

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 7: async 16K random read/write but 50% compressible

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_7.sh
async_rand_16K_comp50: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_16K_comp50: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=654: Sun Aug  8 21:19:10 2021
  write: IOPS=28.8k, BW=450MiB/s (472MB/s)(1024MiB/2277msec); 0 zone resets
    slat (usec): min=7, max=6014, avg=22.17, stdev=31.64
    clat (usec): min=51, max=18572, avg=2189.38, stdev=619.97
     lat (usec): min=72, max=18592, avg=2212.77, stdev=625.16
    clat percentiles (usec):
     |  1.00th=[ 1696],  5.00th=[ 1844], 10.00th=[ 1860], 20.00th=[ 1909],
     | 30.00th=[ 1942], 40.00th=[ 1975], 50.00th=[ 2024], 60.00th=[ 2089],
     | 70.00th=[ 2180], 80.00th=[ 2311], 90.00th=[ 2638], 95.00th=[ 3163],
     | 99.00th=[ 4621], 99.50th=[ 5538], 99.90th=[10028], 99.95th=[10421],
     | 99.99th=[17957]
   bw (  KiB/s): min=379296, max=521152, per=100.00%, avg=467104.00, stdev=64246.06, samples=4
   iops        : min=23706, max=32572, avg=29194.00, stdev=4015.38, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=44.92%, 4=53.49%, 10=1.44%, 20=0.11%
  cpu          : usr=21.75%, sys=55.10%, ctx=67448, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=450MiB/s (472MB/s), 450MiB/s-450MiB/s (472MB/s-472MB/s), io=1024MiB (1074MB), run=2277-2277msec

Disk stats (read/write):
  sda: ios=0/61571, merge=0/0, ticks=0/4103, in_queue=4896, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 2 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_16K_comp50: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
async_rand_16K_comp50: (groupid=0, jobs=1): err= 0: pid=870: Sun Aug  8 21:34:12 2021
  read: IOPS=61.5k, BW=961MiB/s (1007MB/s)(1024MiB/1066msec)
    slat (usec): min=4, max=156, avg= 8.93, stdev= 5.70
    clat (usec): min=293, max=3504, avg=1025.10, stdev=157.22
     lat (usec): min=300, max=3515, avg=1035.27, stdev=158.72
    clat percentiles (usec):
     |  1.00th=[  848],  5.00th=[  881], 10.00th=[  898], 20.00th=[  922],
     | 30.00th=[  947], 40.00th=[  963], 50.00th=[  979], 60.00th=[ 1012],
     | 70.00th=[ 1057], 80.00th=[ 1123], 90.00th=[ 1188], 95.00th=[ 1237],
     | 99.00th=[ 1565], 99.50th=[ 2073], 99.90th=[ 2409], 99.95th=[ 2802],
     | 99.99th=[ 2999]
   bw (  KiB/s): min=908608, max=1054144, per=99.77%, avg=981376.00, stdev=102909.49, samples=2
   iops        : min=56788, max=65884, avg=61336.00, stdev=6431.84, samples=2
  lat (usec)   : 500=0.02%, 750=0.03%, 1000=57.16%
  lat (msec)   : 2=42.05%, 4=0.75%
  cpu          : usr=43.19%, sys=56.81%, ctx=13, majf=0, minf=265
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=961MiB/s (1007MB/s), 961MiB/s-961MiB/s (1007MB/s-1007MB/s), io=1024MiB (1074MB), run=1066-1066msec

Disk stats (read/write):
  sda: ios=54630/0, merge=0/0, ticks=14189/0, in_queue=14168, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1026 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2080 MiB / 2091 MiB / 2072 MiB / 0 MiB / 14 MiB ( 2080 MiB / 2091 MiB / 2072 MiB / 0 MiB / 14 MiB )
2 min passed...2080 MiB / 2389 MiB / 2074 MiB / 0 MiB / 14 MiB ( 0 MiB / 298 MiB / 2 MiB / 0 MiB / 0 MiB )
3 min passed...2080 MiB / 2987 MiB / 2075 MiB / 0 MiB / 14 MiB ( 0 MiB / 598 MiB / 1 MiB / 0 MiB / 0 MiB )
4 min passed...2080 MiB / 4096 MiB / 2075 MiB / 0 MiB / 14 MiB ( 0 MiB / 1109 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2080 MiB / 4608 MiB / 2075 MiB / 0 MiB / 14 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2080 MiB / 4608 MiB / 2080 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 5 MiB / 0 MiB / 0 MiB )
7 min passed...2080 MiB / 4907 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 299 MiB / 5 MiB / 0 MiB / 0 MiB )
8 min passed...2080 MiB / 5461 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 554 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2080 MiB / 5717 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2080 MiB / 5888 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2080 MiB / 6656 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2080 MiB / 6656 MiB / 2085 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2080 MiB / 7424 MiB / 2087 MiB / 0 MiB / 14 MiB ( 0 MiB / 768 MiB / 2 MiB / 0 MiB / 0 MiB )
14 min passed...2080 MiB / 7893 MiB / 2091 MiB / 0 MiB / 14 MiB ( 0 MiB / 469 MiB / 4 MiB / 0 MiB / 0 MiB )
15 min passed...2080 MiB / 7893 MiB / 2095 MiB / 768 MiB / 782 MiB ( 0 MiB / 0 MiB / 4 MiB / 768 MiB / 768 MiB )
16 min passed...2080 MiB / 8064 MiB / 2097 MiB / 768 MiB / 782 MiB ( 0 MiB / 171 MiB / 2 MiB / 0 MiB / 0 MiB )
17 min passed...2080 MiB / 8917 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 853 MiB / 3 MiB / 0 MiB / 0 MiB )
18 min passed...2080 MiB / 9685 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2080 MiB / 9856 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2080 MiB / 9856 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2080 MiB / 10112 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2080 MiB / 10453 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...2080 MiB / 11349 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 896 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...2080 MiB / 12075 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 726 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2080 MiB / 12075 MiB / 2100 MiB / 768 MiB / 782 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2080 MiB / 12117 MiB / 2102 MiB / 768 MiB / 782 MiB ( 0 MiB / 42 MiB / 2 MiB / 0 MiB / 0 MiB )
27 min passed...2080 MiB / 12416 MiB / 2107 MiB / 768 MiB / 782 MiB ( 0 MiB / 299 MiB / 5 MiB / 0 MiB / 0 MiB )
28 min passed...2080 MiB / 12459 MiB / 2107 MiB / 768 MiB / 782 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2080 MiB / 12459 MiB / 2107 MiB / 768 MiB / 782 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2080 MiB / 13227 MiB / 2132 MiB / 768 MiB / 783 MiB ( 0 MiB / 768 MiB / 25 MiB / 0 MiB / 1 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 8: sync 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_8.sh
sync_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=10.5MiB/s][w=335 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=651: Sun Aug  8 21:58:59 2021
  write: IOPS=312, BW=9.78MiB/s (10.3MB/s)(1024MiB/104735msec); 0 zone resets
    clat (usec): min=1839, max=40643, avg=3168.79, stdev=970.29
     lat (usec): min=1841, max=40646, avg=3171.16, stdev=970.33
    clat percentiles (usec):
     |  1.00th=[ 2376],  5.00th=[ 2606], 10.00th=[ 2704], 20.00th=[ 2835],
     | 30.00th=[ 2933], 40.00th=[ 2999], 50.00th=[ 3097], 60.00th=[ 3163],
     | 70.00th=[ 3228], 80.00th=[ 3326], 90.00th=[ 3458], 95.00th=[ 3654],
     | 99.00th=[ 5276], 99.50th=[ 7504], 99.90th=[14484], 99.95th=[22414],
     | 99.99th=[36439]
   bw (  KiB/s): min= 7936, max=12288, per=99.97%, avg=10007.63, stdev=740.07, samples=209
   iops        : min=  248, max=  384, avg=312.72, stdev=23.13, samples=209
  lat (msec)   : 2=0.02%, 4=96.67%, 10=2.87%, 20=0.38%, 50=0.06%
  cpu          : usr=1.13%, sys=4.27%, ctx=76434, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9.78MiB/s (10.3MB/s), 9.78MiB/s-9.78MiB/s (10.3MB/s-10.3MB/s), io=1024MiB (1074MB), run=104735-104735msec

Disk stats (read/write):
  sda: ios=0/98705, merge=0/66643, ticks=0/96378, in_queue=104356, util=99.61%
0 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 1543 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1543 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=46.4MiB/s][r=1484 IOPS][eta 00m:00s]
sync_rand_32K: (groupid=0, jobs=1): err= 0: pid=854: Sun Aug  8 22:14:20 2021
  read: IOPS=1570, BW=49.1MiB/s (51.5MB/s)(1024MiB/20865msec)
    clat (usec): min=381, max=6013, avg=625.31, stdev=204.74
     lat (usec): min=383, max=6015, avg=627.33, stdev=204.76
    clat percentiles (usec):
     |  1.00th=[  449],  5.00th=[  486], 10.00th=[  510], 20.00th=[  545],
     | 30.00th=[  570], 40.00th=[  586], 50.00th=[  611], 60.00th=[  635],
     | 70.00th=[  652], 80.00th=[  676], 90.00th=[  709], 95.00th=[  734],
     | 99.00th=[ 1004], 99.50th=[ 2278], 99.90th=[ 3359], 99.95th=[ 3523],
     | 99.99th=[ 5014]
   bw (  KiB/s): min=45632, max=56832, per=100.00%, avg=50255.29, stdev=2627.08, samples=41
   iops        : min= 1426, max= 1776, avg=1570.41, stdev=82.10, samples=41
  lat (usec)   : 500=7.49%, 750=89.48%, 1000=2.03%
  lat (msec)   : 2=0.42%, 4=0.56%, 10=0.02%
  cpu          : usr=3.10%, sys=9.48%, ctx=32782, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=49.1MiB/s (51.5MB/s), 49.1MiB/s-49.1MiB/s (51.5MB/s-51.5MB/s), io=1024MiB (1074MB), run=20865-20865msec

Disk stats (read/write):
  sda: ios=32436/0, merge=0/0, ticks=18826/0, in_queue=20748, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1543 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1927 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...6528 MiB / 6869 MiB / 6564 MiB / 288 MiB / 235 MiB ( 6528 MiB / 6869 MiB / 6564 MiB / 288 MiB / 235 MiB )
2 min passed...11552 MiB / 12330 MiB / 11592 MiB / 352 MiB / 376 MiB ( 5024 MiB / 5461 MiB / 5028 MiB / 64 MiB / 141 MiB )
3 min passed...11552 MiB / 12586 MiB / 11592 MiB / 352 MiB / 376 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...11552 MiB / 13397 MiB / 11595 MiB / 352 MiB / 376 MiB ( 0 MiB / 811 MiB / 3 MiB / 0 MiB / 0 MiB )
5 min passed...11552 MiB / 14250 MiB / 11595 MiB / 352 MiB / 376 MiB ( 0 MiB / 853 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...11552 MiB / 14591 MiB / 11595 MiB / 352 MiB / 376 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...11552 MiB / 14634 MiB / 11595 MiB / 352 MiB / 376 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...11552 MiB / 14719 MiB / 11599 MiB / 352 MiB / 376 MiB ( 0 MiB / 85 MiB / 4 MiB / 0 MiB / 0 MiB )
9 min passed...11552 MiB / 15402 MiB / 11599 MiB / 352 MiB / 376 MiB ( 0 MiB / 683 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...11552 MiB / 15445 MiB / 11599 MiB / 352 MiB / 376 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...11552 MiB / 15701 MiB / 11602 MiB / 352 MiB / 376 MiB ( 0 MiB / 256 MiB / 3 MiB / 0 MiB / 0 MiB )
12 min passed...11552 MiB / 16511 MiB / 11602 MiB / 352 MiB / 376 MiB ( 0 MiB / 810 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...11584 MiB / 16853 MiB / 11605 MiB / 352 MiB / 376 MiB ( 32 MiB / 342 MiB / 3 MiB / 0 MiB / 0 MiB )
14 min passed...11584 MiB / 17237 MiB / 11605 MiB / 352 MiB / 376 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...11584 MiB / 17407 MiB / 11605 MiB / 352 MiB / 376 MiB ( 0 MiB / 170 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...11584 MiB / 17407 MiB / 11605 MiB / 352 MiB / 376 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...11584 MiB / 17749 MiB / 11605 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 342 MiB / 0 MiB / 1088 MiB / 1024 MiB )
18 min passed...11584 MiB / 18815 MiB / 11609 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 1066 MiB / 4 MiB / 0 MiB / 0 MiB )
19 min passed...11584 MiB / 19541 MiB / 11609 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 726 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...11616 MiB / 19626 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 32 MiB / 85 MiB / 4 MiB / 0 MiB / 0 MiB )
21 min passed...11616 MiB / 19669 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...11616 MiB / 20053 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...11616 MiB / 20565 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...11616 MiB / 21162 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 597 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...11616 MiB / 21717 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 555 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...11616 MiB / 21759 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 42 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...11616 MiB / 21759 MiB / 11613 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...11616 MiB / 22698 MiB / 11616 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 939 MiB / 3 MiB / 0 MiB / 0 MiB )
29 min passed...11616 MiB / 22954 MiB / 11618 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 256 MiB / 2 MiB / 0 MiB / 0 MiB )
30 min passed...11616 MiB / 22954 MiB / 11619 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 0 MiB / 1 MiB / 0 MiB / 0 MiB )
31 min passed...11616 MiB / 23935 MiB / 11619 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 981 MiB / 0 MiB / 0 MiB / 0 MiB )
32 min passed...11616 MiB / 24405 MiB / 11645 MiB / 1440 MiB / 1400 MiB ( 0 MiB / 470 MiB / 26 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 8, 2021

Round 2 - Run 2 - Test 9: async 32K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_9.sh
async_rand_32K: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
async_rand_32K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=647: Sun Aug  8 22:32:58 2021
  write: IOPS=24.6k, BW=769MiB/s (807MB/s)(1024MiB/1331msec); 0 zone resets
    slat (usec): min=8, max=3223, avg=20.70, stdev=26.71
    clat (usec): min=128, max=9157, avg=2555.51, stdev=765.22
     lat (usec): min=138, max=9286, avg=2577.61, stdev=771.44
    clat percentiles (usec):
     |  1.00th=[ 1942],  5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2147],
     | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2343], 60.00th=[ 2409],
     | 70.00th=[ 2540], 80.00th=[ 2737], 90.00th=[ 3261], 95.00th=[ 3785],
     | 99.00th=[ 6063], 99.50th=[ 7832], 99.90th=[ 8586], 99.95th=[ 8717],
     | 99.99th=[ 8848]
   bw (  KiB/s): min=804160, max=875776, per=100.00%, avg=839968.00, stdev=50640.16, samples=2
   iops        : min=25130, max=27368, avg=26249.00, stdev=1582.50, samples=2
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=3.05%, 4=92.80%, 10=4.08%
  cpu          : usr=27.67%, sys=52.56%, ctx=32957, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=769MiB/s (807MB/s), 769MiB/s-769MiB/s (807MB/s-807MB/s), io=1024MiB (1074MB), run=1331-1331msec

Disk stats (read/write):
  sda: ios=0/28461, merge=0/0, ticks=0/4067, in_queue=4296, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1024 MiB / 19 MiB ( 1024 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 19 MiB ( 0 MiB / 0 MiB )
async_rand_32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=0)
async_rand_32K: (groupid=0, jobs=1): err= 0: pid=863: Sun Aug  8 22:48:00 2021
  read: IOPS=36.8k, BW=1151MiB/s (1206MB/s)(1024MiB/890msec)
    slat (usec): min=5, max=424, avg=15.56, stdev=16.59
    clat (usec): min=351, max=5077, avg=1710.08, stdev=416.72
     lat (usec): min=396, max=5089, avg=1727.38, stdev=419.23
    clat percentiles (usec):
     |  1.00th=[  947],  5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1303],
     | 30.00th=[ 1467], 40.00th=[ 1680], 50.00th=[ 1795], 60.00th=[ 1860],
     | 70.00th=[ 1926], 80.00th=[ 1991], 90.00th=[ 2073], 95.00th=[ 2180],
     | 99.00th=[ 3097], 99.50th=[ 3752], 99.90th=[ 4424], 99.95th=[ 4555],
     | 99.99th=[ 4817]
   bw (  MiB/s): min= 1255, max= 1255, per=100.00%, avg=1255.88, stdev= 0.00, samples=1
   iops        : min=40188, max=40188, avg=40188.00, stdev= 0.00, samples=1
  lat (usec)   : 500=0.01%, 750=0.15%, 1000=1.79%
  lat (msec)   : 2=79.41%, 4=18.32%, 10=0.32%
  cpu          : usr=32.17%, sys=65.02%, ctx=239, majf=0, minf=521
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1151MiB/s (1206MB/s), 1151MiB/s-1151MiB/s (1206MB/s-1206MB/s), io=1024MiB (1074MB), run=890-890msec

Disk stats (read/write):
  sda: ios=24937/0, merge=0/0, ticks=16827/0, in_queue=16628, util=100.00%
0 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 1025 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1805 seconds or 30 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...2528 MiB / 3029 MiB / 2531 MiB / 0 MiB / 14 MiB ( 2528 MiB / 3029 MiB / 2531 MiB / 0 MiB / 14 MiB )
2 min passed...2528 MiB / 4266 MiB / 2531 MiB / 0 MiB / 14 MiB ( 0 MiB / 1237 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...2528 MiB / 4736 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 470 MiB / 7 MiB / 0 MiB / 0 MiB )
4 min passed...2528 MiB / 4736 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...2528 MiB / 4821 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...2528 MiB / 5205 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...2528 MiB / 5930 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...2528 MiB / 6400 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 470 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...2528 MiB / 6912 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...2528 MiB / 7040 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 128 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...2528 MiB / 7424 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
12 min passed...2528 MiB / 8192 MiB / 2538 MiB / 0 MiB / 14 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...2528 MiB / 8405 MiB / 2542 MiB / 0 MiB / 14 MiB ( 0 MiB / 213 MiB / 4 MiB / 0 MiB / 0 MiB )
14 min passed...2528 MiB / 8405 MiB / 2545 MiB / 0 MiB / 14 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
15 min passed...2528 MiB / 9301 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 896 MiB / 0 MiB / 1024 MiB / 1025 MiB )
16 min passed...2528 MiB / 9472 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...2528 MiB / 10026 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 554 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...2528 MiB / 10069 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...2528 MiB / 10069 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...2528 MiB / 10410 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...2528 MiB / 11562 MiB / 2545 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 1152 MiB / 0 MiB / 0 MiB / 0 MiB )
22 min passed...2528 MiB / 12586 MiB / 2547 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 1024 MiB / 2 MiB / 0 MiB / 0 MiB )
23 min passed...2528 MiB / 12586 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 0 MiB / 2 MiB / 0 MiB / 0 MiB )
24 min passed...2528 MiB / 12672 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 86 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...2528 MiB / 12970 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 298 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...2528 MiB / 13482 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 512 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...2528 MiB / 13695 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 213 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...2528 MiB / 14421 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 726 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...2528 MiB / 14506 MiB / 2549 MiB / 1024 MiB / 1039 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...2560 MiB / 14805 MiB / 2574 MiB / 1024 MiB / 1039 MiB ( 32 MiB / 299 MiB / 25 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Round 2 - Run 2 - Test 0: VM just idleing for 1 hour

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_0.sh
0 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
1 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
16 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
17 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
18 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
19 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
20 min passed...Guest Writes/Reads (Changes): 0 MiB / 0 MiB ( 0 MiB / 0 MiB )
21 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 1 MiB / 0 MiB )
22 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
23 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
24 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
25 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
26 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
27 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
28 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
29 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
30 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
31 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
32 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
33 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
34 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
35 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
36 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
37 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
38 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
39 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
40 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
41 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
42 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
43 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
44 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
45 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
46 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
47 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
48 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
49 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
50 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
51 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
52 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
53 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
54 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
55 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
56 min passed...Guest Writes/Reads (Changes): 1 MiB / 0 MiB ( 0 MiB / 0 MiB )
57 min passed...Guest Writes/Reads (Changes): 4 MiB / 4 MiB ( 3 MiB / 4 MiB )
58 min passed...Guest Writes/Reads (Changes): 4 MiB / 4 MiB ( 0 MiB / 0 MiB )
59 min passed...Guest Writes/Reads (Changes): 4 MiB / 4 MiB ( 0 MiB / 0 MiB )
60 min passed...Guest Writes/Reads (Changes): 4 MiB / 4 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 4 MiB / 4 MiB ( 0 MiB / 0 MiB )
Script ran 3601 seconds or 60 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_6.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 896 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 896 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...0 MiB / 1152 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 4 MiB / 0 MiB / 0 MiB )
2 min passed...0 MiB / 1152 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
3 min passed...0 MiB / 1792 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
4 min passed...0 MiB / 2176 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
5 min passed...0 MiB / 2816 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 640 MiB / 0 MiB / 0 MiB / 0 MiB )
6 min passed...0 MiB / 2816 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
7 min passed...0 MiB / 2816 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
8 min passed...0 MiB / 3072 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
9 min passed...0 MiB / 3797 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 725 MiB / 0 MiB / 0 MiB / 0 MiB )
10 min passed...0 MiB / 4821 MiB / 4 MiB / 0 MiB / 0 MiB ( 0 MiB / 1024 MiB / 0 MiB / 0 MiB / 0 MiB )
11 min passed...0 MiB / 4821 MiB / 7 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
12 min passed...0 MiB / 4821 MiB / 7 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
13 min passed...0 MiB / 5162 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 341 MiB / 4 MiB / 0 MiB / 0 MiB )
14 min passed...0 MiB / 5845 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 683 MiB / 0 MiB / 0 MiB / 0 MiB )
15 min passed...0 MiB / 6400 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 555 MiB / 0 MiB / 0 MiB / 0 MiB )
16 min passed...0 MiB / 7168 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
17 min passed...0 MiB / 7338 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 170 MiB / 0 MiB / 0 MiB / 0 MiB )
18 min passed...0 MiB / 7381 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 43 MiB / 0 MiB / 0 MiB / 0 MiB )
19 min passed...0 MiB / 8149 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
20 min passed...0 MiB / 8234 MiB / 11 MiB / 0 MiB / 0 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
21 min passed...0 MiB / 8234 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 3 MiB / 0 MiB / 0 MiB )
22 min passed...0 MiB / 8832 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 598 MiB / 0 MiB / 0 MiB / 0 MiB )
23 min passed...0 MiB / 9685 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 853 MiB / 0 MiB / 0 MiB / 0 MiB )
24 min passed...0 MiB / 10453 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 768 MiB / 0 MiB / 0 MiB / 0 MiB )
25 min passed...0 MiB / 10624 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
26 min passed...0 MiB / 10624 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
27 min passed...0 MiB / 10880 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
28 min passed...0 MiB / 11306 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 426 MiB / 0 MiB / 0 MiB / 0 MiB )
29 min passed...0 MiB / 12714 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 1408 MiB / 0 MiB / 0 MiB / 0 MiB )
30 min passed...0 MiB / 12799 MiB / 14 MiB / 0 MiB / 0 MiB ( 0 MiB / 85 MiB / 0 MiB / 0 MiB / 0 MiB )
31 min passed...0 MiB / 12799 MiB / 18 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 4 MiB / 0 MiB / 0 MiB )
32 min passed...0 MiB / 13098 MiB / 18 MiB / 0 MiB / 0 MiB ( 0 MiB / 299 MiB / 0 MiB / 0 MiB / 0 MiB )
33 min passed...0 MiB / 13482 MiB / 18 MiB / 0 MiB / 0 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
34 min passed...0 MiB / 13909 MiB / 18 MiB / 0 MiB / 0 MiB ( 0 MiB / 427 MiB / 0 MiB / 0 MiB / 0 MiB )
35 min passed...0 MiB / 14506 MiB / 20 MiB / 0 MiB / 0 MiB ( 0 MiB / 597 MiB / 2 MiB / 0 MiB / 0 MiB )
36 min passed...0 MiB / 14933 MiB / 21 MiB / 0 MiB / 0 MiB ( 0 MiB / 427 MiB / 1 MiB / 0 MiB / 0 MiB )
37 min passed...0 MiB / 14933 MiB / 21 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
38 min passed...0 MiB / 15615 MiB / 21 MiB / 0 MiB / 0 MiB ( 0 MiB / 682 MiB / 0 MiB / 0 MiB / 0 MiB )
39 min passed...0 MiB / 15786 MiB / 21 MiB / 0 MiB / 0 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
40 min passed...0 MiB / 15786 MiB / 21 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
41 min passed...0 MiB / 16085 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 299 MiB / 4 MiB / 0 MiB / 0 MiB )
42 min passed...0 MiB / 17279 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 1194 MiB / 0 MiB / 0 MiB / 0 MiB )
43 min passed...0 MiB / 18090 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 811 MiB / 0 MiB / 0 MiB / 0 MiB )
44 min passed...0 MiB / 18346 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 256 MiB / 0 MiB / 0 MiB / 0 MiB )
45 min passed...0 MiB / 18346 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
46 min passed...0 MiB / 18517 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 171 MiB / 0 MiB / 0 MiB / 0 MiB )
47 min passed...0 MiB / 18858 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 341 MiB / 0 MiB / 0 MiB / 0 MiB )
48 min passed...0 MiB / 20053 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 1195 MiB / 0 MiB / 0 MiB / 0 MiB )
49 min passed...0 MiB / 20522 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 469 MiB / 0 MiB / 0 MiB / 0 MiB )
50 min passed...0 MiB / 20522 MiB / 25 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
51 min passed...0 MiB / 20735 MiB / 29 MiB / 0 MiB / 1 MiB ( 0 MiB / 213 MiB / 4 MiB / 0 MiB / 1 MiB )
52 min passed...0 MiB / 21205 MiB / 29 MiB / 0 MiB / 1 MiB ( 0 MiB / 470 MiB / 0 MiB / 0 MiB / 0 MiB )
53 min passed...0 MiB / 21759 MiB / 29 MiB / 0 MiB / 1 MiB ( 0 MiB / 554 MiB / 0 MiB / 0 MiB / 0 MiB )
54 min passed...0 MiB / 22143 MiB / 29 MiB / 0 MiB / 1 MiB ( 0 MiB / 384 MiB / 0 MiB / 0 MiB / 0 MiB )
55 min passed...0 MiB / 22741 MiB / 32 MiB / 0 MiB / 1 MiB ( 0 MiB / 598 MiB / 3 MiB / 0 MiB / 0 MiB )
56 min passed...32 MiB / 22783 MiB / 64 MiB / 0 MiB / 4 MiB ( 32 MiB / 42 MiB / 32 MiB / 0 MiB / 3 MiB )
57 min passed...32 MiB / 23381 MiB / 65 MiB / 0 MiB / 4 MiB ( 0 MiB / 598 MiB / 1 MiB / 0 MiB / 0 MiB )
58 min passed...32 MiB / 23679 MiB / 65 MiB / 0 MiB / 4 MiB ( 0 MiB / 298 MiB / 0 MiB / 0 MiB / 0 MiB )
59 min passed...32 MiB / 23679 MiB / 65 MiB / 0 MiB / 4 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
60 min passed...32 MiB / 23765 MiB / 73 MiB / 0 MiB / 4 MiB ( 0 MiB / 86 MiB / 8 MiB / 0 MiB / 0 MiB )

Dunuin · Aug 9, 2021

Here are the diagrams for run 2 (8 disk striped mirror):

Total Write Amplification:

R2T1: sync 4K rand	61,00	7,13	4,00
R2T2: sync 16K rand	20,12	0,75	0,94
R2T3: sync 4M seq	5,02	0,01	1,02
R2T4: async 4K rand	11,50	4,03	3,97
R2T5: async 16K rand	8,29	0,00	1,00
R2T6: async 4M seq	3,95	0,01	0,78
R2T7: async 16K rand 50% cmp	7,71	0,00	0,75
R2T8: sync 32K rand	17,00	0,34	1,06
R2T9: async 32K rand	8,21	0,00	1,00

Detailed Write Amplification:

	Write Amplification Fio -> Guest	Write Amplification Guest -> Host	Write Amplification Host -> NAND
R2T1: sync 4K rand	5,15	9,48	1,25
R2T2: sync 16K rand	2,01	6,32	1,58
R2T3: sync 4M seq	1,00	4,13	1,21
R2T4: async 4K rand	1,01	4,88	2,36
R2T5: async 16K rand	1,00	2,55	3,20
R2T6: async 4M seq	1,00	3,10	1,27
R2T7: async 16K rand 50% cmp	1,00	2,04	3,79
R2T8: sync 32K rand	1,51	7,52	1,50
R2T9: async 32K rand	1,00	2,48	3,32

Performance:

	Write Throughput (MiB/s)	Read Throughput (MiB/s)
R2T1: sync 4K rand	1,14	6,62
R2T2: sync 16K rand	5,37	27,80
R2T3: sync 4M seq	210,00	986,00
R2T4: async 4K rand	93,20	260,00
R2T5: async 16K rand	441,00	953,00
R2T6: async 4M seq	469,00	1.810,00
R2T7: async 16K rand 50% cmp	450,00	961,00
R2T8: sync 32K rand	9,78	49,10
R2T9: async 32K rand	769,00	1.151,00

Dunuin · Aug 9, 2021

This is the pool and VM that will be used in Round 2 Run 3:

Host is mirror of 2x S3710 200GB. atime=off, ashift=12, thin, compression=lz4, encryption=aes-256-gcm, volblocksize=4K, primarycache=metadata.
Guest is a Debian 10 with ext4, ext4 parameters: default+noatime+nodiratime. cachemode=none, io threat=yes, discard=yes, ssd emulation=yes, virtio SCSI, SCSI, virtio blocksize=4K.
(differences to Round 2 Run 2 are bold)

root@Hypervisor:~# zpool status

Code:

  pool: VMpool10
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMpool10                                        ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN  ONLINE       0     0     0
            ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN  ONLINE       0     0     0

errors: No known data errors

root@Hypervisor:~# zpool get all

Code:

NAME      PROPERTY                       VALUE                          SOURCE
VMpool10  size                           186G                           -
VMpool10  capacity                       0%                             -
VMpool10  altroot                        -                              default
VMpool10  health                         ONLINE                         -
VMpool10  guid                           8860610226873824977            -
VMpool10  version                        -                              default
VMpool10  bootfs                         -                              default
VMpool10  delegation                     on                             default
VMpool10  autoreplace                    off                            default
VMpool10  cachefile                      -                              default
VMpool10  failmode                       wait                           default
VMpool10  listsnapshots                  off                            default
VMpool10  autoexpand                     off                            default
VMpool10  dedupratio                     1.00x                          -
VMpool10  free                           186G                           -
VMpool10  allocated                      1.10M                          -
VMpool10  readonly                       off                            -
VMpool10  ashift                         12                             local
VMpool10  comment                        -                              default
VMpool10  expandsize                     -                              -
VMpool10  freeing                        0                              -
VMpool10  fragmentation                  0%                             -
VMpool10  leaked                         0                              -
VMpool10  multihost                      off                            default
VMpool10  checkpoint                     -                              -
VMpool10  load_guid                      8941709205797868984            -
VMpool10  autotrim                       off                            default
VMpool10  feature@async_destroy          enabled                        local
VMpool10  feature@empty_bpobj            active                         local
VMpool10  feature@lz4_compress           active                         local
VMpool10  feature@multi_vdev_crash_dump  enabled                        local
VMpool10  feature@spacemap_histogram     active                         local
VMpool10  feature@enabled_txg            active                         local
VMpool10  feature@hole_birth             active                         local
VMpool10  feature@extensible_dataset     active                         local
VMpool10  feature@embedded_data          active                         local
VMpool10  feature@bookmarks              enabled                        local
VMpool10  feature@filesystem_limits      enabled                        local
VMpool10  feature@large_blocks           enabled                        local
VMpool10  feature@large_dnode            enabled                        local
VMpool10  feature@sha512                 enabled                        local
VMpool10  feature@skein                  enabled                        local
VMpool10  feature@edonr                  enabled                        local
VMpool10  feature@userobj_accounting     active                         local
VMpool10  feature@encryption             active                         local
VMpool10  feature@project_quota          active                         local
VMpool10  feature@device_removal         enabled                        local
VMpool10  feature@obsolete_counts        enabled                        local
VMpool10  feature@zpool_checkpoint       enabled                        local
VMpool10  feature@spacemap_v2            active                         local
VMpool10  feature@allocation_classes     enabled                        local
VMpool10  feature@resilver_defer         enabled                        local
VMpool10  feature@bookmark_v2            enabled                        local
VMpool10  feature@redaction_bookmarks    enabled                        local
VMpool10  feature@redacted_datasets      enabled                        local
VMpool10  feature@bookmark_written       enabled                        local
VMpool10  feature@log_spacemap           active                         local
VMpool10  feature@livelist               enabled                        local
VMpool10  feature@device_rebuild         enabled                        local
VMpool10  feature@zstd_compress          enabled                        local

root@Hypervisor:~# zfs get all VMpool10/VLT/VM

Code:

NAME             PROPERTY              VALUE                  SOURCE
VMpool10/VLT/VM  type                  filesystem             -
VMpool10/VLT/VM  creation              Mon Aug  9  0:27 2021  -
VMpool10/VLT/VM  used                  192K                   -
VMpool10/VLT/VM  available             144G                   -
VMpool10/VLT/VM  referenced            192K                   -
VMpool10/VLT/VM  compressratio         1.00x                  -
VMpool10/VLT/VM  mounted               yes                    -
VMpool10/VLT/VM  quota                 none                   default
VMpool10/VLT/VM  reservation           none                   default
VMpool10/VLT/VM  recordsize            128K                   default
VMpool10/VLT/VM  mountpoint            /VMpool10/VLT/VM       default
VMpool10/VLT/VM  sharenfs              off                    default
VMpool10/VLT/VM  checksum              on                     default
VMpool10/VLT/VM  compression           lz4                    inherited from VMpool10
VMpool10/VLT/VM  atime                 off                    inherited from VMpool10
VMpool10/VLT/VM  devices               on                     default
VMpool10/VLT/VM  exec                  on                     default
VMpool10/VLT/VM  setuid                on                     default
VMpool10/VLT/VM  readonly              off                    default
VMpool10/VLT/VM  zoned                 off                    default
VMpool10/VLT/VM  snapdir               hidden                 default
VMpool10/VLT/VM  aclmode               discard                default
VMpool10/VLT/VM  aclinherit            restricted             default
VMpool10/VLT/VM  createtxg             41                     -
VMpool10/VLT/VM  canmount              on                     local
VMpool10/VLT/VM  xattr                 on                     default
VMpool10/VLT/VM  copies                1                      default
VMpool10/VLT/VM  version               5                      -
VMpool10/VLT/VM  utf8only              off                    -
VMpool10/VLT/VM  normalization         none                   -
VMpool10/VLT/VM  casesensitivity       sensitive              -
VMpool10/VLT/VM  vscan                 off                    default
VMpool10/VLT/VM  nbmand                off                    default
VMpool10/VLT/VM  sharesmb              off                    default
VMpool10/VLT/VM  refquota              none                   default
VMpool10/VLT/VM  refreservation        none                   default
VMpool10/VLT/VM  guid                  3294781934272898903    -
VMpool10/VLT/VM  primarycache          metadata               inherited from VMpool10
VMpool10/VLT/VM  secondarycache        all                    default
VMpool10/VLT/VM  usedbysnapshots       0B                     -
VMpool10/VLT/VM  usedbydataset         192K                   -
VMpool10/VLT/VM  usedbychildren        0B                     -
VMpool10/VLT/VM  usedbyrefreservation  0B                     -
VMpool10/VLT/VM  logbias               latency                default
VMpool10/VLT/VM  objsetid              773                    -
VMpool10/VLT/VM  dedup                 off                    default
VMpool10/VLT/VM  mlslabel              none                   default
VMpool10/VLT/VM  sync                  standard               inherited from VMpool10
VMpool10/VLT/VM  dnodesize             legacy                 default
VMpool10/VLT/VM  refcompressratio      1.00x                  -
VMpool10/VLT/VM  written               192K                   -
VMpool10/VLT/VM  logicalused           69K                    -
VMpool10/VLT/VM  logicalreferenced     69K                    -
VMpool10/VLT/VM  volmode               default                default
VMpool10/VLT/VM  filesystem_limit      none                   default
VMpool10/VLT/VM  snapshot_limit        none                   default
VMpool10/VLT/VM  filesystem_count      none                   default
VMpool10/VLT/VM  snapshot_count        none                   default
VMpool10/VLT/VM  snapdev               hidden                 default
VMpool10/VLT/VM  acltype               off                    default
VMpool10/VLT/VM  context               none                   default
VMpool10/VLT/VM  fscontext             none                   default
VMpool10/VLT/VM  defcontext            none                   default
VMpool10/VLT/VM  rootcontext           none                   default
VMpool10/VLT/VM  relatime              off                    default
VMpool10/VLT/VM  redundant_metadata    all                    default
VMpool10/VLT/VM  overlay               on                     default
VMpool10/VLT/VM  encryption            aes-256-gcm            -
VMpool10/VLT/VM  keylocation           none                   default
VMpool10/VLT/VM  keyformat             passphrase             -
VMpool10/VLT/VM  pbkdf2iters           350000                 -
VMpool10/VLT/VM  encryptionroot        VMpool10/VLT           -
VMpool10/VLT/VM  keystatus             available              -
VMpool10/VLT/VM  special_small_blocks  0                      default

Commands used to create the pool:

Code:

# create mirror of 2 disks (2x Intel S3710 200GB):
zpool create -f -o ashift=12 VMpool10 mirror /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV54630159200MGN /dev/disk/by-id/ata-INTEL_SSDSC2BA200G4_BTHV737402JJ200MGN
# set ZFS options
zpool set autoexpand=off VMpool10
zpool set feature@encryption=enabled VMpool10
zfs set compression=lz4 VMpool10
zfs set sync=standard VMpool10
zfs set canmount=off VMpool10
zfs set atime=off VMpool10
# set quota to 80% so pool can't get full/slow:
zfs set quota=144G VMpool10
# create encrypted dataset:
zfs create -o encryption=on -o keyformat=passphrase VMpool10/VLT
zfs set canmount=off VMpool10/VLT
# create dataset for encrypted VMs
zfs create -o encryption=on VMpool10/VLT/VM
zfs set canmount=on VMpool10/VLT/VM
# add dataset to PVE:
pvesm add zfspool VMpool10_VLT_VM -pool VMpool10/VLT/VM --content images,rootdir --sparse 1 --mountpoint /VMpool10/VLT/VM
# disable ARC caching:
zfs set primarycache=metadata VMpool10
# don't forget to set block size from 8K to 4K using WebUI

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 1: sync 4K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_1.sh
sync_rand_4K: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_4K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1804KiB/s][w=451 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=749: Mon Aug  9 00:51:43 2021
  write: IOPS=477, BW=1912KiB/s (1957kB/s)(1024MiB/548528msec); 0 zone resets
    clat (usec): min=964, max=53141, avg=2077.96, stdev=705.92
     lat (usec): min=965, max=53143, avg=2080.16, stdev=705.92
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1549], 10.00th=[ 1647], 20.00th=[ 1778],
     | 30.00th=[ 1860], 40.00th=[ 1942], 50.00th=[ 2008], 60.00th=[ 2073],
     | 70.00th=[ 2147], 80.00th=[ 2245], 90.00th=[ 2343], 95.00th=[ 2474],
     | 99.00th=[ 5145], 99.50th=[ 6259], 99.90th=[10028], 99.95th=[12387],
     | 99.99th=[20317]
   bw (  KiB/s): min=  928, max= 2744, per=100.00%, avg=1911.43, stdev=230.00, samples=1097
   iops        : min=  232, max=  686, avg=477.85, stdev=57.49, samples=1097
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=49.20%, 4=48.98%, 10=1.72%, 20=0.09%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.19%, sys=7.13%, ctx=554607, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1912KiB/s (1957kB/s), 1912KiB/s-1912KiB/s (1957kB/s-1957kB/s), io=1024MiB (1074MB), run=548528-548528msec

Disk stats (read/write):
  sda: ios=1/799575, merge=0/542776, ticks=1/481374, in_queue=547116, util=99.79%
0 min passed...Guest Writes/Reads (Changes): 5234 MiB / 18 MiB ( 5234 MiB / 18 MiB )
1 min passed...Guest Writes/Reads (Changes): 5234 MiB / 18 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 1 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5235 MiB / 18 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 1 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5236 MiB / 18 MiB ( 0 MiB / 0 MiB )
sync_rand_4K: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=7931KiB/s][r=1982 IOPS][eta 00m:00s]
sync_rand_4K: (groupid=0, jobs=1): err= 0: pid=960: Mon Aug  9 01:08:35 2021
  read: IOPS=2367, BW=9470KiB/s (9698kB/s)(1024MiB/110723msec)
    clat (usec): min=183, max=6965, avg=412.15, stdev=164.44
     lat (usec): min=185, max=6967, avg=414.09, stdev=164.51
    clat percentiles (usec):
     |  1.00th=[  262],  5.00th=[  281], 10.00th=[  297], 20.00th=[  318],
     | 30.00th=[  343], 40.00th=[  363], 50.00th=[  388], 60.00th=[  416],
     | 70.00th=[  445], 80.00th=[  486], 90.00th=[  545], 95.00th=[  578],
     | 99.00th=[  660], 99.50th=[ 1156], 99.90th=[ 2802], 99.95th=[ 3032],
     | 99.99th=[ 3326]
   bw (  KiB/s): min= 7160, max=12896, per=100.00%, avg=9471.01, stdev=1307.04, samples=221
   iops        : min= 1790, max= 3224, avg=2367.72, stdev=326.75, samples=221
  lat (usec)   : 250=0.28%, 500=82.74%, 750=16.23%, 1000=0.17%
  lat (msec)   : 2=0.28%, 4=0.29%, 10=0.01%
  cpu          : usr=3.88%, sys=11.22%, ctx=262545, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=9470KiB/s (9698kB/s), 9470KiB/s-9470KiB/s (9698kB/s-9698kB/s), io=1024MiB (1074MB), run=110723-110723msec

Disk stats (read/write):
  sda: ios=261962/26, merge=0/33, ticks=99090/20, in_queue=110660, util=99.95%
0 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 1 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 5237 MiB / 1042 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 5238 MiB / 1042 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 5239 MiB / 1042 MiB ( 1 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 5239 MiB / 1042 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 5239 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 5239 MiB / 1042 MiB ( 0 MiB / 0 MiB )
Script ran 2462 seconds or 41 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5120 MiB / 3552 MiB / 5145 MiB / 0 MiB / 16 MiB ( 5120 MiB / 3552 MiB / 5145 MiB / 0 MiB / 16 MiB )
2 min passed...10656 MiB / 11648 MiB / 10664 MiB / 0 MiB / 16 MiB ( 5536 MiB / 8096 MiB / 5519 MiB / 0 MiB / 0 MiB )
3 min passed...16224 MiB / 17568 MiB / 16250 MiB / 0 MiB / 16 MiB ( 5568 MiB / 5920 MiB / 5586 MiB / 0 MiB / 0 MiB )
4 min passed...21856 MiB / 23776 MiB / 21891 MiB / 0 MiB / 16 MiB ( 5632 MiB / 6208 MiB / 5641 MiB / 0 MiB / 0 MiB )
5 min passed...27584 MiB / 29824 MiB / 27605 MiB / 0 MiB / 16 MiB ( 5728 MiB / 6048 MiB / 5714 MiB / 0 MiB / 0 MiB )
6 min passed...33504 MiB / 36096 MiB / 33524 MiB / 0 MiB / 16 MiB ( 5920 MiB / 6272 MiB / 5919 MiB / 0 MiB / 0 MiB )
7 min passed...39552 MiB / 42496 MiB / 39563 MiB / 0 MiB / 16 MiB ( 6048 MiB / 6400 MiB / 6039 MiB / 0 MiB / 0 MiB )
8 min passed...45344 MiB / 49312 MiB / 45355 MiB / 0 MiB / 16 MiB ( 5792 MiB / 6816 MiB / 5792 MiB / 0 MiB / 0 MiB )
9 min passed...51360 MiB / 55680 MiB / 51379 MiB / 0 MiB / 16 MiB ( 6016 MiB / 6368 MiB / 6024 MiB / 0 MiB / 0 MiB )
10 min passed...52320 MiB / 56704 MiB / 52329 MiB / 0 MiB / 18 MiB ( 960 MiB / 1024 MiB / 950 MiB / 0 MiB / 2 MiB )
11 min passed...52320 MiB / 56704 MiB / 52339 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
12 min passed...52352 MiB / 56768 MiB / 52349 MiB / 0 MiB / 18 MiB ( 32 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...52352 MiB / 56832 MiB / 52360 MiB / 0 MiB / 18 MiB ( 0 MiB / 64 MiB / 11 MiB / 0 MiB / 0 MiB )
14 min passed...52352 MiB / 57024 MiB / 52372 MiB / 0 MiB / 18 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
15 min passed...52384 MiB / 57184 MiB / 52382 MiB / 0 MiB / 18 MiB ( 32 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
16 min passed...52384 MiB / 57184 MiB / 52391 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
17 min passed...52384 MiB / 57344 MiB / 52401 MiB / 0 MiB / 18 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...52384 MiB / 57472 MiB / 52409 MiB / 0 MiB / 18 MiB ( 0 MiB / 128 MiB / 8 MiB / 0 MiB / 0 MiB )
19 min passed...52416 MiB / 57504 MiB / 52420 MiB / 0 MiB / 18 MiB ( 32 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
20 min passed...52416 MiB / 57792 MiB / 52429 MiB / 0 MiB / 18 MiB ( 0 MiB / 288 MiB / 9 MiB / 0 MiB / 0 MiB )
21 min passed...52416 MiB / 57856 MiB / 52438 MiB / 0 MiB / 18 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
22 min passed...52448 MiB / 58144 MiB / 52449 MiB / 0 MiB / 18 MiB ( 32 MiB / 288 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...52448 MiB / 58208 MiB / 52459 MiB / 0 MiB / 18 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...52448 MiB / 58208 MiB / 52471 MiB / 0 MiB / 18 MiB ( 0 MiB / 0 MiB / 12 MiB / 0 MiB / 0 MiB )
25 min passed...52480 MiB / 58304 MiB / 52480 MiB / 512 MiB / 521 MiB ( 32 MiB / 96 MiB / 9 MiB / 512 MiB / 503 MiB )
26 min passed...52480 MiB / 58816 MiB / 52490 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 512 MiB / 10 MiB / 512 MiB / 521 MiB )
27 min passed...52480 MiB / 58880 MiB / 52500 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...52512 MiB / 58880 MiB / 52510 MiB / 1024 MiB / 1042 MiB ( 32 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
29 min passed...52512 MiB / 58880 MiB / 52521 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...52512 MiB / 58912 MiB / 52531 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
31 min passed...52544 MiB / 59136 MiB / 52542 MiB / 1024 MiB / 1042 MiB ( 32 MiB / 224 MiB / 11 MiB / 0 MiB / 0 MiB )
32 min passed...52544 MiB / 59328 MiB / 52554 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
33 min passed...52544 MiB / 59360 MiB / 52563 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 32 MiB / 9 MiB / 0 MiB / 0 MiB )
34 min passed...52576 MiB / 59392 MiB / 52575 MiB / 1024 MiB / 1042 MiB ( 32 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
35 min passed...52576 MiB / 59552 MiB / 52585 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 160 MiB / 10 MiB / 0 MiB / 0 MiB )
36 min passed...52576 MiB / 59680 MiB / 52595 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 128 MiB / 10 MiB / 0 MiB / 0 MiB )
37 min passed...52576 MiB / 59872 MiB / 52603 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 192 MiB / 8 MiB / 0 MiB / 0 MiB )
38 min passed...52608 MiB / 60032 MiB / 52611 MiB / 1024 MiB / 1042 MiB ( 32 MiB / 160 MiB / 8 MiB / 0 MiB / 0 MiB )
39 min passed...52608 MiB / 60096 MiB / 52620 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
40 min passed...52608 MiB / 60288 MiB / 52631 MiB / 1024 MiB / 1042 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
41 min passed...52704 MiB / 60384 MiB / 52703 MiB / 1024 MiB / 1042 MiB ( 96 MiB / 96 MiB / 72 MiB / 0 MiB / 0 MiB )

Conclusion:
Total write amplification is nearly as terrible as the 8 drives striped mirror. But its the best performance so far for sync 4K writes and nearly 50% faster than the 5 disk raidz1 or 8 disk striped mirror.
But can someone explain why the write amplification is still that bad? Its now fio writiong 4K blocks, to a 4K ext4 that is on a 4K virtio virtual disk, that is on a 4K zvol, that is on a 4K pool (ashift 12) pool that writes to a mirror of 4K LBA SSDs. So why is there still a factor 57 write amplification?
ext4 is journaling so I would expect a factor 2 amplification there. Next should be the ZIL where I also expect a factor 2 amplification. And then there is a factor 2 amplification because everything is written twice because of the mirror. So that would be a write amplification of 8 (2 * 2 * 2).

But how it gets from factor 8 to 57?

Fio was writing 1024 MiB. The guest wrote 5236 MiB to the virtual disks (5,11 amplification), the host wrote 52448 MiB (10,02 amplification) and the SSD wrote 58208 MiB (1,11 amplification) to the NAND. So the SSDs are clearly not the problem but the high amplification inside the guest and on the host.

Dunuin · Aug 9, 2021

Round 2 - Run 3 - Test 2: sync 16K random read/write

Guest:

Code:

root@DebianTest2:~# bash /root/benchmark_c_2.sh
sync_rand_16K: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
sync_rand_16K: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=10.1MiB/s][w=643 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=697: Mon Aug  9 01:35:12 2021
  write: IOPS=459, BW=7358KiB/s (7535kB/s)(1024MiB/142499msec); 0 zone resets
    clat (usec): min=985, max=41066, avg=2154.58, stdev=713.69
     lat (usec): min=986, max=41068, avg=2156.77, stdev=713.67
    clat percentiles (usec):
     |  1.00th=[ 1418],  5.00th=[ 1614], 10.00th=[ 1729], 20.00th=[ 1860],
     | 30.00th=[ 1942], 40.00th=[ 2024], 50.00th=[ 2089], 60.00th=[ 2147],
     | 70.00th=[ 2245], 80.00th=[ 2311], 90.00th=[ 2442], 95.00th=[ 2540],
     | 99.00th=[ 5407], 99.50th=[ 6390], 99.90th=[10028], 99.95th=[11731],
     | 99.99th=[22414]
   bw (  KiB/s): min= 3776, max=10304, per=99.86%, avg=7347.90, stdev=956.74, samples=284
   iops        : min=  236, max=  644, avg=459.24, stdev=59.79, samples=284
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=37.24%, 4=60.76%, 10=1.90%, 20=0.09%, 50=0.01%
  cpu          : usr=1.43%, sys=6.94%, ctx=134772, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=7358KiB/s (7535kB/s), 7358KiB/s-7358KiB/s (7535kB/s-7535kB/s), io=1024MiB (1074MB), run=142499-142499msec

Disk stats (read/write):
  sda: ios=1/196846, merge=0/132275, ticks=0/124187, in_queue=142284, util=99.90%
0 min passed...Guest Writes/Reads (Changes): 2055 MiB / 19 MiB ( 2055 MiB / 19 MiB )
1 min passed...Guest Writes/Reads (Changes): 2055 MiB / 19 MiB ( 0 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2055 MiB / 19 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 1 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2056 MiB / 19 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 1 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2057 MiB / 19 MiB ( 0 MiB / 0 MiB )
sync_rand_16K: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=33.8MiB/s][r=2163 IOPS][eta 00m:00s]
sync_rand_16K: (groupid=0, jobs=1): err= 0: pid=903: Mon Aug  9 01:50:43 2021
  read: IOPS=2104, BW=32.9MiB/s (34.5MB/s)(1024MiB/31134msec)
    clat (usec): min=319, max=11397, avg=464.96, stdev=74.80
     lat (usec): min=320, max=11399, avg=466.83, stdev=74.88
    clat percentiles (usec):
     |  1.00th=[  367],  5.00th=[  388], 10.00th=[  400], 20.00th=[  416],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  469],
     | 70.00th=[  486], 80.00th=[  506], 90.00th=[  537], 95.00th=[  570],
     | 99.00th=[  635], 99.50th=[  660], 99.90th=[  693], 99.95th=[  734],
     | 99.99th=[ 2245]
   bw (  KiB/s): min=29216, max=36896, per=100.00%, avg=33687.21, stdev=1367.48, samples=62
   iops        : min= 1826, max= 2306, avg=2105.44, stdev=85.48, samples=62
  lat (usec)   : 500=77.20%, 750=22.76%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.01%, 20=0.01%
  cpu          : usr=3.60%, sys=9.85%, ctx=65614, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=1024MiB (1074MB), run=31134-31134msec

Disk stats (read/write):
  sda: ios=65053/8, merge=0/6, ticks=28020/5, in_queue=30972, util=99.95%
0 min passed...Guest Writes/Reads (Changes): 2057 MiB / 1043 MiB ( 0 MiB / 1024 MiB )
1 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 1 MiB / 0 MiB )
2 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
3 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
4 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
5 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
6 min passed...Guest Writes/Reads (Changes): 2058 MiB / 1043 MiB ( 0 MiB / 0 MiB )
7 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 1 MiB / 0 MiB )
8 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
9 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
10 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
11 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
12 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
13 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
14 min passed...Guest Writes/Reads (Changes): 2059 MiB / 1043 MiB ( 0 MiB / 0 MiB )
15 min passed...Guest Writes/Reads (Changes): 2060 MiB / 1043 MiB ( 1 MiB / 0 MiB )
Final...Guest Writes/Reads (Changes): 2060 MiB / 1043 MiB ( 0 MiB / 0 MiB )
Script ran 1976 seconds or 32 minutes.

Host:

Code:

root@Hypervisor:~# bash /root/scripts/smartctl_7.sh
1st Nr: Host Writes (how much data the SSDs recieved from host since start of the script)
2nd Nr: NAND Writes (how much data was written to the SSDs NAND chips since start of the script)
3rd Nr: Pool Writes (how much data the host has written to the SSDs since start of the script)
4rd Nr: Host Reads (how much data the SSD has sent to the host since start of the script)
5th Nr: Pool Reads (how much data the host has read from the SSDs since start of the script)
Number between brackets are changes since last minute. Same oder as above.
0 min passed...0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB ( 0 MiB / 0 MiB / 0 MiB / 0 MiB / 0 MiB )
1 min passed...5728 MiB / 6048 MiB / 5715 MiB / 32 MiB / 18 MiB ( 5728 MiB / 6048 MiB / 5715 MiB / 32 MiB / 18 MiB )
2 min passed...11936 MiB / 12640 MiB / 11931 MiB / 32 MiB / 18 MiB ( 6208 MiB / 6592 MiB / 6216 MiB / 0 MiB / 0 MiB )
3 min passed...14592 MiB / 15648 MiB / 14568 MiB / 32 MiB / 19 MiB ( 2656 MiB / 3008 MiB / 2637 MiB / 0 MiB / 1 MiB )
4 min passed...14592 MiB / 15840 MiB / 14580 MiB / 32 MiB / 19 MiB ( 0 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
5 min passed...14592 MiB / 16096 MiB / 14592 MiB / 32 MiB / 19 MiB ( 0 MiB / 256 MiB / 12 MiB / 0 MiB / 0 MiB )
6 min passed...14624 MiB / 16128 MiB / 14603 MiB / 32 MiB / 19 MiB ( 32 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
7 min passed...14624 MiB / 16192 MiB / 14613 MiB / 32 MiB / 19 MiB ( 0 MiB / 64 MiB / 10 MiB / 0 MiB / 0 MiB )
8 min passed...14656 MiB / 16384 MiB / 14625 MiB / 32 MiB / 19 MiB ( 32 MiB / 192 MiB / 12 MiB / 0 MiB / 0 MiB )
9 min passed...14656 MiB / 16416 MiB / 14636 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
10 min passed...14656 MiB / 16672 MiB / 14647 MiB / 32 MiB / 19 MiB ( 0 MiB / 256 MiB / 11 MiB / 0 MiB / 0 MiB )
11 min passed...14656 MiB / 16768 MiB / 14658 MiB / 32 MiB / 19 MiB ( 0 MiB / 96 MiB / 11 MiB / 0 MiB / 0 MiB )
12 min passed...14688 MiB / 16864 MiB / 14668 MiB / 32 MiB / 19 MiB ( 32 MiB / 96 MiB / 10 MiB / 0 MiB / 0 MiB )
13 min passed...14688 MiB / 16896 MiB / 14680 MiB / 32 MiB / 19 MiB ( 0 MiB / 32 MiB / 12 MiB / 0 MiB / 0 MiB )
14 min passed...14720 MiB / 16896 MiB / 14692 MiB / 32 MiB / 20 MiB ( 32 MiB / 0 MiB / 12 MiB / 0 MiB / 1 MiB )
15 min passed...14720 MiB / 16928 MiB / 14703 MiB / 32 MiB / 20 MiB ( 0 MiB / 32 MiB / 11 MiB / 0 MiB / 0 MiB )
16 min passed...14720 MiB / 17280 MiB / 14713 MiB / 32 MiB / 20 MiB ( 0 MiB / 352 MiB / 10 MiB / 0 MiB / 0 MiB )
17 min passed...14752 MiB / 17504 MiB / 14723 MiB / 32 MiB / 20 MiB ( 32 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
18 min passed...14752 MiB / 17504 MiB / 14734 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 11 MiB / 1024 MiB / 1024 MiB )
19 min passed...14752 MiB / 17536 MiB / 14744 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 32 MiB / 10 MiB / 0 MiB / 0 MiB )
20 min passed...14784 MiB / 17536 MiB / 14755 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
21 min passed...14784 MiB / 17856 MiB / 14763 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 320 MiB / 8 MiB / 0 MiB / 0 MiB )
22 min passed...14784 MiB / 18048 MiB / 14774 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 192 MiB / 11 MiB / 0 MiB / 0 MiB )
23 min passed...14784 MiB / 18048 MiB / 14784 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 10 MiB / 0 MiB / 0 MiB )
24 min passed...14816 MiB / 18112 MiB / 14797 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 64 MiB / 13 MiB / 0 MiB / 0 MiB )
25 min passed...14816 MiB / 18176 MiB / 14806 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )
26 min passed...14848 MiB / 18304 MiB / 14818 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 128 MiB / 12 MiB / 0 MiB / 0 MiB )
27 min passed...14848 MiB / 18528 MiB / 14828 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 224 MiB / 10 MiB / 0 MiB / 0 MiB )
28 min passed...14848 MiB / 18656 MiB / 14837 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 128 MiB / 9 MiB / 0 MiB / 0 MiB )
29 min passed...14848 MiB / 18656 MiB / 14848 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 11 MiB / 0 MiB / 0 MiB )
30 min passed...14880 MiB / 18784 MiB / 14859 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 128 MiB / 11 MiB / 0 MiB / 0 MiB )
31 min passed...14880 MiB / 18784 MiB / 14868 MiB / 1056 MiB / 1044 MiB ( 0 MiB / 0 MiB / 9 MiB / 0 MiB / 0 MiB )
32 min passed...14912 MiB / 18848 MiB / 14877 MiB / 1056 MiB / 1044 MiB ( 32 MiB / 64 MiB / 9 MiB / 0 MiB / 0 MiB )

how to best benchmark SSDs?

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member