root@prx1:~# fio /root/fio.conf
test: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
...
fio-3.33
Starting 4 processes
Jobs: 4 (f=4): [m(4)][100.0%][r=2846MiB/s,w=934MiB/s][r=711,w=233 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2758559: Fri Sep 8 15:42:45 2023
read: IOPS=167, BW=670MiB/s (702MB/s)(39.2GiB/60004msec)
slat (usec): min=1320, max=82078, avg=4052.70, stdev=1472.26
clat (usec): min=23, max=548390, avg=280998.55, stdev=43922.49
lat (msec): min=3, max=553, avg=285.05, stdev=44.41
clat percentiles (msec):
| 1.00th=[ 226], 5.00th=[ 243], 10.00th=[ 249], 20.00th=[ 253],
| 30.00th=[ 257], 40.00th=[ 266], 50.00th=[ 271], 60.00th=[ 275],
| 70.00th=[ 288], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 363],
| 99.00th=[ 472], 99.50th=[ 506], 99.90th=[ 531], 99.95th=[ 542],
| 99.99th=[ 550]
bw ( KiB/s): min=122880, max=876544, per=24.59%, avg=683211.63, stdev=112333.37, samples=119
iops : min= 30, max= 214, avg=166.54, stdev=27.40, samples=119
write: IOPS=56, BW=224MiB/s (235MB/s)(13.1GiB/60004msec); 0 zone resets
slat (usec): min=3068, max=32822, avg=5592.28, stdev=1792.55
clat (msec): min=3, max=532, avg=280.97, stdev=42.37
lat (msec): min=7, max=538, avg=286.57, stdev=43.00
clat percentiles (msec):
| 1.00th=[ 226], 5.00th=[ 245], 10.00th=[ 249], 20.00th=[ 253],
| 30.00th=[ 257], 40.00th=[ 266], 50.00th=[ 271], 60.00th=[ 279],
| 70.00th=[ 288], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 363],
| 99.00th=[ 451], 99.50th=[ 481], 99.90th=[ 527], 99.95th=[ 531],
| 99.99th=[ 535]
bw ( KiB/s): min=32768, max=351552, per=24.16%, avg=228060.19, stdev=48081.96, samples=119
iops : min= 8, max= 85, avg=55.34, stdev=11.66, samples=119
lat (usec) : 50=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.06%, 100=0.10%
lat (msec) : 250=15.09%, 500=84.25%, 750=0.47%
cpu : usr=4.13%, sys=91.67%, ctx=11775, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=10047,3361,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
test: (groupid=0, jobs=1): err= 0: pid=2758560: Fri Sep 8 15:42:45 2023
read: IOPS=170, BW=683MiB/s (717MB/s)(40.0GiB/60002msec)
slat (usec): min=1393, max=41838, avg=3918.34, stdev=1315.29
clat (usec): min=23, max=604238, avg=273349.08, stdev=49207.07
lat (msec): min=2, max=610, avg=277.27, stdev=49.73
clat percentiles (msec):
| 1.00th=[ 203], 5.00th=[ 226], 10.00th=[ 241], 20.00th=[ 249],
| 30.00th=[ 253], 40.00th=[ 257], 50.00th=[ 264], 60.00th=[ 271],
| 70.00th=[ 275], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 363],
| 99.00th=[ 485], 99.50th=[ 535], 99.90th=[ 600], 99.95th=[ 600],
| 99.99th=[ 600]
bw ( KiB/s): min=81920, max=933888, per=25.06%, avg=696195.03, stdev=119067.95, samples=119
iops : min= 20, max= 228, avg=169.73, stdev=29.04, samples=119
write: IOPS=59, BW=236MiB/s (248MB/s)(13.8GiB/60002msec); 0 zone resets
slat (usec): min=2882, max=49333, avg=5459.48, stdev=2074.90
clat (msec): min=7, max=596, avg=272.88, stdev=47.85
lat (msec): min=10, max=606, avg=278.34, stdev=48.74
clat percentiles (msec):
| 1.00th=[ 205], 5.00th=[ 226], 10.00th=[ 241], 20.00th=[ 249],
| 30.00th=[ 253], 40.00th=[ 257], 50.00th=[ 262], 60.00th=[ 271],
| 70.00th=[ 275], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 359],
| 99.00th=[ 485], 99.50th=[ 523], 99.90th=[ 592], 99.95th=[ 600],
| 99.99th=[ 600]
bw ( KiB/s): min=40960, max=409600, per=25.59%, avg=241599.24, stdev=51924.35, samples=119
iops : min= 10, max= 100, avg=58.68, stdev=12.66, samples=119
lat (usec) : 50=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.03%, 50=0.05%, 100=0.09%
lat (msec) : 250=21.73%, 500=77.26%, 750=0.81%
cpu : usr=4.62%, sys=90.89%, ctx=11628, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=10250,3542,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
test: (groupid=0, jobs=1): err= 0: pid=2758561: Fri Sep 8 15:42:45 2023
read: IOPS=169, BW=679MiB/s (712MB/s)(39.8GiB/60002msec)
slat (usec): min=1351, max=71337, avg=3965.19, stdev=1649.73
clat (usec): min=25, max=610934, avg=275656.11, stdev=50919.96
lat (msec): min=3, max=615, avg=279.62, stdev=51.44
clat percentiles (msec):
| 1.00th=[ 209], 5.00th=[ 232], 10.00th=[ 245], 20.00th=[ 251],
| 30.00th=[ 253], 40.00th=[ 257], 50.00th=[ 262], 60.00th=[ 268],
| 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 326], 95.00th=[ 368],
| 99.00th=[ 531], 99.50th=[ 567], 99.90th=[ 600], 99.95th=[ 609],
| 99.99th=[ 609]
bw ( KiB/s): min=122880, max=891145, per=24.91%, avg=692204.44, stdev=121792.39, samples=119
iops : min= 30, max= 217, avg=168.72, stdev=29.71, samples=119
write: IOPS=58, BW=232MiB/s (244MB/s)(13.6GiB/60002msec); 0 zone resets
slat (usec): min=2583, max=41498, avg=5473.55, stdev=1993.95
clat (msec): min=3, max=608, avg=275.96, stdev=51.83
lat (msec): min=8, max=615, avg=281.43, stdev=52.58
clat percentiles (msec):
| 1.00th=[ 209], 5.00th=[ 232], 10.00th=[ 245], 20.00th=[ 251],
| 30.00th=[ 253], 40.00th=[ 257], 50.00th=[ 262], 60.00th=[ 271],
| 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 330], 95.00th=[ 368],
| 99.00th=[ 531], 99.50th=[ 558], 99.90th=[ 592], 99.95th=[ 600],
| 99.99th=[ 609]
bw ( KiB/s): min=49152, max=352256, per=25.19%, avg=237781.35, stdev=53436.02, samples=119
iops : min= 12, max= 86, avg=57.73, stdev=13.07, samples=119
lat (usec) : 50=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%, 100=0.10%
lat (msec) : 250=20.91%, 500=77.59%, 750=1.30%
cpu : usr=4.33%, sys=91.03%, ctx=11661, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=10186,3485,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
test: (groupid=0, jobs=1): err= 0: pid=2758562: Fri Sep 8 15:42:45 2023
read: IOPS=170, BW=681MiB/s (714MB/s)(39.9GiB/60001msec)
slat (usec): min=1151, max=94065, avg=3970.43, stdev=1644.05
clat (usec): min=13, max=585914, avg=275584.93, stdev=51115.53
lat (usec): min=1164, max=591057, avg=279555.37, stdev=51735.80
clat percentiles (msec):
| 1.00th=[ 207], 5.00th=[ 232], 10.00th=[ 245], 20.00th=[ 249],
| 30.00th=[ 253], 40.00th=[ 255], 50.00th=[ 257], 60.00th=[ 264],
| 70.00th=[ 275], 80.00th=[ 296], 90.00th=[ 351], 95.00th=[ 376],
| 99.00th=[ 481], 99.50th=[ 527], 99.90th=[ 575], 99.95th=[ 584],
| 99.99th=[ 584]
bw ( KiB/s): min=130810, max=899321, per=24.99%, avg=694293.22, stdev=123196.27, samples=119
iops : min= 31, max= 219, avg=169.26, stdev=30.07, samples=119
write: IOPS=57, BW=230MiB/s (241MB/s)(13.4GiB/60001msec); 0 zone resets
slat (usec): min=1606, max=53706, avg=5496.92, stdev=1998.93
clat (usec): min=1179, max=582594, avg=276297.84, stdev=53049.74
lat (msec): min=2, max=593, avg=281.79, stdev=53.80
clat percentiles (msec):
| 1.00th=[ 207], 5.00th=[ 232], 10.00th=[ 245], 20.00th=[ 249],
| 30.00th=[ 251], 40.00th=[ 255], 50.00th=[ 257], 60.00th=[ 264],
| 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 355], 95.00th=[ 380],
| 99.00th=[ 498], 99.50th=[ 527], 99.90th=[ 575], 99.95th=[ 584],
| 99.99th=[ 584]
bw ( KiB/s): min=57229, max=359728, per=24.79%, avg=234008.38, stdev=50462.01, samples=119
iops : min= 13, max= 87, avg=56.80, stdev=12.29, samples=119
lat (usec) : 20=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.04%, 50=0.07%
lat (msec) : 100=0.09%, 250=23.53%, 500=75.31%, 750=0.91%
cpu : usr=4.39%, sys=91.34%, ctx=11020, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=10221,3443,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=2713MiB/s (2845MB/s), 670MiB/s-683MiB/s (702MB/s-717MB/s), io=159GiB (171GB), run=60001-60004msec
WRITE: bw=922MiB/s (967MB/s), 224MiB/s-236MiB/s (235MB/s-248MB/s), io=54.0GiB (58.0GB), run=60001-60004msec