NVMe SSD. Slow disk speed inside VM

[FessAectan]

Hi there!

I faced a problem with slow disk speed performance inside a VM (when a VM has a disk on host NVMe SSD).
For instance, when I copy 10GB file from one pve to another pve (LAN 10Gbit/s):

rsync -P -avz file10G 10.1.123.1:
sending incremental file list
file10G
 10,737,418,240 100%    2.84GB/s    0:00:03 (xfr#1, to-chk=0/1)

pve (host) to VM

rsync -P -avz file10G 10.1.123.206:
sending incremental file list
file10G
    617,086,976   5%  196.29MB/s    0:00:50

Test with fio. Host

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.25
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=445MiB/s][w=114k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=4129535: Wed Sep 29 14:37:52 2021
  write: IOPS=97.4k, BW=380MiB/s (399MB/s)(22.4GiB/60414msec); 0 zone resets
    slat (nsec): min=533, max=1353.4k, avg=1474.47, stdev=821.76
    clat (nsec): min=117, max=4367.8k, avg=6619.21, stdev=9392.24
     lat (usec): min=4, max=4369, avg= 8.09, stdev= 9.46
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   22], 99.90th=[   53], 99.95th=[   75],
     | 99.99th=[  221]
   bw (  KiB/s): min=20456, max=506048, per=100.00%, avg=452587.73, stdev=74042.85, samples=103
   iops        : min= 5114, max=126512, avg=113146.95, stdev=18510.62, samples=103
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.51%, 20=1.95%, 50=0.42%
  lat (usec)   : 100=0.08%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=18.13%, sys=34.95%, ctx=5888188, majf=0, minf=48
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5883316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=22.4GiB (24.1GB), run=60414-60414msec

Disk stats (read/write):
    dm-6: ios=0/818584, merge=0/0, ticks=0/574964, in_queue=574964, util=14.24%, aggrios=133/944809, aggrmerge=5/2211, aggrticks=29/1313275, aggrin_queue=1347064, aggrutil=70.51%
  nvme0n1: ios=133/944809, merge=5/2211, ticks=29/1313275, in_queue=1347064, util=70.51%

VM

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.16
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=182MiB/s][w=46.7k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2010: Wed Sep 29 12:36:21 2021
  write: IOPS=36.9k, BW=144MiB/s (151MB/s)(8775MiB/60803msec); 0 zone resets
    slat (nsec): min=506, max=1004.6k, avg=4101.30, stdev=2293.91
    clat (nsec): min=166, max=207411k, avg=19751.96, stdev=285681.11
     lat (usec): min=6, max=207414, avg=23.85, stdev=285.70
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   14], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   16], 50.00th=[   17], 60.00th=[   18],
     | 70.00th=[   19], 80.00th=[   21], 90.00th=[   23], 95.00th=[   27],
     | 99.00th=[   51], 99.50th=[   65], 99.90th=[  176], 99.95th=[  506],
     | 99.99th=[ 1319]
   bw (  KiB/s): min=17576, max=237512, per=100.00%, avg=158797.04, stdev=32794.76, samples=112
   iops        : min= 4394, max=59378, avg=39699.25, stdev=8198.70, samples=112
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.99%, 20=78.22%, 50=19.75%
  lat (usec)   : 100=0.83%, 250=0.14%, 500=0.02%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=14.97%, sys=36.10%, ctx=2282024, majf=0, minf=58
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2246364,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8775MiB (9201MB), run=60803-60803msec

Disk stats (read/write):
  sda: ios=170/415892, merge=32/1899, ticks=32/154140, in_queue=17188, util=9.19%

The VM config

agent: 1
bios: seabios
boot: c
bootdisk: scsi0
ciuser: ololosha
cores: 2
cpu: host
description: ClickHouse server ch-01
hotplug: network,disk,usb
ide2: NVMe2:vm-115-cloudinit,media=cdrom,size=4M
ipconfig0: ip=10.1.123.206/24,gw=10.1.123.2
kvm: 1
memory: 65536
name: ch-01
net0: virtio=DA:8B:B7:52:AA:A3,bridge=vmbr1
numa: 0
onboot: 1
scsi0: NVMe2:vm-115-disk-0,backup=0,discard=on,iothread=1,size=30G,ssd=1
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=c9567916-c96f-47d8-bd77-ae448bb739ec
sockets: 2

VM OS details

uname -a
Linux ch-01 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID:   Ubuntu
Description:   Ubuntu 20.04.3 LTS
Release:   20.04
Codename:   focal

dpkg -l | grep qemu
ii  qemu-guest-agent               1:4.2-3ubuntu6.17                     amd64        Guest-side qemu-system agent

I have "Proxmox Virtual Environment 7.0-11" on the host.
Disks for VMs provided as LVM.
A part of my /etc/pve/storage.cfg

lvm: NVMe2
    vgname VM_disks
    content images,rootdir
    nodes pve02
    shared 0

What can I do to achieve disk speed in a VM like speed on a host?

 

[FessAectan]

Linear reading turned out to be quite sane

fio --filename=/dev/sda --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
iops-test-job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=2025MiB/s][r=518k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=2042MiB/s][r=523k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=2212MiB/s][r=566k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=3060MiB/s][r=783k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=3055MiB/s][r=782k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=3024MiB/s][r=774k IOPS][eta 01m:46s]
Jobs: 4 (f=4): [R(4)][12.5%][r=2966MiB/s][r=759k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [R(4)][14.2%][r=3082MiB/s][r=789k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [R(4)][15.8%][r=2977MiB/s][r=762k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [R(4)][17.5%][r=3094MiB/s][r=792k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [R(4)][19.2%][r=3001MiB/s][r=768k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [R(4)][20.8%][r=3023MiB/s][r=774k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [R(4)][22.5%][r=3053MiB/s][r=782k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [R(4)][24.2%][r=2979MiB/s][r=763k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [R(4)][26.1%][r=3076MiB/s][r=788k IOPS][eta 01m:28s]
Jobs: 4 (f=4): [R(4)][27.5%][r=2896MiB/s][r=741k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [R(4)][29.2%][r=2922MiB/s][r=748k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [R(4)][31.1%][r=2985MiB/s][r=764k IOPS][eta 01m:22s]
Jobs: 4 (f=4): [R(4)][32.8%][r=2997MiB/s][r=767k IOPS][eta 01m:20s]
Jobs: 4 (f=4): [R(4)][34.2%][r=3038MiB/s][r=778k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [R(4)][35.8%][r=3053MiB/s][r=782k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [R(4)][37.5%][r=3033MiB/s][r=776k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [R(4)][39.2%][r=2542MiB/s][r=651k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][40.8%][r=2004MiB/s][r=513k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [R(4)][42.5%][r=2050MiB/s][r=525k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [R(4)][44.2%][r=2410MiB/s][r=617k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [R(4)][45.8%][r=2950MiB/s][r=755k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [R(4)][47.5%][r=3072MiB/s][r=786k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [R(4)][49.2%][r=3016MiB/s][r=772k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [R(4)][50.8%][r=3039MiB/s][r=778k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [R(4)][52.5%][r=3154MiB/s][r=808k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [R(4)][54.2%][r=3106MiB/s][r=795k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [R(4)][55.8%][r=3224MiB/s][r=825k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [R(4)][57.5%][r=3007MiB/s][r=770k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [R(4)][59.2%][r=2985MiB/s][r=764k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [R(4)][60.8%][r=3000MiB/s][r=768k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [R(4)][62.5%][r=3036MiB/s][r=777k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [R(4)][64.7%][r=3137MiB/s][r=803k IOPS][eta 00m:42s]
Jobs: 4 (f=4): [R(4)][65.8%][r=3131MiB/s][r=802k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [R(4)][67.5%][r=3037MiB/s][r=778k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [R(4)][69.2%][r=2978MiB/s][r=762k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [R(4)][71.4%][r=3170MiB/s][r=811k IOPS][eta 00m:34s]
Jobs: 4 (f=4): [R(4)][72.5%][r=3036MiB/s][r=777k IOPS][eta 00m:33s]
Jobs: 4 (f=4): [R(4)][74.2%][r=3061MiB/s][r=784k IOPS][eta 00m:31s]
Jobs: 4 (f=4): [R(4)][75.8%][r=2719MiB/s][r=696k IOPS][eta 00m:29s]
Jobs: 4 (f=4): [R(4)][77.5%][r=2467MiB/s][r=631k IOPS][eta 00m:27s]
Jobs: 4 (f=4): [R(4)][79.2%][r=2141MiB/s][r=548k IOPS][eta 00m:25s]
Jobs: 4 (f=4): [R(4)][80.8%][r=2658MiB/s][r=680k IOPS][eta 00m:23s]
Jobs: 4 (f=4): [R(4)][82.5%][r=2620MiB/s][r=671k IOPS][eta 00m:21s]
Jobs: 4 (f=4): [R(4)][84.2%][r=2938MiB/s][r=752k IOPS][eta 00m:19s]
Jobs: 4 (f=4): [R(4)][85.8%][r=3157MiB/s][r=808k IOPS][eta 00m:17s]
Jobs: 4 (f=4): [R(4)][87.5%][r=3081MiB/s][r=789k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [R(4)][89.9%][r=3072MiB/s][r=786k IOPS][eta 00m:12s]
Jobs: 4 (f=4): [R(4)][90.8%][r=3120MiB/s][r=799k IOPS][eta 00m:11s]
Jobs: 4 (f=4): [R(4)][93.3%][r=3119MiB/s][r=799k IOPS][eta 00m:08s]
Jobs: 4 (f=4): [R(4)][94.2%][r=3112MiB/s][r=797k IOPS][eta 00m:07s]
Jobs: 4 (f=4): [R(4)][95.8%][r=3178MiB/s][r=814k IOPS][eta 00m:05s]
Jobs: 4 (f=4): [R(4)][97.5%][r=3171MiB/s][r=812k IOPS][eta 00m:03s]
Jobs: 4 (f=4): [R(4)][99.2%][r=3013MiB/s][r=771k IOPS][eta 00m:01s]
Jobs: 4 (f=4): [R(4)][100.0%][r=3072MiB/s][r=786k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4413: Thu Sep 30 13:29:27 2021
  read: IOPS=738k, BW=2882MiB/s (3022MB/s)(338GiB/120002msec)
    slat (nsec): min=1355, max=7830.0k, avg=2674.53, stdev=4210.37
    clat (usec): min=29, max=12364, avg=1384.28, stdev=405.35
     lat (usec): min=33, max=12366, avg=1387.17, stdev=405.70
    clat percentiles (usec):
     |  1.00th=[  611],  5.00th=[  906], 10.00th=[  963], 20.00th=[ 1045],
     | 30.00th=[ 1123], 40.00th=[ 1221], 50.00th=[ 1303], 60.00th=[ 1418],
     | 70.00th=[ 1565], 80.00th=[ 1729], 90.00th=[ 1893], 95.00th=[ 2089],
     | 99.00th=[ 2606], 99.50th=[ 2769], 99.90th=[ 3228], 99.95th=[ 3490],
     | 99.99th=[ 5014]
   bw (  MiB/s): min= 1816, max= 3544, per=100.00%, avg=2883.56, stdev=98.73, samples=956
   iops        : min=464936, max=907270, avg=738190.29, stdev=25274.38, samples=956
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.27%, 750=1.94%
  lat (usec)   : 1000=12.06%
  lat (msec)   : 2=79.09%, 4=6.61%, 10=0.02%, 20=0.01%
  cpu          : usr=30.12%, sys=43.53%, ctx=6669362, majf=0, minf=1077
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=88547527,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=2882MiB/s (3022MB/s), 2882MiB/s-2882MiB/s (3022MB/s-3022MB/s), io=338GiB (363GB), run=120002-120002msec

Disk stats (read/write):
  sda: ios=20162760/0, merge=68308014/0, ticks=15590425/0, in_queue=15590425, util=100.00%

Random - three times slower than the host

fio --filename=/dev/sda --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][2.5%][r=851MiB/s][r=218k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [r(4)][4.2%][r=868MiB/s][r=222k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [r(4)][5.8%][r=902MiB/s][r=231k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [r(4)][7.5%][r=870MiB/s][r=223k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [r(4)][9.2%][r=912MiB/s][r=233k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [r(4)][10.8%][r=915MiB/s][r=234k IOPS][eta 01m:47s]
Jobs: 4 (f=4): [r(4)][12.5%][r=884MiB/s][r=226k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [r(4)][14.2%][r=863MiB/s][r=221k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [r(4)][15.8%][r=944MiB/s][r=242k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [r(4)][17.5%][r=859MiB/s][r=220k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [r(4)][19.2%][r=941MiB/s][r=241k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [r(4)][20.8%][r=908MiB/s][r=232k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [r(4)][22.5%][r=903MiB/s][r=231k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [r(4)][24.2%][r=870MiB/s][r=223k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [r(4)][25.8%][r=922MiB/s][r=236k IOPS][eta 01m:29s]
Jobs: 4 (f=4): [r(4)][27.5%][r=868MiB/s][r=222k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [r(4)][29.2%][r=918MiB/s][r=235k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [r(4)][30.8%][r=925MiB/s][r=237k IOPS][eta 01m:23s]
Jobs: 4 (f=4): [r(4)][32.5%][r=836MiB/s][r=214k IOPS][eta 01m:21s]
Jobs: 4 (f=4): [r(4)][34.2%][r=919MiB/s][r=235k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [r(4)][35.8%][r=850MiB/s][r=218k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [r(4)][37.5%][r=868MiB/s][r=222k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [r(4)][39.2%][r=870MiB/s][r=223k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [r(4)][40.8%][r=898MiB/s][r=230k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [r(4)][42.5%][r=862MiB/s][r=221k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [r(4)][44.2%][r=915MiB/s][r=234k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [r(4)][45.8%][r=903MiB/s][r=231k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [r(4)][47.5%][r=877MiB/s][r=225k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [r(4)][49.2%][r=872MiB/s][r=223k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [r(4)][50.8%][r=934MiB/s][r=239k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [r(4)][52.5%][r=887MiB/s][r=227k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [r(4)][54.2%][r=857MiB/s][r=219k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [r(4)][55.8%][r=887MiB/s][r=227k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [r(4)][57.5%][r=861MiB/s][r=220k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [r(4)][59.2%][r=930MiB/s][r=238k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [r(4)][60.8%][r=867MiB/s][r=222k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [r(4)][62.5%][r=856MiB/s][r=219k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [r(4)][64.2%][r=917MiB/s][r=235k IOPS][eta 00m:43s]
Jobs: 4 (f=4): [r(4)][65.8%][r=923MiB/s][r=236k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [r(4)][67.5%][r=872MiB/s][r=223k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [r(4)][69.2%][r=913MiB/s][r=234k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [r(4)][70.8%][r=919MiB/s][r=235k IOPS][eta 00m:35s]
Jobs: 4 (f=4): [r(4)][72.5%][r=886MiB/s][r=227k IOPS][eta 00m:33s]
Jobs: 4 (f=4): [r(4)][74.2%][r=904MiB/s][r=231k IOPS][eta 00m:31s]
Jobs: 4 (f=4): [r(4)][75.8%][r=931MiB/s][r=238k IOPS][eta 00m:29s]
Jobs: 4 (f=4): [r(4)][77.5%][r=882MiB/s][r=226k IOPS][eta 00m:27s]
Jobs: 4 (f=4): [r(4)][79.2%][r=902MiB/s][r=231k IOPS][eta 00m:25s]
Jobs: 4 (f=4): [r(4)][80.8%][r=905MiB/s][r=232k IOPS][eta 00m:23s]
Jobs: 4 (f=4): [r(4)][82.5%][r=886MiB/s][r=227k IOPS][eta 00m:21s]
Jobs: 4 (f=4): [r(4)][84.2%][r=863MiB/s][r=221k IOPS][eta 00m:19s]
Jobs: 4 (f=4): [r(4)][85.8%][r=837MiB/s][r=214k IOPS][eta 00m:17s]
Jobs: 4 (f=4): [r(4)][87.5%][r=832MiB/s][r=213k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [r(4)][89.2%][r=905MiB/s][r=232k IOPS][eta 00m:13s]
Jobs: 4 (f=4): [r(4)][90.8%][r=868MiB/s][r=222k IOPS][eta 00m:11s]
Jobs: 4 (f=4): [r(4)][92.5%][r=902MiB/s][r=231k IOPS][eta 00m:09s]
Jobs: 4 (f=4): [r(4)][94.2%][r=845MiB/s][r=216k IOPS][eta 00m:07s]
Jobs: 4 (f=4): [r(4)][95.8%][r=896MiB/s][r=229k IOPS][eta 00m:05s]
Jobs: 4 (f=4): [r(4)][97.5%][r=856MiB/s][r=219k IOPS][eta 00m:03s]
Jobs: 4 (f=4): [r(4)][99.2%][r=903MiB/s][r=231k IOPS][eta 00m:01s]
Jobs: 4 (f=4): [r(4)][100.0%][r=912MiB/s][r=233k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4426: Thu Sep 30 13:32:19 2021
  read: IOPS=228k, BW=889MiB/s (932MB/s)(104GiB/120003msec)
    slat (nsec): min=1787, max=36860k, avg=4686.04, stdev=60345.87
    clat (usec): min=155, max=59143, avg=4492.07, stdev=1379.51
     lat (usec): min=173, max=59147, avg=4497.10, stdev=1381.10
    clat percentiles (usec):
     |  1.00th=[ 2474],  5.00th=[ 3032], 10.00th=[ 3359], 20.00th=[ 3654],
     | 30.00th=[ 3884], 40.00th=[ 4080], 50.00th=[ 4293], 60.00th=[ 4555],
     | 70.00th=[ 4883], 80.00th=[ 5211], 90.00th=[ 5735], 95.00th=[ 6325],
     | 99.00th=[ 7504], 99.50th=[ 8225], 99.90th=[22938], 99.95th=[28705],
     | 99.99th=[36439]
   bw (  KiB/s): min=715840, max=1121864, per=100.00%, avg=911288.44, stdev=16645.13, samples=956
   iops        : min=178960, max=280466, avg=227822.13, stdev=4161.29, samples=956
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.24%, 4=36.20%, 10=63.22%, 20=0.16%, 50=0.13%
  lat (msec)   : 100=0.01%
  cpu          : usr=10.02%, sys=22.49%, ctx=10306418, majf=0, minf=1074
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=27314852,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=889MiB/s (932MB/s), 889MiB/s-889MiB/s (932MB/s-932MB/s), io=104GiB (112GB), run=120003-120003msec

Disk stats (read/write):
  sda: ios=27289369/0, merge=119/0, ticks=111286493/0, in_queue=111286493, util=100.00%

I can live with this, given the expected load (linear reading).

 

[Roman Shein]

I believe KVM (similarly to wmWare) has implemented a nvme device class. Potentially it helps when dealing w ultrafast storage. The nvme is expected to perform using emulated native protocol, instead of using aged SCSI.
Who Pls advice how to implement