NVMe SSD. Slow disk speed inside VM

F

FessAectan

Member
Sep 29, 2021
2
0
6
38
Hi there!

I faced a problem with slow disk speed performance inside a VM (when a VM has a disk on host NVMe SSD).
For instance, when I copy 10GB file from one pve to another pve (LAN 10Gbit/s):
Bash: 
rsync -P -avz file10G 10.1.123.1:
sending incremental file list
file10G
 10,737,418,240 100%    2.84GB/s    0:00:03 (xfr#1, to-chk=0/1)

pve (host) to VM
Bash: 
rsync -P -avz file10G 10.1.123.206:
sending incremental file list
file10G
    617,086,976   5%  196.29MB/s    0:00:50

Test with fio. Host
Code: 
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.25
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=445MiB/s][w=114k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=4129535: Wed Sep 29 14:37:52 2021
  write: IOPS=97.4k, BW=380MiB/s (399MB/s)(22.4GiB/60414msec); 0 zone resets
    slat (nsec): min=533, max=1353.4k, avg=1474.47, stdev=821.76
    clat (nsec): min=117, max=4367.8k, avg=6619.21, stdev=9392.24
     lat (usec): min=4, max=4369, avg= 8.09, stdev= 9.46
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   22], 99.90th=[   53], 99.95th=[   75],
     | 99.99th=[  221]
   bw (  KiB/s): min=20456, max=506048, per=100.00%, avg=452587.73, stdev=74042.85, samples=103
   iops        : min= 5114, max=126512, avg=113146.95, stdev=18510.62, samples=103
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.51%, 20=1.95%, 50=0.42%
  lat (usec)   : 100=0.08%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=18.13%, sys=34.95%, ctx=5888188, majf=0, minf=48
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5883316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=22.4GiB (24.1GB), run=60414-60414msec

Disk stats (read/write):
    dm-6: ios=0/818584, merge=0/0, ticks=0/574964, in_queue=574964, util=14.24%, aggrios=133/944809, aggrmerge=5/2211, aggrticks=29/1313275, aggrin_queue=1347064, aggrutil=70.51%
  nvme0n1: ios=133/944809, merge=5/2211, ticks=29/1313275, in_queue=1347064, util=70.51%

VM
Code: 
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.16
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=182MiB/s][w=46.7k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2010: Wed Sep 29 12:36:21 2021
  write: IOPS=36.9k, BW=144MiB/s (151MB/s)(8775MiB/60803msec); 0 zone resets
    slat (nsec): min=506, max=1004.6k, avg=4101.30, stdev=2293.91
    clat (nsec): min=166, max=207411k, avg=19751.96, stdev=285681.11
     lat (usec): min=6, max=207414, avg=23.85, stdev=285.70
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   14], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   16], 50.00th=[   17], 60.00th=[   18],
     | 70.00th=[   19], 80.00th=[   21], 90.00th=[   23], 95.00th=[   27],
     | 99.00th=[   51], 99.50th=[   65], 99.90th=[  176], 99.95th=[  506],
     | 99.99th=[ 1319]
   bw (  KiB/s): min=17576, max=237512, per=100.00%, avg=158797.04, stdev=32794.76, samples=112
   iops        : min= 4394, max=59378, avg=39699.25, stdev=8198.70, samples=112
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.99%, 20=78.22%, 50=19.75%
  lat (usec)   : 100=0.83%, 250=0.14%, 500=0.02%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=14.97%, sys=36.10%, ctx=2282024, majf=0, minf=58
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2246364,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8775MiB (9201MB), run=60803-60803msec

Disk stats (read/write):
  sda: ios=170/415892, merge=32/1899, ticks=32/154140, in_queue=17188, util=9.19%


The VM config
Code: 
agent: 1
bios: seabios
boot: c
bootdisk: scsi0
ciuser: ololosha
cores: 2
cpu: host
description: ClickHouse server ch-01
hotplug: network,disk,usb
ide2: NVMe2:vm-115-cloudinit,media=cdrom,size=4M
ipconfig0: ip=10.1.123.206/24,gw=10.1.123.2
kvm: 1
memory: 65536
name: ch-01
net0: virtio=DA:8B:B7:52:AA:A3,bridge=vmbr1
numa: 0
onboot: 1
scsi0: NVMe2:vm-115-disk-0,backup=0,discard=on,iothread=1,size=30G,ssd=1
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=c9567916-c96f-47d8-bd77-ae448bb739ec
sockets: 2

VM OS details
Code: 
uname -a
Linux ch-01 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID:   Ubuntu
Description:   Ubuntu 20.04.3 LTS
Release:   20.04
Codename:   focal

dpkg -l | grep qemu
ii  qemu-guest-agent               1:4.2-3ubuntu6.17                     amd64        Guest-side qemu-system agent

I have "Proxmox Virtual Environment 7.0-11" on the host.
Disks for VMs provided as LVM.
A part of my /etc/pve/storage.cfg
Code: 
lvm: NVMe2
    vgname VM_disks
    content images,rootdir
    nodes pve02
    shared 0

What can I do to achieve disk speed in a VM like speed on a host?
 
Last edited:
Linear reading turned out to be quite sane
Code: 
fio --filename=/dev/sda --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
iops-test-job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=2025MiB/s][r=518k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=2042MiB/s][r=523k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=2212MiB/s][r=566k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=3060MiB/s][r=783k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=3055MiB/s][r=782k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=3024MiB/s][r=774k IOPS][eta 01m:46s]
Jobs: 4 (f=4): [R(4)][12.5%][r=2966MiB/s][r=759k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [R(4)][14.2%][r=3082MiB/s][r=789k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [R(4)][15.8%][r=2977MiB/s][r=762k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [R(4)][17.5%][r=3094MiB/s][r=792k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [R(4)][19.2%][r=3001MiB/s][r=768k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [R(4)][20.8%][r=3023MiB/s][r=774k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [R(4)][22.5%][r=3053MiB/s][r=782k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [R(4)][24.2%][r=2979MiB/s][r=763k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [R(4)][26.1%][r=3076MiB/s][r=788k IOPS][eta 01m:28s]
Jobs: 4 (f=4): [R(4)][27.5%][r=2896MiB/s][r=741k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [R(4)][29.2%][r=2922MiB/s][r=748k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [R(4)][31.1%][r=2985MiB/s][r=764k IOPS][eta 01m:22s]
Jobs: 4 (f=4): [R(4)][32.8%][r=2997MiB/s][r=767k IOPS][eta 01m:20s]
Jobs: 4 (f=4): [R(4)][34.2%][r=3038MiB/s][r=778k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [R(4)][35.8%][r=3053MiB/s][r=782k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [R(4)][37.5%][r=3033MiB/s][r=776k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [R(4)][39.2%][r=2542MiB/s][r=651k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][40.8%][r=2004MiB/s][r=513k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [R(4)][42.5%][r=2050MiB/s][r=525k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [R(4)][44.2%][r=2410MiB/s][r=617k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [R(4)][45.8%][r=2950MiB/s][r=755k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [R(4)][47.5%][r=3072MiB/s][r=786k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [R(4)][49.2%][r=3016MiB/s][r=772k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [R(4)][50.8%][r=3039MiB/s][r=778k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [R(4)][52.5%][r=3154MiB/s][r=808k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [R(4)][54.2%][r=3106MiB/s][r=795k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [R(4)][55.8%][r=3224MiB/s][r=825k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [R(4)][57.5%][r=3007MiB/s][r=770k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [R(4)][59.2%][r=2985MiB/s][r=764k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [R(4)][60.8%][r=3000MiB/s][r=768k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [R(4)][62.5%][r=3036MiB/s][r=777k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [R(4)][64.7%][r=3137MiB/s][r=803k IOPS][eta 00m:42s]
Jobs: 4 (f=4): [R(4)][65.8%][r=3131MiB/s][r=802k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [R(4)][67.5%][r=3037MiB/s][r=778k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [R(4)][69.2%][r=2978MiB/s][r=762k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [R(4)][71.4%][r=3170MiB/s][r=811k IOPS][eta 00m:34s]
Jobs: 4 (f=4): [R(4)][72.5%][r=3036MiB/s][r=777k IOPS][eta 00m:33s]
Jobs: 4 (f=4): [R(4)][74.2%][r=3061MiB/s][r=784k IOPS][eta 00m:31s]
Jobs: 4 (f=4): [R(4)][75.8%][r=2719MiB/s][r=696k IOPS][eta 00m:29s]
Jobs: 4 (f=4): [R(4)][77.5%][r=2467MiB/s][r=631k IOPS][eta 00m:27s]
Jobs: 4 (f=4): [R(4)][79.2%][r=2141MiB/s][r=548k IOPS][eta 00m:25s]
Jobs: 4 (f=4): [R(4)][80.8%][r=2658MiB/s][r=680k IOPS][eta 00m:23s]
Jobs: 4 (f=4): [R(4)][82.5%][r=2620MiB/s][r=671k IOPS][eta 00m:21s]
Jobs: 4 (f=4): [R(4)][84.2%][r=2938MiB/s][r=752k IOPS][eta 00m:19s]
Jobs: 4 (f=4): [R(4)][85.8%][r=3157MiB/s][r=808k IOPS][eta 00m:17s]
Jobs: 4 (f=4): [R(4)][87.5%][r=3081MiB/s][r=789k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [R(4)][89.9%][r=3072MiB/s][r=786k IOPS][eta 00m:12s]
Jobs: 4 (f=4): [R(4)][90.8%][r=3120MiB/s][r=799k IOPS][eta 00m:11s]
Jobs: 4 (f=4): [R(4)][93.3%][r=3119MiB/s][r=799k IOPS][eta 00m:08s]
Jobs: 4 (f=4): [R(4)][94.2%][r=3112MiB/s][r=797k IOPS][eta 00m:07s]
Jobs: 4 (f=4): [R(4)][95.8%][r=3178MiB/s][r=814k IOPS][eta 00m:05s]
Jobs: 4 (f=4): [R(4)][97.5%][r=3171MiB/s][r=812k IOPS][eta 00m:03s]
Jobs: 4 (f=4): [R(4)][99.2%][r=3013MiB/s][r=771k IOPS][eta 00m:01s]
Jobs: 4 (f=4): [R(4)][100.0%][r=3072MiB/s][r=786k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4413: Thu Sep 30 13:29:27 2021
  read: IOPS=738k, BW=2882MiB/s (3022MB/s)(338GiB/120002msec)
    slat (nsec): min=1355, max=7830.0k, avg=2674.53, stdev=4210.37
    clat (usec): min=29, max=12364, avg=1384.28, stdev=405.35
     lat (usec): min=33, max=12366, avg=1387.17, stdev=405.70
    clat percentiles (usec):
     |  1.00th=[  611],  5.00th=[  906], 10.00th=[  963], 20.00th=[ 1045],
     | 30.00th=[ 1123], 40.00th=[ 1221], 50.00th=[ 1303], 60.00th=[ 1418],
     | 70.00th=[ 1565], 80.00th=[ 1729], 90.00th=[ 1893], 95.00th=[ 2089],
     | 99.00th=[ 2606], 99.50th=[ 2769], 99.90th=[ 3228], 99.95th=[ 3490],
     | 99.99th=[ 5014]
   bw (  MiB/s): min= 1816, max= 3544, per=100.00%, avg=2883.56, stdev=98.73, samples=956
   iops        : min=464936, max=907270, avg=738190.29, stdev=25274.38, samples=956
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.27%, 750=1.94%
  lat (usec)   : 1000=12.06%
  lat (msec)   : 2=79.09%, 4=6.61%, 10=0.02%, 20=0.01%
  cpu          : usr=30.12%, sys=43.53%, ctx=6669362, majf=0, minf=1077
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=88547527,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=2882MiB/s (3022MB/s), 2882MiB/s-2882MiB/s (3022MB/s-3022MB/s), io=338GiB (363GB), run=120002-120002msec

Disk stats (read/write):
  sda: ios=20162760/0, merge=68308014/0, ticks=15590425/0, in_queue=15590425, util=100.00%

Random - three times slower than the host
Code: 
fio --filename=/dev/sda --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][2.5%][r=851MiB/s][r=218k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [r(4)][4.2%][r=868MiB/s][r=222k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [r(4)][5.8%][r=902MiB/s][r=231k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [r(4)][7.5%][r=870MiB/s][r=223k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [r(4)][9.2%][r=912MiB/s][r=233k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [r(4)][10.8%][r=915MiB/s][r=234k IOPS][eta 01m:47s]
Jobs: 4 (f=4): [r(4)][12.5%][r=884MiB/s][r=226k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [r(4)][14.2%][r=863MiB/s][r=221k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [r(4)][15.8%][r=944MiB/s][r=242k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [r(4)][17.5%][r=859MiB/s][r=220k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [r(4)][19.2%][r=941MiB/s][r=241k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [r(4)][20.8%][r=908MiB/s][r=232k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [r(4)][22.5%][r=903MiB/s][r=231k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [r(4)][24.2%][r=870MiB/s][r=223k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [r(4)][25.8%][r=922MiB/s][r=236k IOPS][eta 01m:29s]
Jobs: 4 (f=4): [r(4)][27.5%][r=868MiB/s][r=222k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [r(4)][29.2%][r=918MiB/s][r=235k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [r(4)][30.8%][r=925MiB/s][r=237k IOPS][eta 01m:23s]
Jobs: 4 (f=4): [r(4)][32.5%][r=836MiB/s][r=214k IOPS][eta 01m:21s]
Jobs: 4 (f=4): [r(4)][34.2%][r=919MiB/s][r=235k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [r(4)][35.8%][r=850MiB/s][r=218k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [r(4)][37.5%][r=868MiB/s][r=222k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [r(4)][39.2%][r=870MiB/s][r=223k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [r(4)][40.8%][r=898MiB/s][r=230k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [r(4)][42.5%][r=862MiB/s][r=221k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [r(4)][44.2%][r=915MiB/s][r=234k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [r(4)][45.8%][r=903MiB/s][r=231k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [r(4)][47.5%][r=877MiB/s][r=225k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [r(4)][49.2%][r=872MiB/s][r=223k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [r(4)][50.8%][r=934MiB/s][r=239k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [r(4)][52.5%][r=887MiB/s][r=227k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [r(4)][54.2%][r=857MiB/s][r=219k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [r(4)][55.8%][r=887MiB/s][r=227k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [r(4)][57.5%][r=861MiB/s][r=220k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [r(4)][59.2%][r=930MiB/s][r=238k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [r(4)][60.8%][r=867MiB/s][r=222k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [r(4)][62.5%][r=856MiB/s][r=219k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [r(4)][64.2%][r=917MiB/s][r=235k IOPS][eta 00m:43s]
Jobs: 4 (f=4): [r(4)][65.8%][r=923MiB/s][r=236k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [r(4)][67.5%][r=872MiB/s][r=223k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [r(4)][69.2%][r=913MiB/s][r=234k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [r(4)][70.8%][r=919MiB/s][r=235k IOPS][eta 00m:35s]
Jobs: 4 (f=4): [r(4)][72.5%][r=886MiB/s][r=227k IOPS][eta 00m:33s]
Jobs: 4 (f=4): [r(4)][74.2%][r=904MiB/s][r=231k IOPS][eta 00m:31s]
Jobs: 4 (f=4): [r(4)][75.8%][r=931MiB/s][r=238k IOPS][eta 00m:29s]
Jobs: 4 (f=4): [r(4)][77.5%][r=882MiB/s][r=226k IOPS][eta 00m:27s]
Jobs: 4 (f=4): [r(4)][79.2%][r=902MiB/s][r=231k IOPS][eta 00m:25s]
Jobs: 4 (f=4): [r(4)][80.8%][r=905MiB/s][r=232k IOPS][eta 00m:23s]
Jobs: 4 (f=4): [r(4)][82.5%][r=886MiB/s][r=227k IOPS][eta 00m:21s]
Jobs: 4 (f=4): [r(4)][84.2%][r=863MiB/s][r=221k IOPS][eta 00m:19s]
Jobs: 4 (f=4): [r(4)][85.8%][r=837MiB/s][r=214k IOPS][eta 00m:17s]
Jobs: 4 (f=4): [r(4)][87.5%][r=832MiB/s][r=213k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [r(4)][89.2%][r=905MiB/s][r=232k IOPS][eta 00m:13s]
Jobs: 4 (f=4): [r(4)][90.8%][r=868MiB/s][r=222k IOPS][eta 00m:11s]
Jobs: 4 (f=4): [r(4)][92.5%][r=902MiB/s][r=231k IOPS][eta 00m:09s]
Jobs: 4 (f=4): [r(4)][94.2%][r=845MiB/s][r=216k IOPS][eta 00m:07s]
Jobs: 4 (f=4): [r(4)][95.8%][r=896MiB/s][r=229k IOPS][eta 00m:05s]
Jobs: 4 (f=4): [r(4)][97.5%][r=856MiB/s][r=219k IOPS][eta 00m:03s]
Jobs: 4 (f=4): [r(4)][99.2%][r=903MiB/s][r=231k IOPS][eta 00m:01s]
Jobs: 4 (f=4): [r(4)][100.0%][r=912MiB/s][r=233k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4426: Thu Sep 30 13:32:19 2021
  read: IOPS=228k, BW=889MiB/s (932MB/s)(104GiB/120003msec)
    slat (nsec): min=1787, max=36860k, avg=4686.04, stdev=60345.87
    clat (usec): min=155, max=59143, avg=4492.07, stdev=1379.51
     lat (usec): min=173, max=59147, avg=4497.10, stdev=1381.10
    clat percentiles (usec):
     |  1.00th=[ 2474],  5.00th=[ 3032], 10.00th=[ 3359], 20.00th=[ 3654],
     | 30.00th=[ 3884], 40.00th=[ 4080], 50.00th=[ 4293], 60.00th=[ 4555],
     | 70.00th=[ 4883], 80.00th=[ 5211], 90.00th=[ 5735], 95.00th=[ 6325],
     | 99.00th=[ 7504], 99.50th=[ 8225], 99.90th=[22938], 99.95th=[28705],
     | 99.99th=[36439]
   bw (  KiB/s): min=715840, max=1121864, per=100.00%, avg=911288.44, stdev=16645.13, samples=956
   iops        : min=178960, max=280466, avg=227822.13, stdev=4161.29, samples=956
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.24%, 4=36.20%, 10=63.22%, 20=0.16%, 50=0.13%
  lat (msec)   : 100=0.01%
  cpu          : usr=10.02%, sys=22.49%, ctx=10306418, majf=0, minf=1074
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=27314852,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=889MiB/s (932MB/s), 889MiB/s-889MiB/s (932MB/s-932MB/s), io=104GiB (112GB), run=120003-120003msec

Disk stats (read/write):
  sda: ios=27289369/0, merge=119/0, ticks=111286493/0, in_queue=111286493, util=100.00%

I can live with this, given the expected load (linear reading).
 
I believe KVM (similarly to wmWare) has implemented a nvme device class. Potentially it helps when dealing w ultrafast storage. The nvme is expected to perform using emulated native protocol, instead of using aged SCSI.
Who Pls advice how to implement
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back