Proxmox slow nvme Speed

Digifreak · Feb 2, 2021

Hello

Hardware: Motherboard: Supermicro X10DRi
Ram: 128GB DDR4 2133MHz ECC
CPU: 2x Intel Xeon E5-2678V3 @ 2.5GHz 12Core
PCI-e Card: ASUS Hyper M.2 X16 Card
NVME: 4x Crucial P5 500GB
SSD: Samsung 830
HDD: WD Red 4TB

My Issue is that the nvme Drives are really slow and I dont know why.

Background When the OS was Windows 10 (Testing and some Benchmarks) i tried with Crystaldiskmark the nvme Speed, each drive had around 3GB/s rw When i did a Raid0 in Windows 10 OS i had around 8GB/s rw After i Installed Proxmox I made Raid0 with zfs and Installed Windows 10 VM
I Tried Crystaldiskmark and had a speed of ~ 1.2GB/s So i tried speedtest in proxmox with fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

and got around 100MB/s rw when i tried it with the ssd or hdd, I got around 350MB/s

I also tried it with ESXi, created a Windows 10 VM and ran Crystaldiskmark and got in the Windows 10 VM 3GB/s (Single nvme, cant create Raid in ESXi)
Does someone in here have a suggestion, what i could try? Im not really good in Linux, so i dont know what the issue in proxmox could be

t.lamprecht · Feb 3, 2021

Hi!

Seems likely stemming from doing two very different tests.
NVMe devices often can not be maxed out when using only a single queue and process.
Also, I'd recommend an async IO engine like libaio (io_uring would be even better, but it's not yet available in the current FIO package), then you can increase iodepth.

Try the following two and see how they work out:

Bash:

fio --rw=write --bs=4k --direct=1 --filename=/dev/nvmeXn0 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
fio --rw=write --bs=4k --direct=1 --filename=/dev/nvmeXn0 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based

Adapt the --filename parameter.
They both use a better, async IO engine and an IO Depth of 32. One uses a single job the other three.

Digifreak · Feb 3, 2021

none did achieve a speed of 3GB/s

1x Jobs

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme0n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=470MiB/s][w=120k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22604: Wed Feb  3 20:31:54 2021
  write: IOPS=259k, BW=1011MiB/s (1060MB/s)(59.2GiB/60001msec); 0 zone resets
    slat (nsec): min=1474, max=65596, avg=2203.31, stdev=1070.70
    clat (usec): min=15, max=5956, avg=120.99, stdev=82.34
     lat (usec): min=17, max=5958, avg=123.29, stdev=82.34
    clat percentiles (usec):
     |  1.00th=[   99],  5.00th=[  104], 10.00th=[  106], 20.00th=[  108],
     | 30.00th=[  109], 40.00th=[  109], 50.00th=[  109], 60.00th=[  110],
     | 70.00th=[  110], 80.00th=[  111], 90.00th=[  114], 95.00th=[  135],
     | 99.00th=[  486], 99.50th=[  660], 99.90th=[ 1123], 99.95th=[ 1336],
     | 99.99th=[ 2147]
   bw (  KiB/s): min=423944, max=1150648, per=100.00%, avg=1039669.52, stdev=238235.25, samples=119
   iops        : min=105988, max=287662, avg=259917.38, stdev=59558.81, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=1.46%, 250=95.64%, 500=1.97%
  lat (usec)   : 750=0.60%, 1000=0.19%
  lat (msec)   : 2=0.13%, 4=0.02%, 10=0.01%
  cpu          : usr=43.98%, sys=47.43%, ctx=41755, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15526449,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1011MiB/s (1060MB/s), 1011MiB/s-1011MiB/s (1060MB/s-1060MB/s), io=59.2GiB (63.6GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=63/15513741, merge=0/0, ticks=22/436893, in_queue=1032, util=99.92%

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme1n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=467MiB/s][w=120k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22230: Wed Feb  3 20:29:54 2021
  write: IOPS=122k, BW=478MiB/s (501MB/s)(28.0GiB/60002msec); 0 zone resets
    slat (nsec): min=1508, max=128874, avg=2488.83, stdev=2142.86
    clat (usec): min=74, max=18028, avg=258.41, stdev=287.21
     lat (usec): min=77, max=18032, avg=261.01, stdev=287.16
    clat percentiles (usec):
     |  1.00th=[   84],  5.00th=[   95], 10.00th=[  105], 20.00th=[  112],
     | 30.00th=[  114], 40.00th=[  119], 50.00th=[  153], 60.00th=[  208],
     | 70.00th=[  273], 80.00th=[  363], 90.00th=[  537], 95.00th=[  709],
     | 99.00th=[ 1188], 99.50th=[ 1516], 99.90th=[ 3818], 99.95th=[ 3949],
     | 99.99th=[ 7832]
   bw (  KiB/s): min=102168, max=1104080, per=99.99%, avg=489589.28, stdev=194861.99, samples=120
   iops        : min=25542, max=276020, avg=122397.32, stdev=48715.45, samples=120
  lat (usec)   : 100=7.13%, 250=59.70%, 500=21.64%, 750=7.28%, 1000=2.45%
  lat (msec)   : 2=1.57%, 4=0.19%, 10=0.04%, 20=0.01%
  cpu          : usr=24.96%, sys=27.63%, ctx=212998, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7344561,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=478MiB/s (501MB/s), 478MiB/s-478MiB/s (501MB/s-501MB/s), io=28.0GiB (30.1GB), run=60002-60002msec

Disk stats (read/write):
  nvme1n1: ios=63/7330891, merge=0/0, ticks=50/1356443, in_queue=12252, util=99.91%

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme2n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=464MiB/s][w=119k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=17407: Wed Feb  3 19:59:54 2021
  write: IOPS=124k, BW=484MiB/s (508MB/s)(28.4GiB/60001msec); 0 zone resets
    slat (nsec): min=1496, max=102375, avg=2454.57, stdev=2090.91
    clat (usec): min=74, max=29904, avg=255.23, stdev=275.85
     lat (usec): min=77, max=29905, avg=257.78, stdev=275.79
    clat percentiles (usec):
     |  1.00th=[   84],  5.00th=[   94], 10.00th=[  103], 20.00th=[  110],
     | 30.00th=[  112], 40.00th=[  120], 50.00th=[  161], 60.00th=[  215],
     | 70.00th=[  273], 80.00th=[  363], 90.00th=[  523], 95.00th=[  685],
     | 99.00th=[ 1106], 99.50th=[ 1385], 99.90th=[ 3687], 99.95th=[ 3884],
     | 99.99th=[ 7701]
   bw (  KiB/s): min=108208, max=1132552, per=100.00%, avg=495995.58, stdev=180289.82, samples=119
   iops        : min=27052, max=283138, avg=123998.91, stdev=45072.45, samples=119
  lat (usec)   : 100=7.70%, 250=58.51%, 500=22.74%, 750=7.28%, 1000=2.27%
  lat (msec)   : 2=1.30%, 4=0.16%, 10=0.04%, 20=0.01%, 50=0.01%
  cpu          : usr=24.91%, sys=27.45%, ctx=225374, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7436753,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=484MiB/s (508MB/s), 484MiB/s-484MiB/s (508MB/s-508MB/s), io=28.4GiB (30.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=63/7424971, merge=0/0, ticks=56/1374981, in_queue=12972, util=99.93%

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme3n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=444MiB/s][w=114k IOPS][eta 00m:00s] 
test: (groupid=0, jobs=1): err= 0: pid=17080: Wed Feb  3 19:58:13 2021
  write: IOPS=119k, BW=463MiB/s (486MB/s)(27.2GiB/60001msec); 0 zone resets
    slat (nsec): min=1458, max=309216, avg=2402.88, stdev=2143.09
    clat (usec): min=73, max=15246, avg=266.81, stdev=319.96
     lat (usec): min=75, max=15248, avg=269.31, stdev=320.02
    clat percentiles (usec):
     |  1.00th=[   82],  5.00th=[   92], 10.00th=[  101], 20.00th=[  105],
     | 30.00th=[  108], 40.00th=[  119], 50.00th=[  163], 60.00th=[  217],
     | 70.00th=[  277], 80.00th=[  367], 90.00th=[  537], 95.00th=[  725],
     | 99.00th=[ 1450], 99.50th=[ 2008], 99.90th=[ 3720], 99.95th=[ 4015],
     | 99.99th=[ 7963]
   bw (  KiB/s): min=48328, max=1167728, per=99.99%, avg=474548.20, stdev=216585.23, samples=120
   iops        : min=12082, max=291932, avg=118637.02, stdev=54146.33, samples=120
  lat (usec)   : 100=9.46%, 250=56.34%, 500=22.50%, 750=7.17%, 1000=2.29%
  lat (msec)   : 2=1.73%, 4=0.45%, 10=0.05%, 20=0.01%
  cpu          : usr=23.03%, sys=26.28%, ctx=225073, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7118855,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s), io=27.2GiB (29.2GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=63/7106158, merge=0/0, ticks=52/1421294, in_queue=16204, util=99.91%

Digifreak · Feb 3, 2021

3x Jobs

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme0n1 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based

test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32

...

fio-3.12

Starting 3 processes

Jobs: 3 (f=3): [W(3)][100.0%][w=502MiB/s][w=128k IOPS][eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=12993: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=471MiB/s (494MB/s)(27.6GiB/60001msec); 0 zone resets

    slat (nsec): min=1499, max=354182, avg=2228.50, stdev=1288.75

    clat (usec): min=78, max=20077, avg=262.57, stdev=291.75

     lat (usec): min=81, max=20078, avg=264.91, stdev=291.99

    clat percentiles (usec):

     |  1.00th=[  157],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  914],

     | 99.00th=[ 1483], 99.50th=[ 1663], 99.90th=[ 2180], 99.95th=[ 2606],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=128744, max=742632, per=33.27%, avg=482372.53, stdev=289635.59, samples=120

   iops        : min=32186, max=185658, avg=120593.10, stdev=72408.87, samples=120

  lat (usec)   : 100=0.03%, 250=86.67%, 500=2.12%, 750=3.64%, 1000=3.49%

  lat (msec)   : 2=3.92%, 4=0.13%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.41%, sys=27.48%, ctx=1954862, majf=0, minf=85

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7235652,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=12994: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=473MiB/s (495MB/s)(27.7GiB/60001msec); 0 zone resets

    slat (nsec): min=1539, max=113413, avg=2278.22, stdev=1180.38

    clat (usec): min=79, max=20170, avg=261.71, stdev=288.09

     lat (usec): min=81, max=20172, avg=264.10, stdev=288.24

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  906],

     | 99.00th=[ 1467], 99.50th=[ 1647], 99.90th=[ 2180], 99.95th=[ 2573],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=131720, max=742584, per=33.37%, avg=483854.45, stdev=288189.30, samples=120

   iops        : min=32930, max=185646, avg=120963.63, stdev=72047.34, samples=120

  lat (usec)   : 100=0.01%, 250=86.54%, 500=2.23%, 750=3.83%, 1000=3.48%

  lat (msec)   : 2=3.78%, 4=0.12%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.80%, sys=27.51%, ctx=1884742, majf=0, minf=20

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7257884,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=12995: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=472MiB/s (495MB/s)(27.7GiB/60001msec); 0 zone resets

    slat (nsec): min=1538, max=190223, avg=2278.79, stdev=1190.31

    clat (usec): min=78, max=20199, avg=261.78, stdev=288.28

     lat (usec): min=82, max=20201, avg=264.17, stdev=288.44

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  906],

     | 99.00th=[ 1467], 99.50th=[ 1631], 99.90th=[ 2180], 99.95th=[ 2573],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=131816, max=742784, per=33.36%, avg=483695.31, stdev=287995.45, samples=120

   iops        : min=32954, max=185696, avg=120923.83, stdev=71998.87, samples=120

  lat (usec)   : 100=0.01%, 250=86.54%, 500=2.23%, 750=3.83%, 1000=3.47%

  lat (msec)   : 2=3.77%, 4=0.12%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.09%, sys=28.25%, ctx=1884453, majf=0, minf=19

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7255830,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):

  WRITE: bw=1416MiB/s (1485MB/s), 471MiB/s-473MiB/s (494MB/s-495MB/s), io=82.0GiB (89.1GB), run=60001-60001msec


Disk stats (read/write):

  nvme0n1: ios=116/21737538, merge=0/0, ticks=37/5579850, in_queue=17292, util=99.97%

Code:

fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme1n1 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based

test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32

...

fio-3.12

Starting 3 processes

Jobs: 3 (f=3): [W(3)][100.0%][w=460MiB/s][w=118k IOPS][eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=13726: Wed Feb  3 19:38:09 2021

  write: IOPS=39.7k, BW=155MiB/s (163MB/s)(9299MiB/60001msec); 0 zone resets

    slat (nsec): min=1493, max=120418, avg=2803.77, stdev=2340.81

    clat (usec): min=74, max=20896, avg=803.15, stdev=755.49

     lat (usec): min=78, max=20900, avg=806.06, stdev=755.65

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  165], 10.00th=[  169], 20.00th=[  227],

     | 30.00th=[  457], 40.00th=[  594], 50.00th=[  701], 60.00th=[  799],

     | 70.00th=[  930], 80.00th=[ 1106], 90.00th=[ 1401], 95.00th=[ 1762],

     | 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[10159], 99.95th=[11863],

     | 99.99th=[14353]

   bw (  KiB/s): min=11368, max=732480, per=33.53%, avg=158673.76, stdev=113697.48, samples=119

   iops        : min= 2842, max=183120, avg=39668.46, stdev=28424.36, samples=119

  lat (usec)   : 100=0.05%, 250=20.70%, 500=11.95%, 750=22.14%, 1000=19.67%

  lat (msec)   : 2=21.68%, 4=3.10%, 10=0.60%, 20=0.10%, 50=0.01%

  cpu          : usr=9.75%, sys=11.75%, ctx=553881, majf=0, minf=16

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2380478,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=13727: Wed Feb  3 19:38:09 2021

  write: IOPS=39.0k, BW=152MiB/s (160MB/s)(9141MiB/60001msec); 0 zone resets

    slat (nsec): min=1460, max=572622, avg=2945.77, stdev=2599.31

    clat (usec): min=74, max=20923, avg=816.87, stdev=761.17

     lat (usec): min=77, max=20925, avg=819.94, stdev=761.35

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  163], 10.00th=[  169], 20.00th=[  225],

     | 30.00th=[  465], 40.00th=[  611], 50.00th=[  717], 60.00th=[  824],

     | 70.00th=[  947], 80.00th=[ 1123], 90.00th=[ 1434], 95.00th=[ 1795],

     | 99.00th=[ 3720], 99.50th=[ 4424], 99.90th=[ 9765], 99.95th=[11731],

     | 99.99th=[14091]

   bw (  KiB/s): min=12352, max=733016, per=32.97%, avg=156003.84, stdev=113365.82, samples=120

   iops        : min= 3088, max=183254, avg=39000.96, stdev=28341.45, samples=120

  lat (usec)   : 100=0.11%, 250=20.74%, 500=11.07%, 750=21.25%, 1000=20.23%

  lat (msec)   : 2=22.66%, 4=3.18%, 10=0.65%, 20=0.10%, 50=0.01%

  cpu          : usr=9.72%, sys=12.65%, ctx=478163, majf=0, minf=74

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2340131,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=13728: Wed Feb  3 19:38:09 2021

  write: IOPS=39.6k, BW=155MiB/s (162MB/s)(9289MiB/60001msec); 0 zone resets

    slat (nsec): min=1513, max=98327, avg=2828.69, stdev=2342.69

    clat (usec): min=76, max=20900, avg=803.95, stdev=756.93

     lat (usec): min=77, max=20905, avg=806.88, stdev=757.09

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  165], 10.00th=[  169], 20.00th=[  229],

     | 30.00th=[  457], 40.00th=[  594], 50.00th=[  701], 60.00th=[  807],

     | 70.00th=[  930], 80.00th=[ 1106], 90.00th=[ 1418], 95.00th=[ 1762],

     | 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[10159], 99.95th=[11863],

     | 99.99th=[14484]

   bw (  KiB/s): min=11384, max=731904, per=33.50%, avg=158519.17, stdev=113624.24, samples=119

   iops        : min= 2846, max=182976, avg=39629.76, stdev=28405.98, samples=119

  lat (usec)   : 100=0.05%, 250=20.69%, 500=11.93%, 750=22.14%, 1000=19.67%

  lat (msec)   : 2=21.71%, 4=3.10%, 10=0.60%, 20=0.10%, 50=0.01%

  cpu          : usr=9.83%, sys=11.84%, ctx=546526, majf=0, minf=17

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2378028,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):

  WRITE: bw=462MiB/s (485MB/s), 152MiB/s-155MiB/s (160MB/s-163MB/s), io=27.1GiB (29.1GB), run=60001-60001msec


Disk stats (read/write):

  nvme1n1: ios=118/7083837, merge=0/0, ticks=64/5602516, in_queue=237168, util=99.92%

Had to delete the last 2 Benchmark, because of limited Forum size, bellow a screenshot of Windows 10, with Crystaldiskmark, to show that 3GB/s is possible

spirit · Feb 4, 2021

if you use num-jobs, you should use --group_reporting. (or you'll have separated results for each job)

also, maybe try with more jobs, and lower queue depth. (it's possible that fio process is cpu bound, so more jobs=more fio process)

Digifreak · Feb 4, 2021

spirit said:
if you use num-jobs, you should use --group_reporting. (or you'll have separated results for each job)

also, maybe try with more jobs, and lower queue depth. (it's possible that fio process is cpu bound, so more jobs=more fio process)

with some tweaking i got around 2GB/s

t.lamprecht · Feb 5, 2021

Sounds not to bad for such a fio command with 4 KiB blocks!
I mean, different benchmark tools are differing in their approach a lot. For example, I took another look at your screenshot and saw that CrystalDiskMark uses 1 MiB and 128 KiB block sizes for the read tests (one with Q depth of 8 and one with Q depth of 32, if my guess from the test labels is correct - did not use that software since years) whereas the fio command always used 4 KiB, meaning much more IOPS to be handled by the disk.

Digifreak · Feb 5, 2021

Did some tweaking and managed to get that speed in a vm. So im not sure if that "low speed" is because of different benchmarks

If the drive has a different block size, does this "decrease" the overall performance or is it just a visual decrease?

diversity · May 5, 2021

Could you please elaborate on the tweaking you did?

I am getting abysmal results on 2 x Samsung SSD 980 PRO zfs mirrored using default zfs settings during proxmox installation. Used PVE < 6.4 for installation and currently fully upgraded to most recent version.
pve-manager/6.4-4/337d6701 (running kernel: 5.11.7-1-pve)

results when running:
fio --ioengine=psync --filename=/rpool/ROOT/pve-1/test/test --size=9G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --iodepth=1 --rw=write --bs=4k --numjobs=32
is about 10 x slower than found in
https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744/

Any tips on getting normal performance would be great

f!R3W4LL · Jul 16, 2021

diversity said:
Could you please elaborate on the tweaking you did?

I am getting abysmal results on 2 x Samsung SSD 980 PRO zfs mirrored using default zfs settings during proxmox installation. Used PVE < 6.4 for installation and currently fully upgraded to most recent version.
pve-manager/6.4-4/337d6701 (running kernel: 5.11.7-1-pve)

results when running:
fio --ioengine=psync --filename=/rpool/ROOT/pve-1/test/test --size=9G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --iodepth=1 --rw=write --bs=4k --numjobs=32
is about 10 x slower than found in
https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744/

Any tips on getting normal performance would be great

Interested to…. I installed proxmox yesterday but the vm speed of ubuntu is so slow.
What can i do to get the normal speed i had in esxi? More or less?

t.lamprecht · Jul 17, 2021

f!R3W4LL said:
Interested to…. I installed proxmox yesterday but the vm speed of ubuntu is so slow.

Determined by what measurement? And it would be interesting what storage/FS type did you choose for the VM.

f!R3W4LL · Jul 17, 2021

t.lamprecht said:
Determined by what measurement? And it would be interesting what storage/FS type did you choose for the VM.

I found the problem! I dint see that, every time i created a new vm, the standard setting of the cd/dvd drive for the iso, was on IDE! Im so fucking dumb! I tought that the vm was slow, or my NVME! Storage type i left it on standard scsi!

Search

Search

Proxmox slow nvme Speed

Digifreak

New Member

t.lamprecht

Proxmox Staff Member

Digifreak

New Member

Digifreak

New Member

spirit

Distinguished Member

Digifreak

New Member

t.lamprecht

Proxmox Staff Member

Digifreak

New Member

diversity

Active Member

f!R3W4LL

Member

t.lamprecht

Proxmox Staff Member

f!R3W4LL

Member