Proxmox slow nvme Speed

D

Digifreak

New Member
Feb 2, 2021
5
0
1
Hello

Hardware: Motherboard: Supermicro X10DRi
Ram: 128GB DDR4 2133MHz ECC
CPU: 2x Intel Xeon E5-2678V3 @ 2.5GHz 12Core
PCI-e Card: ASUS Hyper M.2 X16 Card
NVME: 4x Crucial P5 500GB
SSD: Samsung 830
HDD: WD Red 4TB

My Issue is that the nvme Drives are really slow and I dont know why.

Background When the OS was Windows 10 (Testing and some Benchmarks) i tried with Crystaldiskmark the nvme Speed, each drive had around 3GB/s rw When i did a Raid0 in Windows 10 OS i had around 8GB/s rw After i Installed Proxmox I made Raid0 with zfs and Installed Windows 10 VM
I Tried Crystaldiskmark and had a speed of ~ 1.2GB/s So i tried speedtest in proxmox with fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

and got around 100MB/s rw when i tried it with the ssd or hdd, I got around 350MB/s

I also tried it with ESXi, created a Windows 10 VM and ran Crystaldiskmark and got in the Windows 10 VM 3GB/s (Single nvme, cant create Raid in ESXi)
Does someone in here have a suggestion, what i could try? Im not really good in Linux, so i dont know what the issue in proxmox could be
 
Hi!

Seems likely stemming from doing two very different tests.
NVMe devices often can not be maxed out when using only a single queue and process.
Also, I'd recommend an async IO engine like libaio (io_uring would be even better, but it's not yet available in the current FIO package), then you can increase iodepth.

Try the following two and see how they work out:
Bash: 
fio --rw=write --bs=4k --direct=1 --filename=/dev/nvmeXn0 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
fio --rw=write --bs=4k --direct=1 --filename=/dev/nvmeXn0 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
Adapt the --filename parameter.
They both use a better, async IO engine and an IO Depth of 32. One uses a single job the other three.
 
Last edited:
none did achieve a speed of 3GB/s

1x Jobs

Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme0n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=470MiB/s][w=120k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22604: Wed Feb  3 20:31:54 2021
  write: IOPS=259k, BW=1011MiB/s (1060MB/s)(59.2GiB/60001msec); 0 zone resets
    slat (nsec): min=1474, max=65596, avg=2203.31, stdev=1070.70
    clat (usec): min=15, max=5956, avg=120.99, stdev=82.34
     lat (usec): min=17, max=5958, avg=123.29, stdev=82.34
    clat percentiles (usec):
     |  1.00th=[   99],  5.00th=[  104], 10.00th=[  106], 20.00th=[  108],
     | 30.00th=[  109], 40.00th=[  109], 50.00th=[  109], 60.00th=[  110],
     | 70.00th=[  110], 80.00th=[  111], 90.00th=[  114], 95.00th=[  135],
     | 99.00th=[  486], 99.50th=[  660], 99.90th=[ 1123], 99.95th=[ 1336],
     | 99.99th=[ 2147]
   bw (  KiB/s): min=423944, max=1150648, per=100.00%, avg=1039669.52, stdev=238235.25, samples=119
   iops        : min=105988, max=287662, avg=259917.38, stdev=59558.81, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=1.46%, 250=95.64%, 500=1.97%
  lat (usec)   : 750=0.60%, 1000=0.19%
  lat (msec)   : 2=0.13%, 4=0.02%, 10=0.01%
  cpu          : usr=43.98%, sys=47.43%, ctx=41755, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15526449,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1011MiB/s (1060MB/s), 1011MiB/s-1011MiB/s (1060MB/s-1060MB/s), io=59.2GiB (63.6GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=63/15513741, merge=0/0, ticks=22/436893, in_queue=1032, util=99.92%


Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme1n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=467MiB/s][w=120k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22230: Wed Feb  3 20:29:54 2021
  write: IOPS=122k, BW=478MiB/s (501MB/s)(28.0GiB/60002msec); 0 zone resets
    slat (nsec): min=1508, max=128874, avg=2488.83, stdev=2142.86
    clat (usec): min=74, max=18028, avg=258.41, stdev=287.21
     lat (usec): min=77, max=18032, avg=261.01, stdev=287.16
    clat percentiles (usec):
     |  1.00th=[   84],  5.00th=[   95], 10.00th=[  105], 20.00th=[  112],
     | 30.00th=[  114], 40.00th=[  119], 50.00th=[  153], 60.00th=[  208],
     | 70.00th=[  273], 80.00th=[  363], 90.00th=[  537], 95.00th=[  709],
     | 99.00th=[ 1188], 99.50th=[ 1516], 99.90th=[ 3818], 99.95th=[ 3949],
     | 99.99th=[ 7832]
   bw (  KiB/s): min=102168, max=1104080, per=99.99%, avg=489589.28, stdev=194861.99, samples=120
   iops        : min=25542, max=276020, avg=122397.32, stdev=48715.45, samples=120
  lat (usec)   : 100=7.13%, 250=59.70%, 500=21.64%, 750=7.28%, 1000=2.45%
  lat (msec)   : 2=1.57%, 4=0.19%, 10=0.04%, 20=0.01%
  cpu          : usr=24.96%, sys=27.63%, ctx=212998, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7344561,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=478MiB/s (501MB/s), 478MiB/s-478MiB/s (501MB/s-501MB/s), io=28.0GiB (30.1GB), run=60002-60002msec

Disk stats (read/write):
  nvme1n1: ios=63/7330891, merge=0/0, ticks=50/1356443, in_queue=12252, util=99.91%


Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme2n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=464MiB/s][w=119k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=17407: Wed Feb  3 19:59:54 2021
  write: IOPS=124k, BW=484MiB/s (508MB/s)(28.4GiB/60001msec); 0 zone resets
    slat (nsec): min=1496, max=102375, avg=2454.57, stdev=2090.91
    clat (usec): min=74, max=29904, avg=255.23, stdev=275.85
     lat (usec): min=77, max=29905, avg=257.78, stdev=275.79
    clat percentiles (usec):
     |  1.00th=[   84],  5.00th=[   94], 10.00th=[  103], 20.00th=[  110],
     | 30.00th=[  112], 40.00th=[  120], 50.00th=[  161], 60.00th=[  215],
     | 70.00th=[  273], 80.00th=[  363], 90.00th=[  523], 95.00th=[  685],
     | 99.00th=[ 1106], 99.50th=[ 1385], 99.90th=[ 3687], 99.95th=[ 3884],
     | 99.99th=[ 7701]
   bw (  KiB/s): min=108208, max=1132552, per=100.00%, avg=495995.58, stdev=180289.82, samples=119
   iops        : min=27052, max=283138, avg=123998.91, stdev=45072.45, samples=119
  lat (usec)   : 100=7.70%, 250=58.51%, 500=22.74%, 750=7.28%, 1000=2.27%
  lat (msec)   : 2=1.30%, 4=0.16%, 10=0.04%, 20=0.01%, 50=0.01%
  cpu          : usr=24.91%, sys=27.45%, ctx=225374, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7436753,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=484MiB/s (508MB/s), 484MiB/s-484MiB/s (508MB/s-508MB/s), io=28.4GiB (30.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=63/7424971, merge=0/0, ticks=56/1374981, in_queue=12972, util=99.93%


Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme3n1 --numjobs=1 --ioengine=libaio --iodepth=32 --runtime=60 --time_based
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=444MiB/s][w=114k IOPS][eta 00m:00s] 
test: (groupid=0, jobs=1): err= 0: pid=17080: Wed Feb  3 19:58:13 2021
  write: IOPS=119k, BW=463MiB/s (486MB/s)(27.2GiB/60001msec); 0 zone resets
    slat (nsec): min=1458, max=309216, avg=2402.88, stdev=2143.09
    clat (usec): min=73, max=15246, avg=266.81, stdev=319.96
     lat (usec): min=75, max=15248, avg=269.31, stdev=320.02
    clat percentiles (usec):
     |  1.00th=[   82],  5.00th=[   92], 10.00th=[  101], 20.00th=[  105],
     | 30.00th=[  108], 40.00th=[  119], 50.00th=[  163], 60.00th=[  217],
     | 70.00th=[  277], 80.00th=[  367], 90.00th=[  537], 95.00th=[  725],
     | 99.00th=[ 1450], 99.50th=[ 2008], 99.90th=[ 3720], 99.95th=[ 4015],
     | 99.99th=[ 7963]
   bw (  KiB/s): min=48328, max=1167728, per=99.99%, avg=474548.20, stdev=216585.23, samples=120
   iops        : min=12082, max=291932, avg=118637.02, stdev=54146.33, samples=120
  lat (usec)   : 100=9.46%, 250=56.34%, 500=22.50%, 750=7.17%, 1000=2.29%
  lat (msec)   : 2=1.73%, 4=0.45%, 10=0.05%, 20=0.01%
  cpu          : usr=23.03%, sys=26.28%, ctx=225073, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7118855,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s), io=27.2GiB (29.2GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=63/7106158, merge=0/0, ticks=52/1421294, in_queue=16204, util=99.91%
 
3x Jobs


Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme0n1 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based

test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32

...

fio-3.12

Starting 3 processes

Jobs: 3 (f=3): [W(3)][100.0%][w=502MiB/s][w=128k IOPS][eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=12993: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=471MiB/s (494MB/s)(27.6GiB/60001msec); 0 zone resets

    slat (nsec): min=1499, max=354182, avg=2228.50, stdev=1288.75

    clat (usec): min=78, max=20077, avg=262.57, stdev=291.75

     lat (usec): min=81, max=20078, avg=264.91, stdev=291.99

    clat percentiles (usec):

     |  1.00th=[  157],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  914],

     | 99.00th=[ 1483], 99.50th=[ 1663], 99.90th=[ 2180], 99.95th=[ 2606],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=128744, max=742632, per=33.27%, avg=482372.53, stdev=289635.59, samples=120

   iops        : min=32186, max=185658, avg=120593.10, stdev=72408.87, samples=120

  lat (usec)   : 100=0.03%, 250=86.67%, 500=2.12%, 750=3.64%, 1000=3.49%

  lat (msec)   : 2=3.92%, 4=0.13%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.41%, sys=27.48%, ctx=1954862, majf=0, minf=85

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7235652,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=12994: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=473MiB/s (495MB/s)(27.7GiB/60001msec); 0 zone resets

    slat (nsec): min=1539, max=113413, avg=2278.22, stdev=1180.38

    clat (usec): min=79, max=20170, avg=261.71, stdev=288.09

     lat (usec): min=81, max=20172, avg=264.10, stdev=288.24

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  906],

     | 99.00th=[ 1467], 99.50th=[ 1647], 99.90th=[ 2180], 99.95th=[ 2573],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=131720, max=742584, per=33.37%, avg=483854.45, stdev=288189.30, samples=120

   iops        : min=32930, max=185646, avg=120963.63, stdev=72047.34, samples=120

  lat (usec)   : 100=0.01%, 250=86.54%, 500=2.23%, 750=3.83%, 1000=3.48%

  lat (msec)   : 2=3.78%, 4=0.12%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.80%, sys=27.51%, ctx=1884742, majf=0, minf=20

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7257884,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=12995: Wed Feb  3 19:33:51 2021

  write: IOPS=121k, BW=472MiB/s (495MB/s)(27.7GiB/60001msec); 0 zone resets

    slat (nsec): min=1538, max=190223, avg=2278.79, stdev=1190.31

    clat (usec): min=78, max=20199, avg=261.78, stdev=288.28

     lat (usec): min=82, max=20201, avg=264.17, stdev=288.44

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  159], 10.00th=[  161], 20.00th=[  163],

     | 30.00th=[  165], 40.00th=[  167], 50.00th=[  169], 60.00th=[  172],

     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  594], 95.00th=[  906],

     | 99.00th=[ 1467], 99.50th=[ 1631], 99.90th=[ 2180], 99.95th=[ 2573],

     | 99.99th=[ 5407]

   bw (  KiB/s): min=131816, max=742784, per=33.36%, avg=483695.31, stdev=287995.45, samples=120

   iops        : min=32954, max=185696, avg=120923.83, stdev=71998.87, samples=120

  lat (usec)   : 100=0.01%, 250=86.54%, 500=2.23%, 750=3.83%, 1000=3.47%

  lat (msec)   : 2=3.77%, 4=0.12%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=23.09%, sys=28.25%, ctx=1884453, majf=0, minf=19

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,7255830,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):

  WRITE: bw=1416MiB/s (1485MB/s), 471MiB/s-473MiB/s (494MB/s-495MB/s), io=82.0GiB (89.1GB), run=60001-60001msec


Disk stats (read/write):

  nvme0n1: ios=116/21737538, merge=0/0, ticks=37/5579850, in_queue=17292, util=99.97%


Code: 
fio --rw=write --name=test --bs=4k --direct=1 --filename=/dev/nvme1n1 --numjobs=3 --ioengine=libaio --iodepth=32 --runtime=60 --time_based

test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32

...

fio-3.12

Starting 3 processes

Jobs: 3 (f=3): [W(3)][100.0%][w=460MiB/s][w=118k IOPS][eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=13726: Wed Feb  3 19:38:09 2021

  write: IOPS=39.7k, BW=155MiB/s (163MB/s)(9299MiB/60001msec); 0 zone resets

    slat (nsec): min=1493, max=120418, avg=2803.77, stdev=2340.81

    clat (usec): min=74, max=20896, avg=803.15, stdev=755.49

     lat (usec): min=78, max=20900, avg=806.06, stdev=755.65

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  165], 10.00th=[  169], 20.00th=[  227],

     | 30.00th=[  457], 40.00th=[  594], 50.00th=[  701], 60.00th=[  799],

     | 70.00th=[  930], 80.00th=[ 1106], 90.00th=[ 1401], 95.00th=[ 1762],

     | 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[10159], 99.95th=[11863],

     | 99.99th=[14353]

   bw (  KiB/s): min=11368, max=732480, per=33.53%, avg=158673.76, stdev=113697.48, samples=119

   iops        : min= 2842, max=183120, avg=39668.46, stdev=28424.36, samples=119

  lat (usec)   : 100=0.05%, 250=20.70%, 500=11.95%, 750=22.14%, 1000=19.67%

  lat (msec)   : 2=21.68%, 4=3.10%, 10=0.60%, 20=0.10%, 50=0.01%

  cpu          : usr=9.75%, sys=11.75%, ctx=553881, majf=0, minf=16

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2380478,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=13727: Wed Feb  3 19:38:09 2021

  write: IOPS=39.0k, BW=152MiB/s (160MB/s)(9141MiB/60001msec); 0 zone resets

    slat (nsec): min=1460, max=572622, avg=2945.77, stdev=2599.31

    clat (usec): min=74, max=20923, avg=816.87, stdev=761.17

     lat (usec): min=77, max=20925, avg=819.94, stdev=761.35

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  163], 10.00th=[  169], 20.00th=[  225],

     | 30.00th=[  465], 40.00th=[  611], 50.00th=[  717], 60.00th=[  824],

     | 70.00th=[  947], 80.00th=[ 1123], 90.00th=[ 1434], 95.00th=[ 1795],

     | 99.00th=[ 3720], 99.50th=[ 4424], 99.90th=[ 9765], 99.95th=[11731],

     | 99.99th=[14091]

   bw (  KiB/s): min=12352, max=733016, per=32.97%, avg=156003.84, stdev=113365.82, samples=120

   iops        : min= 3088, max=183254, avg=39000.96, stdev=28341.45, samples=120

  lat (usec)   : 100=0.11%, 250=20.74%, 500=11.07%, 750=21.25%, 1000=20.23%

  lat (msec)   : 2=22.66%, 4=3.18%, 10=0.65%, 20=0.10%, 50=0.01%

  cpu          : usr=9.72%, sys=12.65%, ctx=478163, majf=0, minf=74

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2340131,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32

test: (groupid=0, jobs=1): err= 0: pid=13728: Wed Feb  3 19:38:09 2021

  write: IOPS=39.6k, BW=155MiB/s (162MB/s)(9289MiB/60001msec); 0 zone resets

    slat (nsec): min=1513, max=98327, avg=2828.69, stdev=2342.69

    clat (usec): min=76, max=20900, avg=803.95, stdev=756.93

     lat (usec): min=77, max=20905, avg=806.88, stdev=757.09

    clat percentiles (usec):

     |  1.00th=[  155],  5.00th=[  165], 10.00th=[  169], 20.00th=[  229],

     | 30.00th=[  457], 40.00th=[  594], 50.00th=[  701], 60.00th=[  807],

     | 70.00th=[  930], 80.00th=[ 1106], 90.00th=[ 1418], 95.00th=[ 1762],

     | 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[10159], 99.95th=[11863],

     | 99.99th=[14484]

   bw (  KiB/s): min=11384, max=731904, per=33.50%, avg=158519.17, stdev=113624.24, samples=119

   iops        : min= 2846, max=182976, avg=39629.76, stdev=28405.98, samples=119

  lat (usec)   : 100=0.05%, 250=20.69%, 500=11.93%, 750=22.14%, 1000=19.67%

  lat (msec)   : 2=21.71%, 4=3.10%, 10=0.60%, 20=0.10%, 50=0.01%

  cpu          : usr=9.83%, sys=11.84%, ctx=546526, majf=0, minf=17

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2378028,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):

  WRITE: bw=462MiB/s (485MB/s), 152MiB/s-155MiB/s (160MB/s-163MB/s), io=27.1GiB (29.1GB), run=60001-60001msec


Disk stats (read/write):

  nvme1n1: ios=118/7083837, merge=0/0, ticks=64/5602516, in_queue=237168, util=99.92%



Had to delete the last 2 Benchmark, because of limited Forum size, bellow a screenshot of Windows 10, with Crystaldiskmark, to show that 3GB/s is possible



20210128_195913.jpg
 
spirit said:
if you use num-jobs, you should use --group_reporting. (or you'll have separated results for each job)

also, maybe try with more jobs, and lower queue depth. (it's possible that fio process is cpu bound, so more jobs=more fio process)
Click to expand...
with some tweaking i got around 2GB/s
 
Sounds not to bad for such a fio command with 4 KiB blocks!
I mean, different benchmark tools are differing in their approach a lot. For example, I took another look at your screenshot and saw that CrystalDiskMark uses 1 MiB and 128 KiB block sizes for the read tests (one with Q depth of 8 and one with Q depth of 32, if my guess from the test labels is correct - did not use that software since years) whereas the fio command always used 4 KiB, meaning much more IOPS to be handled by the disk.
 
1612541172977.png

Did some tweaking and managed to get that speed in a vm. So im not sure if that "low speed" is because of different benchmarks

If the drive has a different block size, does this "decrease" the overall performance or is it just a visual decrease?
 
Could you please elaborate on the tweaking you did?

I am getting abysmal results on 2 x Samsung SSD 980 PRO zfs mirrored using default zfs settings during proxmox installation. Used PVE < 6.4 for installation and currently fully upgraded to most recent version.
pve-manager/6.4-4/337d6701 (running kernel: 5.11.7-1-pve)

results when running:
fio --ioengine=psync --filename=/rpool/ROOT/pve-1/test/test --size=9G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --iodepth=1 --rw=write --bs=4k --numjobs=32
is about 10 x slower than found in
https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744/


Any tips on getting normal performance would be great
 
Last edited:
diversity said:
Could you please elaborate on the tweaking you did?

I am getting abysmal results on 2 x Samsung SSD 980 PRO zfs mirrored using default zfs settings during proxmox installation. Used PVE < 6.4 for installation and currently fully upgraded to most recent version.
pve-manager/6.4-4/337d6701 (running kernel: 5.11.7-1-pve)

results when running:
fio --ioengine=psync --filename=/rpool/ROOT/pve-1/test/test --size=9G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --iodepth=1 --rw=write --bs=4k --numjobs=32
is about 10 x slower than found in
https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744/


Any tips on getting normal performance would be great
Click to expand...
Interested to…. I installed proxmox yesterday but the vm speed of ubuntu is so slow.
What can i do to get the normal speed i had in esxi? More or less?
 
t.lamprecht said:
Determined by what measurement? And it would be interesting what storage/FS type did you choose for the VM.
Click to expand...
I found the problem! I dint see that, every time i created a new vm, the standard setting of the cd/dvd drive for the iso, was on IDE! Im so fucking dumb! I tought that the vm was slow, or my NVME! Storage type i left it on standard scsi!
 
Last edited:
You must log in or register to reply here.
Top Bottom