Slow speed in Proxmox VE installed on Samsung 980Pro NVME

sintei

Member
Jul 29, 2020
10
3
8
46
Hello,

I have a fresh installation of proxmox 6.4-13 on a Samsung 980 Pro NVME combined with a PCIe4 motherboard.
Speed should be blazing fast but when I do a test the results are really poor.
This is from proxmox ve shell:
Code:
root@node1:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/nvme0n1
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=373MiB/s][r=95.5k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=16261: Mon Oct  4 19:58:10 2021
  read: IOPS=94.5k, BW=369MiB/s (387MB/s)(21.6GiB/60001msec)
    slat (nsec): min=971, max=762817, avg=1259.83, stdev=565.09
    clat (nsec): min=190, max=4015.1k, avg=9050.88, stdev=4149.74
     lat (usec): min=8, max=4017, avg=10.35, stdev= 4.22
    clat percentiles (nsec):
     |  1.00th=[ 8512],  5.00th=[ 8512], 10.00th=[ 8640], 20.00th=[ 8640],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
     | 70.00th=[ 8896], 80.00th=[ 9024], 90.00th=[ 9408], 95.00th=[ 9920],
     | 99.00th=[12352], 99.50th=[17280], 99.90th=[32384], 99.95th=[68096],
     | 99.99th=[83456]
   bw (  KiB/s): min=346576, max=391576, per=99.99%, avg=378125.12, stdev=9733.96, samples=119
   iops        : min=86644, max=97894, avg=94531.26, stdev=2433.49, samples=119
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=95.59%, 20=4.19%, 50=0.14%
  lat (usec)   : 100=0.08%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=5.34%, sys=21.54%, ctx=5672413, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5672545,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=369MiB/s (387MB/s), 369MiB/s-369MiB/s (387MB/s-387MB/s), io=21.6GiB (23.2GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5662856/2138, merge=7/862, ticks=50024/1207, in_queue=636, util=99.88%
root@node1:~#
Speed around 3-400MiB/s?
Proxmox is also spiking every 7-8 seconds with increased CPU and IO delay. If no VMs or LXCs are running then CPU is around 0% but spikes up to 8-10% and IO delay goes from 0 to 8-15% every 7-8 seconds.

I'm using a AMD ryzen 5600X with a gigabyte aorus x570 pro motherboard.
Proxmox is installed on the NVME as basic as it gets. EXT4 for the HDD and then country/language/network settings.
BIOS is updated on both the NVME as well as the motherboard.

Anyone have tips on how to troubleshoot further?
Thanks
 
Last edited:
Another test with hdparam
Code:
root@node1:~# hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   50424 MB in  2.00 seconds = 25262.78 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 10922 MB in  3.00 seconds = 3639.97 MB/sec
root@node1:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   48908 MB in  2.00 seconds = 24499.67 MB/sec
 Timing buffered disk reads: 640 MB in  3.00 seconds = 213.13 MB/sec

Around 3600MB/s is still about 40% of the advertised speed. If it was 10% I would not mind.. but 40-50%?
Also, the SDA is RAIDZ1 and testing this one yealds normal results for spinning disks.

I checked the scheduler. It is set to NONE. Is the scheduler used for NVME harddrives?
Some say it should be NOOP and others say it doesn't matter as its deprecated and NVME use bl-mk (something like this, I cant remember).
Code:
cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline
 
Last edited:
Another test with hdparam
Code:
root@node1:~# hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   50424 MB in  2.00 seconds = 25262.78 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 10922 MB in  3.00 seconds = 3639.97 MB/sec
root@node1:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   48908 MB in  2.00 seconds = 24499.67 MB/sec
 Timing buffered disk reads: 640 MB in  3.00 seconds = 213.13 MB/sec

Around 3600MB/s is still about 40% of the advertised speed. If it was 10% I would not mind.. but 40-50%?
Also, the SDA is RAIDZ1 and testing this one yealds normal results for spinning disks.

I checked the scheduler. It is set to NONE. Is the scheduler used for NVME harddrives?
Some say it should be NOOP and others say it doesn't matter as its deprecated and NVME use bl-mk (something like this, I cant remember).
Code:
cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline

Did you have any success with this?
I'm having similair perfoamnce differences with two different NVMe's

cheers,
carlheinz
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!