Need Help Optimizing NVMe SSD Performance on Proxmox

[honeyfairy]

I have a setup that I believe should be capable of much higher read speeds than what I am currently achieving, and I'm hoping the community can help me troubleshoot and get the most out of my hardware.

Here are the details of my setup:

• Hardware:
  • HP ML350 Gen10
  • CPU: Xeon 3106 1.7GHz (single CPU)
  • Memory: 16GB HP SmartMemory DDR4
  • Storage Controllers: HPE Smart Array P408i-a SR Gen10 Controller
  • Storage Devices:
    • MO000800JWTBR HP G8-G10 800-GB 2.5 SAS 12G MU SSD
    • Western Digital Ultrastar DC SN200 PCIe 3.0 x8 NVMe SSD 1600GB HUSMR7616BHP301 LP

I've run some performance tests using hdparm -t and found that both my SAS SSD and NVMe SSD are performing at similar speeds, which are below the advertised speed of the NVMe SSD. Here are the results:

SAS SSD:

hdparm -t /dev/vda
Timing buffered disk reads: 3438 MB in 3.00 seconds = 1144.42 MB/sec

NVMe SSD:

hdparm -t /dev/sdb
Timing buffered disk reads: 3630 MB in 3.00 seconds = 1209.68 MB/sec

However direct reads seem to be better:

 hdparm --DIRECT -Tt /dev/vda
/dev/vda:
 Timing O_DIRECT cached reads:   8574 MB in  1.99 seconds = 4311.27 MB/sec
 Timing O_DIRECT disk reads: 13168 MB in  3.00 seconds = 4388.97 MB/sec

I believe the NVMe SSD should be capable of much higher speeds, closer to its advertised 6,000MB/s. I've checked PCIe lane configurations, BIOS settings, and made sure I have the latest firmware and drivers installed. However, I'm still not achieving the expected performance.

I'm reaching out to the Proxmox community for any insights, suggestions, or guidance on how to maximize the read speed of my NVMe SSD. If anyone has experience with a similar setup or has encountered similar issues, I'd greatly appreciate your input.

Additionally, if there are specific Proxmox configurations or optimizations that I should be aware of to fully leverage the speed of NVMe storage, please share your expertise.
I've tried adding the drives in a ZFS, added them in the VM with writethrough cache enabled. All similar results.

Thank you in advance for your assistance, and I look forward to your valuable insights and recommendations.

 

[nullswitch]

I have a setup that I believe should be capable of much higher read speeds than what I am currently achieving, and I'm hoping the community can help me troubleshoot and get the most out of my hardware.

Here are the details of my setup:

• Hardware:
  • HP ML350 Gen10
  • CPU: Xeon 3106 1.7GHz (single CPU)
  • Memory: 16GB HP SmartMemory DDR4
  • Storage Controllers: HPE Smart Array P408i-a SR Gen10 Controller
  • Storage Devices:
    • MO000800JWTBR HP G8-G10 800-GB 2.5 SAS 12G MU SSD
    • Western Digital Ultrastar DC SN200 PCIe 3.0 x8 NVMe SSD 1600GB HUSMR7616BHP301 LP

I've run some performance tests using hdparm -t and found that both my SAS SSD and NVMe SSD are performing at similar speeds, which are below the advertised speed of the NVMe SSD. Here are the results:

SAS SSD:

hdparm -t /dev/vda
Timing buffered disk reads: 3438 MB in 3.00 seconds = 1144.42 MB/sec

NVMe SSD:

hdparm -t /dev/sdb
Timing buffered disk reads: 3630 MB in 3.00 seconds = 1209.68 MB/sec

However direct reads seem to be better:

 hdparm --DIRECT -Tt /dev/vda
/dev/vda:
 Timing O_DIRECT cached reads:   8574 MB in  1.99 seconds = 4311.27 MB/sec
 Timing O_DIRECT disk reads: 13168 MB in  3.00 seconds = 4388.97 MB/sec

I believe the NVMe SSD should be capable of much higher speeds, closer to its advertised 6,000MB/s. I've checked PCIe lane configurations, BIOS settings, and made sure I have the latest firmware and drivers installed. However, I'm still not achieving the expected performance.

I'm reaching out to the Proxmox community for any insights, suggestions, or guidance on how to maximize the read speed of my NVMe SSD. If anyone has experience with a similar setup or has encountered similar issues, I'd greatly appreciate your input.

Additionally, if there are specific Proxmox configurations or optimizations that I should be aware of to fully leverage the speed of NVMe storage, please share your expertise.
I've tried adding the drives in a ZFS, added them in the VM with writethrough cache enabled. All similar results.

Thank you in advance for your assistance, and I look forward to your valuable insights and recommendations.

Following up on this, did you ever improve performance?

 

[tigra4]

You have found the solution?
I have the same problems with NVME speed on HP DL560 Gen9

sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2388 MB in  3.00 seconds = 795.93 MB/sec
adm1n@pve1:~$ sudo hdparm --DIRECT -Tt /dev/nvme1n1

/dev/nvme1n1:
 Timing O_DIRECT cached reads:   3864 MB in  2.00 seconds = 1936.69 MB/sec
 Timing O_DIRECT disk reads: 5660 MB in  3.00 seconds = 1886.44 MB/sec

 

[nullswitch]

You have found the solution?
I have the same problems with NVME speed on HP DL560 Gen9

sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2388 MB in  3.00 seconds = 795.93 MB/sec
adm1n@pve1:~$ sudo hdparm --DIRECT -Tt /dev/nvme1n1

/dev/nvme1n1:
 Timing O_DIRECT cached reads:   3864 MB in  2.00 seconds = 1936.69 MB/sec
 Timing O_DIRECT disk reads: 5660 MB in  3.00 seconds = 1886.44 MB/sec

I wrote out

You have found the solution?
I have the same problems with NVME speed on HP DL560 Gen9

sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2388 MB in  3.00 seconds = 795.93 MB/sec
adm1n@pve1:~$ sudo hdparm --DIRECT -Tt /dev/nvme1n1

/dev/nvme1n1:
 Timing O_DIRECT cached reads:   3864 MB in  2.00 seconds = 1936.69 MB/sec
 Timing O_DIRECT disk reads: 5660 MB in  3.00 seconds = 1886.44 MB/sec

I wrote out my solution on Reddit. Tldr, I had sata controller to the nvme instead of virtio. Now I get 3500 MB/s which is what I would expect for my setup since the PCIE lane is the bottleneck.

https://www.reddit.com/r/homelab/s/S3mEIU7k82