[SOLVED] Proxmox and NVMe/TCP

[JesperAP]

Hello,

We recently got a NetApp AFF-A250 and we want to test NVMe over TCP with proxmox.
We do have NVMe/TCP working on VMware and in a windows environment it gives us 33k IOPs with NVMe/TCP.

We got NVMe/TCP working following this tutorial:
https://linbit.com/blog/configuring-highly-available-nvme-of-attached-storage-in-proxmox-ve/

But when testing the IOPs we only get around 16k IOPs. The network speed is 20Gbps so that should not be the issue (tested with iperf3).

Are there settings we need to finetune in proxmox for this to work better? How can I troubleshoot this?

 

[JesperAP]

I did some more testing, i've installed the nvme storage on multiple nodes (one with SSD, one with HDD) but result stays the same (14/15k IOPS).

I am testing the IOPS with IOmeter and using the same test settings on VMware.

 

[bbgeek17]

Hello @JesperAP,

The level of performance is disappointing; even 30K IOPS is poor.

I advise debugging performance using `fio` on the bare metal host. Use the raw device paths to ensure your networking and storage work as expected. Double-check that you don't have any MTU issues.

For hypervisor-level tuning I invite you to look at https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html#tuning-procedure

If everything works as expected, this technote/series on optimizing Windows for Proxmox should be helpful: https://kb.blockbridge.com/technote/proxmox-optimizing-windows-server/part-1.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[JesperAP]

Hello @JesperAP,

The level of performance is disappointing; even 30K IOPS is poor.

I advise debugging performance using `fio` on the bare metal host. Use the raw device paths to ensure your networking and storage work as expected. Double-check that you don't have any MTU issues.

For hypervisor-level tuning I invite you to look at https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html#tuning-procedure

If everything works as expected, this technote/series on optimizing Windows for Proxmox should be helpful: https://kb.blockbridge.com/technote/proxmox-optimizing-windows-server/part-1.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hello @bbgeek17

I did a test using 'fio'

This is the result:


with this file:


I know the NetApp Storage VM is capped at 50k IOPS.

Should I be looking in Proxmox to use the right settings or should I finetune windows?

 

[bbgeek17]

Your fio benchmark uses num_jobs=16. This equates to 16 threads of execution, each submitting I/Os and processing completions. This is not how QEMU/Virtio/AIO works in the real world. So, its not an accurate baseline for performance of a single VM.

A few questions:
- What do you get with num_jobs=1 and iodepth=256 (i.e., an equivalent logical queue depth)?
- Could you provide the specific AIO settings for your VM, such as iouring, native, iothreads, or any other relevant settings? VM config would be helpful (Please use TEXT encoded with CODE tags).
- What virtual storage controller are you using?
- Are you using the virtio drivers in Windows?

I know the NetApp Storage VM is capped at 50k IOPS.

Is this a hardware or administrative limit?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Falk R.]

Is this a hardware or administrative limit?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Normally this is a software / pay for more limit.

 

[JesperAP]

Your fio benchmark uses num_jobs=16. This equates to 16 threads of execution, each submitting I/Os and processing completions. This is not how QEMU/Virtio/AIO works in the real world. So, its not an accurate baseline for performance of a single VM.

A few questions:
- What do you get with num_jobs=1 and iodepth=256 (i.e., an equivalent logical queue depth)?
- Could you provide the specific AIO settings for your VM, such as iouring, native, iothreads, or any other relevant settings? VM config would be helpful (Please use TEXT encoded with CODE tags).
- What virtual storage controller are you using?
- Are you using the virtio drivers in Windows?


Is this a hardware or administrative limit?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I did a test again with the settings changed you said, same results.

I solved the problem myself, I had the windows disks on sata instead of scsi... Now they get around 45k IOPS using IOmeter.


It is a administrative limit I can change it myself


Thanks for the help.

 

[bbgeek17]

I solved the problem myself, I had the windows disks on sata instead of scsi... Now they get around 45k IOPS using IOmeter.

It's great to hear that the controller was the issue. If you are looking for an optimal configuration, here's a detailed performance comparison of every storage controller, AIO mode, and threading model for Windows

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[JesperAP]

It's great to hear that the controller was the issue. If you are looking for an optimal configuration, here's a detailed performance comparison of every storage controller, AIO mode, and threading model for Windows

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

What do you use for cache?



this is my config now.

 

[bbgeek17]

Our storage does not need LVM, filesystems, or other layers on top, so the caching modes don't apply to our solution (i.e., we pass through raw NVMe devices). In general, we advise avoiding QEMU caching modes in a shared storage situation.
We also advise against it in all cases where your data is valuable.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Falk R.]

Like the previous speaker, I always recommend without a cache.
With a all-flash storage, please enable discard and SSD emulation.

 

[SInisterPisces]

Our storage does not need LVM, filesystems, or other layers on top, so the caching modes don't apply to our solution (i.e., we pass through raw NVMe devices). In general, we advise avoiding QEMU caching modes in a shared storage situation.
We also advise against it in all cases where your data is valuable.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I found this thread while I was trying to read up on NVME over TCP, as I'm planning on trying to use it for shared storage once TrueNAS 25.10 ships, and really appreciated this discussion of Windows-specific settings.

Per the recommendations here-- https://pve.proxmox.com/wiki/Windows_11_guest_best_practices -- QEMU Write Back cache is recommended for Windows 11 for better performance than no cache.

Is that wiki page just outdated, or do those best practices apply only to locally stored QEMU virtual disks (you specifically mentioned shared storage above)? I'm also assuming, perhaps incorrectly, that all my data is properly backed up via two Proxmox Backup Server instances in two different places. In either case, I think some clarifying language on the wiki page might be useful. I'll have to look in to how to do that.