NVMe Over TCP - Proxmox VE

[jsterr]

Hello Community,

im currently in the progress of trying out nvme over tcp in Proxmox VE and wanted to ask, if somebody has ressources for:

* Basic Setup of NVMe-Over-TCP in Proxmox VE?
* Information about using basic nvme-cli vs nvme-stas -> benefits?
* Measuring performance of nvme-over-tcp storages (howto know if performance is right, which tools to use for latency measuring?
* Any tweaks and tunings to have best practice regarding performance?

Performance wise nvme-over-tcp should be better than using iscsi or nfs on the same storage, is this assumption correct?

Thanks for your help!

Greetings Jonas

 

[Khensu]

Hey!

just sharing my two cents here of what I have tried so far.

nvme-cli vs nvme-stas:
We recently had this conversation with a customer.

First we configured it with nvme-cli natively, but this turned out to be rather complex.
It works, but it needs a few tweaks to behave like one would expect.
- Sometimes the network connection is not available quick enough on boot, which leads to not connected controllers when the nvmf-autoconnect service tries to connect the portals. => Dirty solution: Added a delay to the service, Pretty solution: Modify the sytemd order for the nvmf-autoconnect service.
- Normally, if a controller gets unavailable for more than 10 minutes, the controller gets dropped and permanently disconnect / gets removed. Even if the controller comes back, it wont reconnect. This needs to be tuned during connection with --ctrl-loss-tmo=-1. This way the controller never gets removed and stays in state reconnecting.
- We also tuned the --keep-alive-tmo to 10 to detect silently failed paths quicker. => Not sure about the default value, maybe it is already 10.
- iopolicy needs to be tuned with udev rules for Round-Robin or queue-depth. => https://docs.redhat.com/en/document...pathing_enabling-multipathing-on-nvme-devices

Good thing about nvme, it implements multipathing natively. Which is an advantage compared to iSCSI.
No multipath-tools required.

With nvme-stas the daemon manages the connections.
You still need to define the portals in the /etc/stas/stacd.conf instead of /etc/nvme/discovery.conf:

1. Log into the Proxmox node via SSH and install the package from the Linux repositories:
apt update
apt install nvme-stas -y
2. Configure “stacd.conf”
nano /etc/stas/stacd.conf

# Specify all reachable portal IPs for redundancy
[Controllers]
controller=transport=tcp;traddr=192.168.150.50;trsvcid=4420
controller=transport=tcp;traddr=192.168.150.51;trsvcid=4420
controller=transport=tcp;traddr=192.168.151.50;trsvcid=4420
controller=transport=tcp;traddr=192.168.151.51;trsvcid=4420

3. Enable and Start the Services
# Disable the old static service if it was enabled
systemctl disable --now nvmf-autoconnect.service

# Enable and start the new resilient daemons
systemctl enable --now stafd
systemctl enable --now stacd

You still need to modify the ctrl-loss-tmo and keep-alive value (also in stacd.conf).
See: https://nvme-stas.readthedocs.io/en/latest/stacd.conf.html
ctrl-loss-tmo -1, even tho nvme-stas would reconnect removed controllers on its own, is still needed!
Especially if you loose all paths. For example if the whole storage goes down for a while.
If the paths come back LVM could run into issues. Without the controllers being removed, everything should come back normally.
The VMs probably timed out by then, but the Storage is directly accessible again.

Also I have not found a way to easily change IO-Policy with nvme-stas, except with the udev rules used before.

In my opinion, the benefit of using nvme-stas is not big, if you do not use the auto-discovery features.

Benchmarking:
Normally we measure performance with fio and simulate some workloads.
Or for quick latency measurement with ioping (basically ping, but for storage).

What latency to expect depends on the storage behind NVMe/TCP.
For NVMe storages I would suspect lower latencies like 1-3ms, sudden spikes are okay under load but not on average.

Tweaks:
The only tweak we really applied is MTU 9000 (Jumbo Frames) on storage and client side.
Not much more. But I think this depends on the workload, what needs to be tuned.
Maybe expand memory buffers and queues, but please take this with a grain of salt. Never tried it and my experience in this field is limitied.
Just found this:
https://www.lightbitslabs.com/blog/...-i-o-and-i-matter-more-than-you-think-part-1/
https://www.lightbitslabs.com/blog/...-storage-the-simple-guide-to-nvme-tcp-tuning/


NVMe-TCP / iSCSI / NFS:
We prefer using block storage (iSCSI, NVMe/TCP/RDMA) instead of file (NFS,SMB etc.).

The snapshot feature is still in technology preview with LVM, but if the customer does not necessarily need it, we stay with disk type RAW and LVM without the snapshot feature. Which I recommend everyone in a production environment. Blockbridge summarized the reasons pretty well:
=> https://kb.blockbridge.com/technote/proxmox-qcow-snapshots-on-lvm/index.html
NFS and RAW is also fine (no snapshots), but I do not like using qcow2 (performance penalty, especially with snapshots and some other reasons).

As the NVMe/TCP is implemented natively without a separate translation layer like with iSCSI, it should give you lower latencies and less CPU-Overhead.
But depends on the workload.

No real tests behind it, but I would argue NVMe/TCP outperforms NFS.

Hope this helps and also very curious about other answers / experiences!

Best regards,
Kevin

 

[bbgeek17]

A few notes based on our experience with NVMe/TCP and PVE:

• Your performance expectation should come from your storage stack, not your choice of protocol. The transport is usually not the bottleneck in setups like this. If you're seeing latencies above a few hundred microseconds with iSCSI (high for all-flash over the network), the time is being spent somewhere else, and switching to NVMe/TCP will gain you very little. Find out where the latency lives first.
• We have a large number of both NVMe/TCP and iSCSI deployments in the field. In controlled experiments, the gap between the two is small at low queue depths, and we've even seen iSCSI edge ahead marginally in certain synthetic workloads. Where NVMe/TCP typically shines is in systems with multiple physical CPUs, since it natively gives you a queue per CPU without multiple sessions, and in truly high IOPS workloads (200K+). But again: your storage needs to be fast enough to actually see the difference.
• Both iSCSI and NVMe/TCP should be significantly faster than NFS. However, if you end up layering LVM/QCOW on top of iSCSI or NVMe, expect that to slow things down a bit.
• We published a detailed iSCSI vs NVMe/TCP comparison on PVE here: https://kb.blockbridge.com/technote/proxmox-iscsi-vs-nvmetcp/. Keep in mind these results were measured against fast backend storage. Whether the protocol difference shows up in your environment depends on the rest of your stack.
• nvme-cli vs nvme-stas: for a typical PVE cluster with static targets, stas is overkill. It exists for dynamic discovery (centralized discovery controllers, mDNS), which you almost certainly don't have. Use nvme-cli with a connect unit or /etc/nvme/discovery.conf + nvmf-autoconnect, and set ctrl_loss_tmo=-1 so the kernel retries indefinitely after a path loss. Reconnect handling is in the kernel. You don't need a userspace daemon for it.
• On multipathing: base the choice between native NVMe multipath (ANA) and dm-multipath on your kernel version. There were native multipath issues in earlier kernels. The kernel shipped with PVE 9.2 is fine for native. Check nvme list-subsys to confirm path states.
• Benchmarking: run fio against the raw block device before adding VM layers so you have a baseline. 4k randread/randwrite at QD1 for latency, high QD for IOPS, 128k sequential for throughput. Then repeat inside a VM to measure the virtualization cost. Watch CPU on both initiator and target.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox