One more GX10 post, the last one for now. After battling the black screen, the hard resets and the power cap, we were finally able to see what this box and PVE on arm64 can do.
Out of the box, the kernel initiator managed NVMe/TCP reads at 6.3 GB/s. Two kernel parameters later it was 28.1 GB/s, and SPDK got 28.9 GB/s, against a calculated 29.9 GB/s for the two PCIe links behind the NIC. This is stock PVE kernel on bare metal.
Most of the gain came from
iommu.strict=0, since strict is the default in the PVE arm64 kernel and it costs 4x performance penalty. pci=pcie_bus_perf added another 10%, because the firmware leaves MaxPayload at 128. The rest was working out that the ConnectX-7 is really two Gen5 x4 devices, so you need traffic on all four PFs. If you have a multi-PF NIC, also keep in mind that two PFs on the same port will answer each other's ARP unless you set arp_ignore=1.The full write-up, with the numbers, the link math and the exact commands can be found here: https://kb.blockbridge.com/technote/proxmox-gb10-nvme-tcp/
The target is Blockbridge storage, and this is part of how we qualify PVE on arm64. Any target that can handle the traffic will do to reproduce these results.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
