Proxmox vs ESXi Storage Performance - Tuning iSCSI?

[PwrBank]

Hello,

I'm trying to evaluate the performance differences on storage between ESXi and Proxmox. I'm having some trouble identifying where the performance issues are. Using the same hardware between tests I'm getting drastically different results when comparing the two hypervisors. What can I look into to increase performance on NFS or iSCSI? I'm kind of disappointed with having no ability to create snapshots on iSCSI, and with the current performance on NFS, I don't think it's a viable option.

Hardware:
Dedicated 25GbE cards are installed on the test machine and the SAN. When using iSCSI it is fully multipathed and verified on both ends to be using all paths. When using NFS the 25GbE connections are bonded with LACP using balance-rr.
All flash network appliance with dual controllers with two 25GbE each, all four NICs used for multipath

All tests were ran using the same fio scripts on the same VM that was transferred between Proxmox and ESXi using Veeam.



Here are the test results:
(The VMware NFS was inaccurate, as it was 10GbE at the time, but based on the IOPS, I suspect it is much faster than Proxmox)

Setup	Write IOPS	Read IOPS	Write Throughput (MB/s)	Read Throughput (MB/s)
Proxmox iSCSI LVM	60000	54000	2300	2000
Proxmox iSCSI Direct	84200	85600	2300	1000
Proxmox NFS	48400	17600	1870	181
VMware iSCSI	54700	107000	2800	5530
VMware NFS	46300	53400	1160	1160

Is there any tuning that can be done to increase performance on either NFS or iSCSI?

I've already changed the iSCSI tuning settings to reflect this post with no change in performance:
https://forum.proxmox.com/threads/s...th-dell-equallogic-storage.43018/#post-323461

Any help would be appreciated.

 

[bbgeek17]

Hi @PwrBank , welcome to the forum.

You may want to take a look at these knowledgebase articles we wrote, covering block storage and performance, for some tips:

Low Latency Storage Optimizations for Proxmox, KVM, & QEMU:
https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html

Optimizing Proxmox: iothreads, aio, & io_uring
https://kb.blockbridge.com/technote/proxmox-aio-vs-iouring/index.html

Proxmox vs. VMware ESXi: A Performance Comparison Using NVMe/TCP:
https://kb.blockbridge.com/technote/proxmox-vs-vmware-nvmetcp/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Blueloop]

I am in the very long process of migrating myself, my company and my customers away from VMware. I was a VMware afficianado for well over 20 years.

VMware nailed it when they created VMFS and their clustering stuff back when it was still Linux under the hood. MS created an almighty bodge with their clustering effort but got away with it. Proxmox and that don't bother with the shared monolithic block device thing as such, which is why you find yourself pissed off with iSCSI SANs not able to do snapshots.

I've reconciled myself (and quite a few customer budgets) with going "hyper converged". If the cluster hosts have enough slots, then populate them with SAS SSDs and do Ceph and dump the SAN or use it for backups or whatever. You must have at least three nodes and 10Gb+ networking for Ceph. With three nodes, two NICs are enough in each host without a switch and some careful networking.

Once you get away from the SAN thing (call it legacy if it makes you feel better) you really release yourself! A Proxmox Ceph cluster and Proxmox cluster with local flash storage compares rather well with a SAN. The VMs are always on local storage. Ceph ensures that they are replicated and the Proxmox clustering ensures availability.

Note that even if you don't do shared storage, you still have the equivalent of "Storage vMotion" available out of the box. Whilst I'm writing this post, I am continuing migrating a small business customer from VMware to Proxmox - a single ESXi host.

I delivered a migration box to site a couple of days ago with PVE on it. I fixed up the networking, iDRACs etc and mounted the VMFS on the PVE box. My colleagues migrated the VMs over last night and sorted out virtio drivers etc. Backups (Veeam) were fixed up and verified. I updated the BIOS etc, enabled the TPM, EFI and Secure Boot (yes ESXi can but it wasn't). I mounted the PVE install .iso from my laptop at home via the iDRAC and installed PVE on the box.

I created a temporary cluster on the migration box and joined the customer's new PVE to it. I am now live migrating the VMs over. When that's done, I will destroy the clustering (carefully and after cheching backups), fix up the backups and move on to the next customer.

 

[PwrBank]

Hi @PwrBank , welcome to the forum.

You may want to take a look at these knowledgebase articles we wrote, covering block storage and performance, for some tips:

Low Latency Storage Optimizations for Proxmox, KVM, & QEMU:
https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html

Optimizing Proxmox: iothreads, aio, & io_uring
https://kb.blockbridge.com/technote/proxmox-aio-vs-iouring/index.html

Proxmox vs. VMware ESXi: A Performance Comparison Using NVMe/TCP:
https://kb.blockbridge.com/technote/proxmox-vs-vmware-nvmetcp/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hey bbgeek17! I've read a LOT of your posts on this forum and am very interested in Blockbridge as a whole as well. I reached out to the sales team to discuss if Blockbridge could be combined with the SAN that's in this environment, and while they said it could, but it probably wouldn't be cost effective. Which I understand.

I'll take a look at these articles and report back my findings.

 

[PwrBank]

I am in the very long process of migrating myself, my company and my customers away from VMware. I was a VMware afficianado for well over 20 years.

VMware nailed it when they created VMFS and their clustering stuff back when it was still Linux under the hood. MS created an almighty bodge with their clustering effort but got away with it. Proxmox and that don't bother with the shared monolithic block device thing as such, which is why you find yourself pissed off with iSCSI SANs not able to do snapshots.

I utilize CEPH in a non-production cluster with pretty good results using 40GbE and 3 nodes with only 3 CBDs. However, a very large investment was put into these storage devices - So kind of stuck with it in a way. My only gripe with CEPH is the overhead of the system, it's not insanely resource intensive but you do need to take in account many more things than just a simple shared block device. One of these days I need to try out Linstor to see if there's much of difference in performance and up keep.

 

[Blueloop]

I utilize CEPH in a non-production cluster with pretty good results using 40GbE and 3 nodes with only 3 CBDs. However, a very large investment was put into these storage devices - So kind of stuck with it in a way. My only gripe with CEPH is the overhead of the system, it's not insanely resource intensive but you do need to take in account many more things than just a simple shared block device. One of these days I need to try out Linstor to see if there's much of difference in performance and up keep.

Do note that with the hyper converged model and something like Ceph, the virtual hard discs are nearly always local and not on the end of a network block device.

I think that Ceph is rather light on resources - it "only" has to ensure data integrity and that's a latency thing these days. Blocks need to be replicated and considered "correct" at the point of access.

You mention three CBDs (did you mean OBD) on three nodes and that sounds to me like three nodes with three RAID or other block devices. Ceph isn't designed to work like that. Ceph ideally gets to see each disc as a separate OBD. Your fancy RAID controller can still use its battery to store writes in the event of a power out and its cache for each disc individually. You need to set it to JBOD mode for the discs that do Ceph.

Your 40GbE network is a very nice touch.

I recently put in a three node cluster with mostly 10GbE and SAS SSDs (and a budget). Each node has six 1.5Tb SSDs. I devoted two NICs per box in a ring with Open vSwitch and STP for Ceph. With three nodes this works but for more, you'll need to use a switch. That's 18 OBDs.

It all works rather rapidly. I have several other systems to compare it with including VMware with a totally SAS flash SAN over iSCSI and others.

I am now sold on the Ceph approach for smaller clusters where compute nodes == storage nodes. That's most of my customers.

For larger systems, I'll get my slide rule out!