I am submitting this to the forum as I haven't purchased the subscription version yet due to the performance issues noted below.
I have 2 of the exact same servers (Lenovo SR650V3) with 10 3.2TB enterprise SSD (Model: MZILG3T2HCLS-V7) in a single ZFS Raid 10 pool. I spun up two separate Windows VMs (one on each server) with the same specs 8vCPU, 16GB memory, 500GB disk, etc... for which I noticed the Windows installation going a little slow compared to other Proxmox servers I have spun up. I ran iometer, diskspd and other disk performance utilities for which I only get ~10-15k IOPS on both VMs. I ran FIO on the one proxmox server and I'm getting the proper IOPS as expected...info below. These Lenovo servers have Dual HBA controllers and dual CPU, compared to another proxmox server I have that is ok with a single CPU and a single HBA controller. This Proxmox server (actual HA pair) I get IOPS in the Windows VM ~250k. The only difference between these servers and the slower performing ones are having dual processors and dual HBA ***these are true HBA Lenovo 440i no cache***
For the Windows VM disk layout of the "slower" hosts, it's the same as my other "normal" performing proxmox server. VirtIO SCSI, cache=writeback.
As another test, I reimaged one of the "slower - SR650V3" servers with ESXi 8.0u3 and spun up a Server 2025 VM. I ran the same tests as I did on the other Windows VM and got ~30-35k IOPS on a single disk. I then went back to the proxmox server and deleted the zfs pool, and created a single disk datastore for which the IOPS was now ~1-2k.
I'm pretty much at a loss of what I should do next. If this continues to go on I will need to just reimage the servers with VMware and call it a day...but hoping I do not need to do that.
FIO on proxmox server:
root@testpve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/test/testfile --size=4G
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1707MiB/s][r=437k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=366415: Wed May 28 15:27:51 2025
read: IOPS=407k, BW=1590MiB/s (1667MB/s)(93.2GiB/60001msec)
slat (nsec): min=1238, max=159463, avg=2088.36, stdev=4004.79
clat (nsec): min=194, max=46854, avg=227.53, stdev=70.74
lat (nsec): min=1449, max=173415, avg=2315.90, stdev=4012.70
I have 2 of the exact same servers (Lenovo SR650V3) with 10 3.2TB enterprise SSD (Model: MZILG3T2HCLS-V7) in a single ZFS Raid 10 pool. I spun up two separate Windows VMs (one on each server) with the same specs 8vCPU, 16GB memory, 500GB disk, etc... for which I noticed the Windows installation going a little slow compared to other Proxmox servers I have spun up. I ran iometer, diskspd and other disk performance utilities for which I only get ~10-15k IOPS on both VMs. I ran FIO on the one proxmox server and I'm getting the proper IOPS as expected...info below. These Lenovo servers have Dual HBA controllers and dual CPU, compared to another proxmox server I have that is ok with a single CPU and a single HBA controller. This Proxmox server (actual HA pair) I get IOPS in the Windows VM ~250k. The only difference between these servers and the slower performing ones are having dual processors and dual HBA ***these are true HBA Lenovo 440i no cache***
For the Windows VM disk layout of the "slower" hosts, it's the same as my other "normal" performing proxmox server. VirtIO SCSI, cache=writeback.
As another test, I reimaged one of the "slower - SR650V3" servers with ESXi 8.0u3 and spun up a Server 2025 VM. I ran the same tests as I did on the other Windows VM and got ~30-35k IOPS on a single disk. I then went back to the proxmox server and deleted the zfs pool, and created a single disk datastore for which the IOPS was now ~1-2k.
I'm pretty much at a loss of what I should do next. If this continues to go on I will need to just reimage the servers with VMware and call it a day...but hoping I do not need to do that.
FIO on proxmox server:
root@testpve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/test/testfile --size=4G
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1707MiB/s][r=437k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=366415: Wed May 28 15:27:51 2025
read: IOPS=407k, BW=1590MiB/s (1667MB/s)(93.2GiB/60001msec)
slat (nsec): min=1238, max=159463, avg=2088.36, stdev=4004.79
clat (nsec): min=194, max=46854, avg=227.53, stdev=70.74
lat (nsec): min=1449, max=173415, avg=2315.90, stdev=4012.70
