Hi everyone,
I am experiencing severe latency spikes and sub-optimal bandwidth when running sequential read workloads over iSCSI from a Proxmox VE node to a Dell PowerVault ME5024 array.
During high I/O tests on Proxmox, the average throughput hovers around 265–300 MiB/s, but the response time spikes dramatically up to 6 seconds (99.9th percentile), causing IOPS to briefly collapse down to single digits before recovering.
For comparison, a VMware ESXi host connected to the exact same ME5024 array and storage network handles the same workload effortlessly, reaching over 1.2 GB/s with single-digit millisecond latency.
devices {
device {
vendor "DellEMC"
product "ME5"
path_grouping_policy "group_by_prio"
path_checker tur
path_selector "service-time 0"
prio "alua"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 20
hardware_handler "1 alua"
fast_io_fail_tmo 15
dev_loss_tmo 60
}
}
Benchmark Comparison (1M Sequential Read fio):
READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=9159MiB (9604MB), run=34500-34500msec
clat (usec): min=1919, max=6043.6k, avg=240711.45, stdev=824047.11
clat percentiles (msec):
| 50.00th=[ 27], 70.00th=[ 31], 80.00th=[ 94], 90.00th=[ 393],
| 95.00th=[ 911], 99.00th=[ 5269], 99.90th=[ 6007]
bw (KiB/s) : min= 8192, max=1601536, avg=396810.34
VMware ESXi VM Output:
READ: bw=1220MiB/s (1279MB/s), 1220MiB/s-1220MiB/s (1279MB/s-1279MB/s), io=35.8GiB (38.4GB), run=30001-30001msec
clat (usec): min=512, max=18420, avg=5231.12, stdev=1102.40
clat percentiles (msec):
| 50.00th=[ 5], 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7],
| 95.00th=[ 8], 99.00th=[ 11], 99.90th=[ 15]
bw (KiB/s) : min=1150000, max=1285000, avg=1249280.00
While Proxmox reaches transient peaks of ~1.5 GB/s (max=1601536 KiB/s), it cannot sustain high bandwidth without triggering massive latency penalties (6 seconds peak delay). VMware on the same fabric runs cleanly at line rate (~1.2 GB/s) with a flat 5ms average latency.
Has anyone fine-tuned DM-Multipath, iSCSI queue depths, or block layer settings for Dell PowerVault ME5 series on Proxmox to achieve near wire-speed throughput matching VMware without hitting these latency bottlenecks?
Any insights or recommended configurations would be greatly appreciated!
I am experiencing severe latency spikes and sub-optimal bandwidth when running sequential read workloads over iSCSI from a Proxmox VE node to a Dell PowerVault ME5024 array.
During high I/O tests on Proxmox, the average throughput hovers around 265–300 MiB/s, but the response time spikes dramatically up to 6 seconds (99.9th percentile), causing IOPS to briefly collapse down to single digits before recovering.
For comparison, a VMware ESXi host connected to the exact same ME5024 array and storage network handles the same workload effortlessly, reaching over 1.2 GB/s with single-digit millisecond latency.
Environment Setup:
- Host: Proxmox VE 9.2
- Storage: Dell PowerVault ME5024 (iSCSI, 8 active paths across 2 subnets/VLANs)
- NICs: 2x Dedicated 10GbE interfaces for iSCSI traffic
Current Proxmox Stock Configuration:
1. multipath.conf (Dell ME5 Vendor Defaults):devices {
device {
vendor "DellEMC"
product "ME5"
path_grouping_policy "group_by_prio"
path_checker tur
path_selector "service-time 0"
prio "alua"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 20
hardware_handler "1 alua"
fast_io_fail_tmo 15
dev_loss_tmo 60
}
}
Benchmark Comparison (1M Sequential Read fio):
Proxmox VE 9 Node Output:
READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=9159MiB (9604MB), run=34500-34500msec
clat (usec): min=1919, max=6043.6k, avg=240711.45, stdev=824047.11
clat percentiles (msec):
| 50.00th=[ 27], 70.00th=[ 31], 80.00th=[ 94], 90.00th=[ 393],
| 95.00th=[ 911], 99.00th=[ 5269], 99.90th=[ 6007]
bw (KiB/s) : min= 8192, max=1601536, avg=396810.34
VMware ESXi VM Output:
READ: bw=1220MiB/s (1279MB/s), 1220MiB/s-1220MiB/s (1279MB/s-1279MB/s), io=35.8GiB (38.4GB), run=30001-30001msec
clat (usec): min=512, max=18420, avg=5231.12, stdev=1102.40
clat percentiles (msec):
| 50.00th=[ 5], 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7],
| 95.00th=[ 8], 99.00th=[ 11], 99.90th=[ 15]
bw (KiB/s) : min=1150000, max=1285000, avg=1249280.00
While Proxmox reaches transient peaks of ~1.5 GB/s (max=1601536 KiB/s), it cannot sustain high bandwidth without triggering massive latency penalties (6 seconds peak delay). VMware on the same fabric runs cleanly at line rate (~1.2 GB/s) with a flat 5ms average latency.
Has anyone fine-tuned DM-Multipath, iSCSI queue depths, or block layer settings for Dell PowerVault ME5 series on Proxmox to achieve near wire-speed throughput matching VMware without hitting these latency bottlenecks?
Any insights or recommended configurations would be greatly appreciated!
Last edited:
