High latency and catastrophic write performance with iSCSI + shared LVM

rboulay · Dec 18, 2025

Hello community

I am experiencing extremely high latency and abnormal read/write performance on my Proxmox nodes as well as on my VMs.
It takes several hours to install a VM running Debian 13 (Trixie), and the same issue occurs with Windows.

Environment

Proxmox VE: 8.4.0 (no-subscription)
Cluster: 3-node cluster with 10 Gb SFP links
Storage: DELL ME4012 (7.2k SAS disks) connected via 10 GbE
iSCSI: open-iscsi 2.1.8, multipath 0.9.4.3

multipath.conf configuration

defaults {
user_friendly_names yes #utilisation autorisee des alias wwid-nom de lun
find_multipaths yes
}

blacklist {
devnode "^sd[a]$"
}

devices {
device {
vendor "DELL"
product "*ME4*"
path_grouping_policy group_by_prio #priorite basse sur des groupes de chemin, 1 groupe sur port SFP 0 et 2, 1 groupe >
path_checker readsector0 #lit le premier secteur du device pour determiner l'etat des chemins
path_selector "round-robin 0" #distribution egale des io sur chaque groupe de port
failback immediate
no_path_retry fail
}
}

multipath -ll output

multipath -ll
mpatha (3600c0ff000531302c994be6801000000) dm-11 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:1 sdc 8:32 active ready running
`- 15:0:0:1 sdb 8:16 active ready running
mpathb (3600c0ff000531302cf0dc46801000000) dm-10 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:3 sdd 8:48 active ready running
`- 16:0:0:3 sdf 8:80 active ready running
mpathc (3600c0ff000531302575bee6801000000) dm-9 DellEMC,ME4
size=559G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:4 sdh 8:112 active ready running
`- 15:0:0:4 sde 8:64 active ready running
mpathd (3600c0ff000531302cda91c6901000000) dm-12 DellEMC,ME4
size=93G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:5 sdg 8:96 active ready running
`- 16:0:0:5 sdi 8:128 active ready running

VM-side performance tests

Read 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w0 -Sh -L -Z1G testfile.dat
Write 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w100 -Sh -L -Z1G testfile.dat

iSCSI + shared LVM

Windows Server 2022 – 100% read:
322 MB/s – 322 IOPS – 12.4 ms latency
Windows Server 2022 – 100% write:
1.1 MB/s – 1.1 IOPS – 3561 ms latency

iSCSI + LVM thin

Windows Server 2022 – 100% read:
1011 MB/s – 1011 IOPS – 3.95 ms latency
Windows Server 2022 – 100% write:
4.7 MB/s – 4.7 IOPS – 856 ms latency

Proxmox node-side performance tests

Command: iostat -x 1 10

Partitions sdb and sdc with %util at 53.7% and 66.2%
await latency: ~230 ms
dm-7: await latency 299 ms, %util 116%

I have already tested several approaches:

identical MTU end-to-end
various multipath configurations
queue_depth tuning

Could you please help me identify the root cause of this issue?

tlobo · Dec 18, 2025

rboulay said:
Hello community

I am experiencing extremely high latency and abnormal read/write performance on my Proxmox nodes as well as on my VMs.
It takes several hours to install a VM running Debian 13 (Trixie), and the same issue occurs with Windows.

Environment

Proxmox VE: 8.4.0 (no-subscription)

Cluster: 3-node cluster with 10 Gb SFP links

Storage: DELL ME4012 (7.2k SAS disks) connected via 10 GbE

iSCSI: open-iscsi 2.1.8, multipath 0.9.4.3

multipath.conf configuration

defaults {
user_friendly_names yes #utilisation autorisee des alias wwid-nom de lun
find_multipaths yes
}

blacklist {
devnode "^sd[a]$"
}

devices {
device {
vendor "DELL"
product "*ME4*"
path_grouping_policy group_by_prio #priorite basse sur des groupes de chemin, 1 groupe sur port SFP 0 et 2, 1 groupe >
path_checker readsector0 #lit le premier secteur du device pour determiner l'etat des chemins
path_selector "round-robin 0" #distribution egale des io sur chaque groupe de port
failback immediate
no_path_retry fail
}
}

multipath -ll output
multipath -ll
mpatha (3600c0ff000531302c994be6801000000) dm-11 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:1 sdc 8:32 active ready running
`- 15:0:0:1 sdb 8:16 active ready running
mpathb (3600c0ff000531302cf0dc46801000000) dm-10 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:3 sdd 8:48 active ready running
`- 16:0:0:3 sdf 8:80 active ready running
mpathc (3600c0ff000531302575bee6801000000) dm-9 DellEMC,ME4
size=559G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:4 sdh 8:112 active ready running
`- 15:0:0:4 sde 8:64 active ready running
mpathd (3600c0ff000531302cda91c6901000000) dm-12 DellEMC,ME4
size=93G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:5 sdg 8:96 active ready running
`- 16:0:0:5 sdi 8:128 active ready running

VM-side performance tests

Read 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w0 -Sh -L -Z1G testfile.dat
Write 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w100 -Sh -L -Z1G testfile.dat

iSCSI + shared LVM

Windows Server 2022 – 100% read:
322 MB/s – 322 IOPS – 12.4 ms latency

Windows Server 2022 – 100% write:
1.1 MB/s – 1.1 IOPS – 3561 ms latency

iSCSI + LVM thin

Windows Server 2022 – 100% read:
1011 MB/s – 1011 IOPS – 3.95 ms latency

Windows Server 2022 – 100% write:
4.7 MB/s – 4.7 IOPS – 856 ms latency

Proxmox node-side performance tests

Command: iostat -x 1 10

Partitions sdb and sdc with %util at 53.7% and 66.2%

await latency: ~230 ms

dm-7: await latency 299 ms, %util 116%

I have already tested several approaches:

identical MTU end-to-end

various multipath configurations

queue_depth tuning

Could you please help me identify the root cause of this issue?

Looking at the documentation, the vendor, product, and several other things appear to be incorrect. I don't have a Dell device, but the correct configuration would be something like this:

defaults {
user_friendly_names yes
find_multipaths yes
}

blacklist {
devnode "^sda$"
}

devices {
device {
vendor "DellEMC"
product "ME4"
path_grouping_policy group_by_prio
path_checker tur
path_selector "service-time 0"
failback immediate
no_path_retry 5
prio alua
hardware_handler "1 alua"
rr_min_io_rq 1
fast_io_fail_tmo 15
dev_loss_tmo 45
}
}

tlobo · Dec 19, 2025

rboulay said:
Hello,

Thank you for your feedback

I followed the Dell ME4 recommendations and modified the iscsid.conf and multipath.conf files as follows on my 3 nodes + reboot:
iscsid.conf before
node.session.timeo.replacement_timeout = 120
node.session.cmds_max = 128
node.session.queue_depth = 32

iscsid.conf after
node.session.timeo.replacement_timeout = 5
node.session.cmds_max = 1024
node.session.queue_depth = 128

multipath.conf before
defaults {
user_friendly_names yes
find_multipaths yes
features "1 queue_if_no_path"
path_grouping_policy failover
no_path_retry 100
}

multipath.conf after
defaults {
find_multipaths yes
user_friendly_names yes
}
blacklist {
}
devices {
device {
vendor "DellEMC"
product "ME4"
path_grouping_policy "group_by_prio"
path_checker "tur"
hardware_handler "1 alua"
prio "alua"
failback immediate
rr_weight "uniform"
path_selector "service-time 0"
}
}
multipaths {
}

Load test before
iSCSI + shared LVM
Windows Server 2022 – 100% read:
322 MB/s – 322 IOPS – 12.4 ms latency
Windows Server 2022 – 100% write:
1.1 MB/s – 1.1 IOPS – 3561 ms latency

Load test after
iSCSI + shared LVM
Windows Server 2022 – 100% read:
12248 MB/s – 734916 IOPS – 0.312 ms latency
Windows Server 2022 – 100% write:
2.1 MB/s – 128 IOPS – 1876ms latency

I see that the reading has improved significantly, but the writing remains problematic.

In case it helps:

https://www.dell.com/support/kbdoc/en-us/000124601/me4-md-series-combined-linux-mpio-multipath-conf

https://infohub.delltechnologies.com/section-assets/powervault-me4-series-linux

Also check if the pools are set to Write-Through or Write-Back.

softworx · Jan 6, 2026

Also, check to make sure TCP Delayed Ack is disabled.

Delayed acks will sometimes hold tcp acks for other data. This is normally good, and the impact is not that dramatic, for regular network traffic. For iscsi it can do bad things for your latency. I've seen latency go from milliseconds to microseconds by making this change. This was on an IBM FlashSystem where expected latency was basically sub-millisecond for our application. We weren't able to achieve that until all iscsi hbas and the iscsi nas were set to disable tcp delay ack.

bbgeek17 · Jan 6, 2026

softworx said:
I've seen latency go from 20 to 60 ms to 20 to 60 ns by making this change.

Hey @softworx , did you mean to use µ (us) ? 20 ns is less than DRAM

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Search

Search

High latency and catastrophic write performance with iSCSI + shared LVM

rboulay

New Member

Environment

multipath.conf configuration

multipath -ll output

VM-side performance tests

iSCSI + shared LVM

iSCSI + LVM thin

Proxmox node-side performance tests

tlobo

Member

Environment

multipath.conf configuration

multipath -ll output

VM-side performance tests

iSCSI + shared LVM

iSCSI + LVM thin

Proxmox node-side performance tests

tlobo

Member

softworx

Member

bbgeek17

Distinguished Member

We value your privacy

High latency and catastrophic write performance with iSCSI + shared LVM

rboulay

New Member

Environment​

multipath.conf configuration​

multipath -ll output​

VM-side performance tests​

iSCSI + shared LVM​

iSCSI + LVM thin​

Proxmox node-side performance tests​

tlobo

Member

Environment​

multipath.conf configuration​

multipath -ll output​

VM-side performance tests​

iSCSI + shared LVM​

iSCSI + LVM thin​

Proxmox node-side performance tests​

tlobo

Member

softworx

Member

bbgeek17

Distinguished Member

We value your privacy

Environment

multipath.conf configuration

multipath -ll output

VM-side performance tests

iSCSI + shared LVM

iSCSI + LVM thin

Proxmox node-side performance tests

Environment

multipath.conf configuration

multipath -ll output

VM-side performance tests

iSCSI + shared LVM

iSCSI + LVM thin

Proxmox node-side performance tests