High latency and catastrophic write performance with iSCSI + shared LVM

R

rboulay

New Member
Dec 18, 2025
1
0
1
Hello community :)

I am experiencing extremely high latency and abnormal read/write performance on my Proxmox nodes as well as on my VMs.
It takes several hours to install a VM running Debian 13 (Trixie), and the same issue occurs with Windows.

Environment​

  • Proxmox VE: 8.4.0 (no-subscription)
  • Cluster: 3-node cluster with 10 Gb SFP links
  • Storage: DELL ME4012 (7.2k SAS disks) connected via 10 GbE
  • iSCSI: open-iscsi 2.1.8, multipath 0.9.4.3

multipath.conf configuration​


defaults {
user_friendly_names yes #utilisation autorisee des alias wwid-nom de lun
find_multipaths yes
}

blacklist {
devnode "^sd[a]$"
}

devices {
device {
vendor "DELL"
product "*ME4*"
path_grouping_policy group_by_prio #priorite basse sur des groupes de chemin, 1 groupe sur port SFP 0 et 2, 1 groupe >
path_checker readsector0 #lit le premier secteur du device pour determiner l'etat des chemins
path_selector "round-robin 0" #distribution egale des io sur chaque groupe de port
failback immediate
no_path_retry fail
}
}

multipath -ll output​

multipath -ll
mpatha (3600c0ff000531302c994be6801000000) dm-11 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:1 sdc 8:32 active ready running
`- 15:0:0:1 sdb 8:16 active ready running
mpathb (3600c0ff000531302cf0dc46801000000) dm-10 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:3 sdd 8:48 active ready running
`- 16:0:0:3 sdf 8:80 active ready running
mpathc (3600c0ff000531302575bee6801000000) dm-9 DellEMC,ME4
size=559G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:4 sdh 8:112 active ready running
`- 15:0:0:4 sde 8:64 active ready running
mpathd (3600c0ff000531302cda91c6901000000) dm-12 DellEMC,ME4
size=93G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:5 sdg 8:96 active ready running
`- 16:0:0:5 sdi 8:128 active ready running



VM-side performance tests​


Read 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w0 -Sh -L -Z1G testfile.dat
Write 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w100 -Sh -L -Z1G testfile.dat

iSCSI + shared LVM​

  • Windows Server 2022 – 100% read:
    322 MB/s – 322 IOPS – 12.4 ms latency
  • Windows Server 2022 – 100% write:
    1.1 MB/s – 1.1 IOPS – 3561 ms latency

iSCSI + LVM thin​

  • Windows Server 2022 – 100% read:
    1011 MB/s – 1011 IOPS – 3.95 ms latency
  • Windows Server 2022 – 100% write:
    4.7 MB/s – 4.7 IOPS – 856 ms latency

Proxmox node-side performance tests​


Command: iostat -x 1 10
  • Partitions sdb and sdc with %util at 53.7% and 66.2%
  • await latency: ~230 ms
  • dm-7: await latency 299 ms, %util 116%


I have already tested several approaches:

  • identical MTU end-to-end
  • various multipath configurations
  • queue_depth tuning
Could you please help me identify the root cause of this issue?
 
rboulay said:
Hello community :)

I am experiencing extremely high latency and abnormal read/write performance on my Proxmox nodes as well as on my VMs.
It takes several hours to install a VM running Debian 13 (Trixie), and the same issue occurs with Windows.

Environment​

  • Proxmox VE: 8.4.0 (no-subscription)
  • Cluster: 3-node cluster with 10 Gb SFP links
  • Storage: DELL ME4012 (7.2k SAS disks) connected via 10 GbE
  • iSCSI: open-iscsi 2.1.8, multipath 0.9.4.3

multipath.conf configuration​


defaults {
user_friendly_names yes #utilisation autorisee des alias wwid-nom de lun
find_multipaths yes
}

blacklist {
devnode "^sd[a]$"
}

devices {
device {
vendor "DELL"
product "*ME4*"
path_grouping_policy group_by_prio #priorite basse sur des groupes de chemin, 1 groupe sur port SFP 0 et 2, 1 groupe >
path_checker readsector0 #lit le premier secteur du device pour determiner l'etat des chemins
path_selector "round-robin 0" #distribution egale des io sur chaque groupe de port
failback immediate
no_path_retry fail
}
}

multipath -ll output​

multipath -ll
mpatha (3600c0ff000531302c994be6801000000) dm-11 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:1 sdc 8:32 active ready running
`- 15:0:0:1 sdb 8:16 active ready running
mpathb (3600c0ff000531302cf0dc46801000000) dm-10 DellEMC,ME4
size=931G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:3 sdd 8:48 active ready running
`- 16:0:0:3 sdf 8:80 active ready running
mpathc (3600c0ff000531302575bee6801000000) dm-9 DellEMC,ME4
size=559G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 16:0:0:4 sdh 8:112 active ready running
`- 15:0:0:4 sde 8:64 active ready running
mpathd (3600c0ff000531302cda91c6901000000) dm-12 DellEMC,ME4
size=93G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 15:0:0:5 sdg 8:96 active ready running
`- 16:0:0:5 sdi 8:128 active ready running



VM-side performance tests​


Read 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w0 -Sh -L -Z1G testfile.dat
Write 100% command: .\diskspd.exe -b1M -d60 -o4 -t1 -w100 -Sh -L -Z1G testfile.dat

iSCSI + shared LVM​

  • Windows Server 2022 – 100% read:
    322 MB/s – 322 IOPS – 12.4 ms latency
  • Windows Server 2022 – 100% write:
    1.1 MB/s – 1.1 IOPS – 3561 ms latency

iSCSI + LVM thin​

  • Windows Server 2022 – 100% read:
    1011 MB/s – 1011 IOPS – 3.95 ms latency
  • Windows Server 2022 – 100% write:
    4.7 MB/s – 4.7 IOPS – 856 ms latency

Proxmox node-side performance tests​


Command: iostat -x 1 10
  • Partitions sdb and sdc with %util at 53.7% and 66.2%
  • await latency: ~230 ms
  • dm-7: await latency 299 ms, %util 116%


I have already tested several approaches:

  • identical MTU end-to-end
  • various multipath configurations
  • queue_depth tuning
Could you please help me identify the root cause of this issue?
Click to expand...
Looking at the documentation, the vendor, product, and several other things appear to be incorrect. I don't have a Dell device, but the correct configuration would be something like this:

defaults {
user_friendly_names yes
find_multipaths yes
}

blacklist {
devnode "^sda$"
}

devices {
device {
vendor "DellEMC"
product "ME4"
path_grouping_policy group_by_prio
path_checker tur
path_selector "service-time 0"
failback immediate
no_path_retry 5
prio alua
hardware_handler "1 alua"
rr_min_io_rq 1
fast_io_fail_tmo 15
dev_loss_tmo 45
}
}
 
  • Like
Reactions: waltar
rboulay said:
Hello,

Thank you for your feedback :)

I followed the Dell ME4 recommendations and modified the iscsid.conf and multipath.conf files as follows on my 3 nodes + reboot:
iscsid.conf before
node.session.timeo.replacement_timeout = 120
node.session.cmds_max = 128
node.session.queue_depth = 32

iscsid.conf after
node.session.timeo.replacement_timeout = 5
node.session.cmds_max = 1024
node.session.queue_depth = 128

multipath.conf before
defaults {
user_friendly_names yes
find_multipaths yes
features "1 queue_if_no_path"
path_grouping_policy failover
no_path_retry 100
}

multipath.conf after
defaults {
find_multipaths yes
user_friendly_names yes
}
blacklist {
}
devices {
device {
vendor "DellEMC"
product "ME4"
path_grouping_policy "group_by_prio"
path_checker "tur"
hardware_handler "1 alua"
prio "alua"
failback immediate
rr_weight "uniform"
path_selector "service-time 0"
}
}
multipaths {
}

Load test before
iSCSI + shared LVM
Windows Server 2022 – 100% read:
322 MB/s – 322 IOPS – 12.4 ms latency
Windows Server 2022 – 100% write:
1.1 MB/s – 1.1 IOPS – 3561 ms latency

Load test after
iSCSI + shared LVM
Windows Server 2022 – 100% read:
12248 MB/s – 734916 IOPS – 0.312 ms latency
Windows Server 2022 – 100% write:
2.1 MB/s – 128 IOPS – 1876ms latency

I see that the reading has improved significantly, but the writing remains problematic.
Click to expand...
In case it helps:

https://www.dell.com/support/kbdoc/en-us/000124601/me4-md-series-combined-linux-mpio-multipath-conf

https://infohub.delltechnologies.com/section-assets/powervault-me4-series-linux

Also check if the pools are set to Write-Through or Write-Back.
 
  • Like
Reactions: Kingneutron
You must log in or register to reply here.
Top Bottom