Very bad iSCSI performance despite no config change

[Oldsocks]

Hello!

I have a small cluster with my VMs living on LVM over iSCSI on a hp msa2050 SAN. I built it back in January and it was nice and quick - didn't even bother with multipath or any tuning because even with 10+ VMs running, it was fast enough.

I left it running, doing nothing, until this week. Now the performance is terrible! Transfer speed in guests can be measured in kb/s. Sometimes during a large single file transfer we'll see bursts of appropriate speed, but usually it is so slow that nothing is usable.

I have not made any changes to any configuration on the hosts, the network or the SAN. Now that 6.2 is out I did upgrade all to the new version hoping that would fix it - it hasn't. Does anybody have any idea what might be going on? I have rebooted everything, I have checked all devices and confirmed all the hardware is OK at each point.

 

[Stoiko Ivanov]

anything related to the iscsi storage or the exported luns in the journal?
(`journalctl -b` gives you the journal since last boot - scroll through it and look for errors or other information)

I hope this helps

 

[Oldsocks]

Thank you so much for pointing me in the right direction; it's been a rough few days - I'll jump on this like I should have done at the very start.