Problems with SAN when writing large IOs - need to set max_sectors_kb

Jun 1, 2017
4
0
1
37
Hello Proxmox Community!

We've got a 6 node Proxmox cluster running the current version of proxmox 4.4. The disks of our VMs are located on one of our 2 multipathed iSCSI SANs. We got a new one this year and rebuilt our network with 10 GbE DAC Ethernet, a physically seperated 10 GbE LAN and Jumbo Frames activated - for iSCSI traffic only.

Before this we only had 10 GbE on one of our servers and the old SAN without active Jumbo Frames.

Since this change in May this year our older SAN (NEXSAN E18) is having serious problems. We're in contact with the technical support of NEXSAN and are currently in the process of pinning down the problem. One recommendation of the support is to change the "max_sectors_kb" to a lower value. They are suspecting that large writes to the SAN make it crash and setting "max_sectors_kb" to a lower value might fix the problem.

When we try to set the value with udev the system reverts the max_sectors_kb to what I believe is a calculated value. Echoing a number into /sys/block/.../max_sectors_kb also resets to the value after some time.

We tried to find the process that resets that value with auditd - without success.

I'd like to ask the community if anybody is able to set max_sectors_kb for /dev/dm-X and /dev/sdX devices that persist and how one can achieve that.

Furthermore I'd like to ask if there are any other recommendations to decrease the IO size written to a specific SAN.

On a side note: The new SAN is working without any problems whatsover and the "max_sectors_kb" is "automagically" set to the correct value recommended by the other vendor.

Kind regards,
igot2b
 
I've been down this road with our iscsi san. Never ran into the issue of the value's getting set back after I manually change them. What is the value for max_sectors_kb out of the box without making the change?
 
Hello adamb, thank you for your quick reply.

We've got 2 different types of disks that are represented independently.
The max_sectors_kb for the SATA array defaults to 640.
The max_sectors_kb for the SAS array defaults is 1152.
 
Hello adamb, thank you for your quick reply.

We've got 2 different types of disks that are represented independently.
The max_sectors_kb for the SATA array defaults to 640.
The max_sectors_kb for the SAS array defaults is 1152.

Oh wow that is quite small, I was expecting a very large value. For me they were defaulting to like 32767 or something. Not sure what else to do other than wait for the vendor to figure it out. From what I have seen this is typically a vendor issue.
 
Thank you again for the quick reply!

Can you - or anybody else - elaborate to me how the value is set? Is this a calculated value based on other values or is this maybe negotiated between the hosts and the SAN - and if it is calculated, based on what parameters? I've been using Linux systems for quite some time now but I can't wrap my head around this behaviour...

If setting the max_sectors_kb isn't an option, is there any recommended way to limit the maximum IO size that is sent to a specific multipathed iSCSI target?
 
Hello again,

since there was no response and the system is still not working as expected I'd like to ask again:

- Can anybody tell me how the "max_sectors_kb" value is set?
- Can anybody give me a recommendation for limiting the maximum IO size sent to a multipathed iSCSI target?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!