Troubles iSCSI and max_sectors_kb

[ManfredCC]

Hiho,

since the update to 4.1 (and previous 4.0) the setting for max_sectors_kb is 8192. In <=3.4 it was 512. When connecting to my SAN and using it i get continuous error messages.
Setting this value to 512 (/sys/block/sdx/queue/max_sectors_kb) solves the problem and everything is ok. But this helps only a short time. When testing the iSCSI drive with e.g. dd ... the value is set back again to 8192, which results in errors again.
I think this value is set in the kernel. Is there a way to set max_sectors_kb permanent to 512 or where can i set this value in proxmox?

My config:
- Server with Proxmox 4.1
- Thecus n7770-10G
-- iSCSI drive mounted as directory (because accessing to the virtual hosts and using qcow2 is the fastest way to work with. Tried also with NFS, but NFS works much slower)

Thanks for help

BR
ManfredCC

 

[dietmar]

When connecting to my SAN and using it i get continuous error messages.

What error messages exactly?

 

[ManfredCC]

Buffer I/O error on dev sdb1, logical block xxxxxxxx
.
.
.
blk_update_request: I/O error,dev sdb1, sector xxxxxxx
.
.
.
.
xxxxxx = different blocks, sectors

...and the file is corrupted...

as mentioned before - this happens only when max_sectors_kb is 8192 (working only with lower numbers - best with 512 ... without error and the transferred file is 100% ok ... tested with 50G file and diff)

 

[dietmar]

Any infos from Thecus support?

 

[dietmar]

Seems some people create udev rules to do it:

https://lists.centos.org/pipermail/centos/2008-November/068551.html

 

[ManfredCC]

No answer from Thecus up to now...

Clean installation of debian 8.2 (64) uses max_sectors_kb = 512 as standard setting. Means iscsi is working without errors...

I will try udev-rules from your posting (tried to define udev rule to set max_sectors_kb to 512, but with no success)...

 

[ManfredCC]

Doesn't change anything. The only thing which helps is to change max_sector_kb to 512.

 

[ManfredCC]

PS: running dd doesn't change max_sectors_kb.

Funny enough...dd resets the max_sectors_kb again to 8192

 

[adamb]

Yea I think the only real option is to use udev rules. For me I just do it with rc.local and it sticks.

multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN
do
echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
done

On proxmox 4 my iSCSI disks come up with some insain high max_sectors_kb (32768). I have found this number varies depending on the storage providing the iSCSI disks.

 

[ManfredCC]

I'll give it a try. I've installed ubuntu to check the max_sectors_kb and it shows 8192 too. Proxmox switched to ubuntu kernel and therefore it is clear :-(

 

[ManfredCC]

Seems to be a problem at Thecus' side. They can reproduce it (testing it with Ubuntu) in their labs and they are working on a solution.

 

[tomstephens89]

I am also suffering from this issue. However when I manually set max_sectors_kb to 512 on all my block devices, it reverts back to 32767 as soon as I try and make a VM on the disk.

I cannot seem to get a udev rule to work?

 

[anigwei]

Hi,

Same problem here, instead of iSCSI i have a LSILogic SAS controller (SAS1064ET).

Obviously disks are fine, RAID is OK... and latest Proxmox 4.2 with 4.4.40-1-pve.

Any news from this?

[1038862.772719] blk_update_request: I/O error, dev sda, sector 14050080
[1038862.778778] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[1038862.778781] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
[1038862.778784] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
[1038862.778786] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 40 00 00 28 00
[1038862.778788] blk_update_request: I/O error, dev sda, sector 14050112
[1038862.784948] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[1038862.784951] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
[1038862.784953] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
[1038862.784956] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 68 00 00 08 00
[1038862.784957] blk_update_request: I/O error, dev sda, sector 14050152
[1038862.790993] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[1038862.790998] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
[1038862.791002] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
[1038862.791006] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 70 00 00 10 00
[1038862.791008] blk_update_request: I/O error, dev sda, sector 14050160
[1038862.796996] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[1038862.797002] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
[1038862.797006] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
[1038862.797010] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 80 00 00 08 00
[1038862.797012] blk_update_request: I/O error, dev sda, sector 14050176

Will try to force max_sectors_kb to 512.

Raid is:

ioc0 vol_id 0 type IM, 2 phy, 557 GB, state OPTIMAL, flags ENABLED
ioc0 phy 0 scsi_id 1 TOSHIBA  MBF2600RC        5208, 558 GB, state ONLINE, flags NONE
ioc0 phy 1 scsi_id 2 TOSHIBA  MBF2600RC        5208, 558 GB, state ONLINE, flags NONE

Thanks!