Troubles iSCSI and max_sectors_kb

Discussion in 'Proxmox VE: Installation and configuration' started by ManfredCC, Jan 24, 2016.

  1. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    Hiho,

    since the update to 4.1 (and previous 4.0) the setting for max_sectors_kb is 8192. In <=3.4 it was 512. When connecting to my SAN and using it i get continuous error messages.
    Setting this value to 512 (/sys/block/sdx/queue/max_sectors_kb) solves the problem and everything is ok. But this helps only a short time. When testing the iSCSI drive with e.g. dd ... the value is set back again to 8192, which results in errors again.
    I think this value is set in the kernel. Is there a way to set max_sectors_kb permanent to 512 or where can i set this value in proxmox?

    My config:
    - Server with Proxmox 4.1
    - Thecus n7770-10G
    -- iSCSI drive mounted as directory (because accessing to the virtual hosts and using qcow2 is the fastest way to work with. Tried also with NFS, but NFS works much slower)

    Thanks for help

    BR
    ManfredCC
     
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,461
    Likes Received:
    311
    What error messages exactly?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    Buffer I/O error on dev sdb1, logical block xxxxxxxx
    .
    .
    .
    blk_update_request: I/O error,dev sdb1, sector xxxxxxx
    .
    .
    .
    .
    xxxxxx = different blocks, sectors

    ...and the file is corrupted...

    as mentioned before - this happens only when max_sectors_kb is 8192 (working only with lower numbers - best with 512 ... without error and the transferred file is 100% ok ... tested with 50G file and diff)
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,461
    Likes Received:
    311
    Any infos from Thecus support?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,461
    Likes Received:
    311
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    No answer from Thecus up to now...

    Clean installation of debian 8.2 (64) uses max_sectors_kb = 512 as standard setting. Means iscsi is working without errors...

    I will try udev-rules from your posting (tried to define udev rule to set max_sectors_kb to 512, but with no success)...
     
  7. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    Doesn't change anything. The only thing which helps is to change max_sector_kb to 512.
     
  8. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    PS: running dd doesn't change max_sectors_kb.

    Funny enough...dd resets the max_sectors_kb again to 8192
     
    #8 ManfredCC, Jan 24, 2016
    Last edited: Jan 25, 2016
  9. adamb

    adamb Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2012
    Messages:
    999
    Likes Received:
    24
    Yea I think the only real option is to use udev rules. For me I just do it with rc.local and it sticks.

    multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN
    do
    echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
    done

    On proxmox 4 my iSCSI disks come up with some insain high max_sectors_kb (32768). I have found this number varies depending on the storage providing the iSCSI disks.
     
  10. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    I'll give it a try. I've installed ubuntu to check the max_sectors_kb and it shows 8192 too. Proxmox switched to ubuntu kernel and therefore it is clear :-(
     
  11. ManfredCC

    ManfredCC New Member
    Proxmox Subscriber

    Joined:
    Jan 24, 2016
    Messages:
    7
    Likes Received:
    0
    Seems to be a problem at Thecus' side. They can reproduce it (testing it with Ubuntu) in their labs and they are working on a solution.
     
  12. tomstephens89

    tomstephens89 Member

    Joined:
    Mar 10, 2014
    Messages:
    140
    Likes Received:
    1
    I am also suffering from this issue. However when I manually set max_sectors_kb to 512 on all my block devices, it reverts back to 32767 as soon as I try and make a VM on the disk.

    I cannot seem to get a udev rule to work?
     
  13. anigwei

    anigwei Member

    Joined:
    Nov 25, 2015
    Messages:
    33
    Likes Received:
    1
    Hi,

    Same problem here, instead of iSCSI i have a LSILogic SAS controller (SAS1064ET).

    Obviously disks are fine, RAID is OK... and latest Proxmox 4.2 with 4.4.40-1-pve.

    Any news from this?

    Code:
    [1038862.772719] blk_update_request: I/O error, dev sda, sector 14050080
    [1038862.778778] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [1038862.778781] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
    [1038862.778784] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
    [1038862.778786] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 40 00 00 28 00
    [1038862.778788] blk_update_request: I/O error, dev sda, sector 14050112
    [1038862.784948] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [1038862.784951] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
    [1038862.784953] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
    [1038862.784956] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 68 00 00 08 00
    [1038862.784957] blk_update_request: I/O error, dev sda, sector 14050152
    [1038862.790993] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [1038862.790998] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
    [1038862.791002] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
    [1038862.791006] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 70 00 00 10 00
    [1038862.791008] blk_update_request: I/O error, dev sda, sector 14050160
    [1038862.796996] sd 4:1:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [1038862.797002] sd 4:1:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
    [1038862.797006] sd 4:1:0:0: [sda] tag#1 Add. Sense: No additional sense information
    [1038862.797010] sd 4:1:0:0: [sda] tag#1 CDB: Write Same(10) 41 00 00 d6 63 80 00 00 08 00
    [1038862.797012] blk_update_request: I/O error, dev sda, sector 14050176
    Will try to force max_sectors_kb to 512.

    Raid is:
    Code:
    ioc0 vol_id 0 type IM, 2 phy, 557 GB, state OPTIMAL, flags ENABLED
    ioc0 phy 0 scsi_id 1 TOSHIBA  MBF2600RC        5208, 558 GB, state ONLINE, flags NONE
    ioc0 phy 1 scsi_id 2 TOSHIBA  MBF2600RC        5208, 558 GB, state ONLINE, flags NONE
    
    Thanks!
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice