Iscsi devices errors : read failed

T

t-m

New Member
Mar 27, 2012
24
0
1
France
Dear members,

After some weeks to configure 2 proxmox servers + 1 iscsi DELL MD3220i storage array I'm having problems with a testing KVM machine : impossible to start afet a reboot.
Proxmox 2.1 is up-to-date.

So I found some errors :
Code: 
root@proxmox1:/# pvdisplay
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 536870846464: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 536870903808: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1610612670464: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1610612727808: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 53687025664: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 53687083008: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 4096: Input/output error
  --- Physical volume ---
  PV Name               /dev/disk/by-id/dm-name-mpath1
  VG Name               kvm1-vm
  PV Size               500.00 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              127999
  Free PE               109055
  Allocated PE          18944
  PV UUID               pzcSlx-VelX-Qr0V-NbC1-3dh9-lwhw-PLLCBc
   
  --- Physical volume ---
  PV Name               /dev/disk/by-id/dm-name-mpath2
  VG Name               kvm2-storage
  PV Size               1.46 TiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              383999
  Free PE               332799
  Allocated PE          51200
  PV UUID               RZUURl-Hiln-L4dD-iaSJ-KjgH-CFFm-SXNc7G
   
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               231.88 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              59359
  Free PE               4095
  Allocated PE          55264
  PV UUID               dXAvDd-kEAN-3Uel-l0kc-PLqf-l7DI-7wZB0n

sda is the local disk.
sdb, sdc and sdd was iSCSI devices used for KVM storages.

Code: 
root@proxmox1:/# lvdisplay 
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 536870846464: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 536870903808: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1610612670464: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 1610612727808: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 53687025664: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 53687083008: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 4096: Input/output error
  --- Logical volume ---
  LV Path                /dev/kvm1-vm/vm-105-disk-1
  LV Name                vm-105-disk-1
  VG Name                kvm1-vm
  LV UUID                8yRLfQ-dyIY-o1Zk-awPz-gA74-Mz6E-LUdXvE
  LV Write Access        read/write
  LV Creation host, time proxmox1, 2012-09-11 18:58:19 +0200
  LV Status              available
  # open                 1
  LV Size                32.00 GiB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:7
   
  --- Logical volume ---
  LV Path                /dev/kvm1-vm/vm-106-disk-1
  LV Name                vm-106-disk-1
  VG Name                kvm1-vm
  LV UUID                O85OR1-QLYP-fw4y-0rZI-E6mk-IsKX-5bN1tz
  LV Write Access        read/write
  LV Creation host, time proxmox1, 2012-09-14 17:10:12 +0200
  LV Status              available
  # open                 1
  LV Size                32.00 GiB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:8
   
  --- Logical volume ---
  LV Path                /dev/kvm1-vm/vm-120-disk-1
  LV Name                vm-120-disk-1
  VG Name                kvm1-vm
  LV UUID                wmqc5y-VlFD-eBKv-CZIV-tNfU-15cx-zWXP6O
  LV Write Access        read/write
  LV Creation host, time proxmox1, 2012-09-21 17:25:34 +0200
  LV Status              NOT available
  LV Size                10.00 GiB
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/kvm2-storage/vm-105-disk-1
  LV Name                vm-105-disk-1
  VG Name                kvm2-storage
  LV UUID                AmWwRk-mHIR-QL15-j8kE-w8ad-Kc3j-wcJWWJ
  LV Write Access        read/write
  LV Creation host, time proxmox1, 2012-09-11 18:59:16 +0200
  LV Status              NOT available
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/kvm2-storage/vm-105-disk-2
  LV Name                vm-105-disk-2
  VG Name                kvm2-storage
  LV UUID                OuICGp-WT3w-AXPa-vgva-iCuC-zpjn-wcsj1u
  LV Write Access        read/write
  LV Creation host, time proxmox1, 2012-09-11 19:29:39 +0200
  LV Status              available
  # open                 1
  LV Size                150.00 GiB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:6
   
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                FtFvPc-PSde-f60i-IlTb-WOMj-uITz-8zUHic
  LV Write Access        read/write
  LV Creation host, time proxmox, 2012-06-06 16:58:53 +0200
  LV Status              available
  # open                 1
  LV Size                15.00 GiB
  Current LE             3840
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                0wwx99-ypIB-XAXT-6eHJ-rbNK-ze2t-umxNhc
  LV Write Access        read/write
  LV Creation host, time proxmox, 2012-06-06 16:58:53 +0200
  LV Status              available
  # open                 1
  LV Size                58.00 GiB
  Current LE             14848
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/pve/data
  LV Name                data
  VG Name                pve
  LV UUID                GWl7b9-05aR-DBvh-Olp9-4ePm-TFNI-pwTElx
  LV Write Access        read/write
  LV Creation host, time proxmox, 2012-06-06 16:58:53 +0200
  LV Status              available
  # open                 1
  LV Size                142.88 GiB
  Current LE             36576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Code: 
root@proxmox1:/# multipath -ll
mpath2 (36d4ae5200099eac7000003083e291df5) dm-3 DELL,MD32xxi
size=1.5T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| `- 4:0:0:1 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 3:0:0:1 sdc 8:32 active ghost running
mpath1 (36d4ae5200099eac7000003063e291db8) dm-4 DELL,MD32xxi
size=500G features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| `- 4:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 3:0:0:0 sdb 8:16 active ghost running
mpath3 (36d4ae5200099eac7000003013e291adc) dm-5 DELL,MD32xxi
size=50G features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| `- 4:0:0:2 sdg 8:96 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 3:0:0:2 sdd 8:48 active ghost running
Someone has an idea why these errors occurs ?
 
Last edited:
OK.
Could you help me to check where are instruction or conf files to check if my iscsi volumes are mapped directly with sd* devices ?
I presume it could be better if UUID or aliased mapping name were used to mount iscsi volumes.

in dmsg :
Code: 
...
sd 4:0:0:1: [sdf] Attached SCSI disk
 unknown partition table
sd 4:0:0:2: [sdg] Attached SCSI disk
sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:0: [sdb] Sense Key : Illegal Request [current] 
sd 3:0:0:0: [sdb] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Buffer I/O error on device sdb, logical block 0
sd 3:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:1: [sdc] Sense Key : Illegal Request [current] 
sd 3:0:0:1: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
sd 3:0:0:1: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Buffer I/O error on device sdc, logical block 0
sd 3:0:0:2: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:2: [sdd] Sense Key : Illegal Request [current] 
sd 3:0:0:2: [sdd] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
sd 3:0:0:2: [sdd] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Buffer I/O error on device sdd, logical block 0
sd 3:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:1: [sdc] Sense Key : Illegal Request [current] 
sd 3:0:0:1: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
sd 3:0:0:1: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Buffer I/O error on device sdc, logical block 0
sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:0: [sdb] Sense Key : Illegal Request [current] 
sd 3:0:0:0: [sdb] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Buffer I/O error on device sdb, logical block 0
...

- is it an old mapping between sdb, sdc or sdd with iscsi disk ?
- how could I check it ?
- how can I delete this message without dammage on my system ?

Any idea will be really appreciate :)
 
Last edited:
Dear all,

s/o - an idea ?
My proxmox server works fine, but I have this error (see previous messages)...
Do you think it could be solve by modify lvm.conf file ? another suggestion ?

Thanks.


EDIT.
In lvm.conf file, some leads with "prefered names" and "filters". But I don't want to break my server without more explanations...
Code: 
# be used by the LVM system.
devices {

    # Where do you want your volume groups to appear ?
    dir = "/dev"

    # An array of directories that contain the device nodes you wish
    # to use with LVM2.
    scan = [ "/dev" ]

    # If set, the cache of block device nodes with all associated symlinks
    # will be constructed out of the existing udev database content.
    # This avoids using and opening any inapplicable non-block devices or
    # subdirectories found in the device directory. This setting is applied
    # to udev-managed device directory only, other directories will be scanned
    # fully. LVM2 needs to be compiled with udev support for this setting to
    # take effect. N.B. Any device node or symlink not managed by udev in
    # udev directory will be ignored with this setting on.
    obtain_device_list_from_udev = 1

    # If several entries in the scanned directories correspond to the
    # same block device and the tools need to display a name for device,
    # all the pathnames are matched against each item in the following
    # list of regular expressions in turn and the first match is used.
    preferred_names = [ ]

    # Try to avoid using undescriptive /dev/dm-N names, if present.
    # preferred_names = [ "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d" ]

    # A filter that tells LVM2 to only use a restricted set of devices.
    # The filter consists of an array of regular expressions.  These
    # expressions can be delimited by a character of your choice, and
    # prefixed with either an 'a' (for accept) or 'r' (for reject).
    # The first expression found to match a device name determines if
    # the device will be accepted or rejected (ignored).  Devices that
    # don't match any patterns are accepted.

    # Be careful if there there are symbolic links or multiple filesystem
    # entries for the same device as each name is checked separately against
    # the list of patterns.  The effect is that if the first pattern in the
    # list to match a name is an 'a' pattern for any of the names, the device
    # is accepted; otherwise if the first pattern in the list to match a name
    # is an 'r' pattern for any of the names it is rejected; otherwise it is
    # accepted.

    # Don't have more than one filter line active at once: only one gets used.
 
Last edited:
I try once again...

These errors continue
Code: 
[COLOR=#000000][FONT=Times New Roman][FONT=tahoma]INFO:   /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdb: read failed after 0 of 4096 at 536870846464: Input/output error
INFO:   /dev/sdb: read failed after 0 of 4096 at 536870903808: Input/output error
INFO:   /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdd: read failed after 0 of 4096 at 1610612670464: Input/output error
INFO:   /dev/sdd: read failed after 0 of 4096 at 1610612727808: Input/output error
INFO:   /dev/sdd: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdf: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdf: read failed after 0 of 4096 at 53687025664: Input/output error
INFO:   /dev/sdf: read failed after 0 of 4096 at 53687083008: Input/output error
INFO:   /dev/sdf: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdi: read failed after 0 of 4096 at 2147483582464: Input/output error
INFO:   /dev/sdi: read failed after 0 of 4096 at 2147483639808: Input/output error
INFO:   /dev/sdi: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdk: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdk: read failed after 0 of 4096 at 536870846464: Input/output error
INFO:   /dev/sdk: read failed after 0 of 4096 at 536870903808: Input/output error
INFO:   /dev/sdk: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdm: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdm: read failed after 0 of 4096 at 1610612670464: Input/output error
INFO:   /dev/sdm: read failed after 0 of 4096 at 1610612727808: Input/output error
INFO:   /dev/sdm: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdo: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdo: read failed after 0 of 4096 at 53687025664: Input/output error
INFO:   /dev/sdo: read failed after 0 of 4096 at 53687083008: Input/output error
INFO:   /dev/sdo: read failed after 0 of 4096 at 4096: Input/output error
INFO:   /dev/sdp: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/sdp: read failed after 0 of 4096 at 2147483582464: Input/output error
INFO:   /dev/sdp: read failed after 0 of 4096 at 2147483639808: Input/output error
INFO:   /dev/sdp: read failed after 0 of 4096 at 4096: Input/output error

any idea ??
[/FONT][/FONT][/COLOR]
 
I have this same issue with my MD3200i system.

I have purchased Proxmox support so that they can fix it for me.

The errors are due to the second slave controller being the backup controller for your active virtual disk - as this only becomes active once the primary controller fails the iSCSI connection gets refused.

To test it, go into to your array controller and swap the active controller for your virtual disk.

The errors will stay but be on a different set of /dev/sd devices.
 
Hi,

The problem comes from a bad "filter" in /etc/lvm/lvm.conf.

If you have only one local disk /dev/sda, adapt your lvm.conf to accept /dev/sda and refuse any /dev/sd* with something like :
filter = ["a/sda/",r/sd*/"] (IIRC, I'm at home and md3220i is at work!).


Christophe.
 
THanks for your answers.@hotwired007 : could you give more detail about "To test it, go into to your array controller and swap the active controller for your virtual disk."

@Christophe : I think so... Do you thinks it could explain this new problem (sd_ 8:192 failed undef running, 1 for each path):
Code: 
root@proxmox1:~# multipath -l
mpath2 (36d4ae5200099eac7000003083e291df5) dm-3 DELL,MD32xxi
size=1.5T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 5:0:0:1 sdm 8:192 failed undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 3:0:0:1 sdd 8:48  active undef running
|-+- policy='round-robin 0' prio=-1 status=active
| `- 4:0:0:1 sde 8:64  active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 6:0:0:1 sdl 8:176 active undef running
mpath1 (36d4ae5200099eac7000003063e291db8) dm-4 DELL,MD32xxi
size=500G features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| `- 6:0:0:0 sdj 8:144 active undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 4:0:0:0 sdc 8:32  active undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 3:0:0:0 sdb 8:16  active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 5:0:0:0 sdk 8:160 failed undef running
mpath4 (36d4ae5200099e7d80000055750d01ec9) dm-6 DELL,MD32xxi
size=2.0T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 5:0:0:4 sdq 65:0  failed undef running
|-+- policy='round-robin 0' prio=-1 status=active
| `- 3:0:0:4 sdh 8:112 active undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 4:0:0:4 sdi 8:128 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 6:0:0:4 sdp 8:240 active undef running
mpath3 (36d4ae5200099eac7000003013e291adc) dm-5 DELL,MD32xxi
size=50G features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| `- 6:0:0:2 sdn 8:208 active undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 3:0:0:2 sdf 8:80  active undef running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 4:0:0:2 sdg 8:96  active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 5:0:0:2 sdo 8:224 failed undef running



Something else...
| `- 5:0:0:4 sdq 65:0 failed undef running
Do you think it's possible to refer to the disk with id/uuid instead of devnode (sdq) reference ?

Thanks.
 
Last edited:
christophe said:
The problem comes from a bad "filter" in /etc/lvm/lvm.conf.
If you have only one local disk /dev/sda, adapt your lvm.conf to accept /dev/sda and refuse any /dev/sd* with something like :
filter = ["a/sda/",r/sd*/"] (IIRC, I'm at home and md3220i is at work!).
Click to expand...

Thanks for your help : I changed lvm.conf with
Code: 
    filter = [ "a/sda/", "r/sd.*/" ]

Then I reboot my node n°2 (no VM running on it), and it seems to be OK.
I will try tonight with the 1st node when users will be asleep.

To resume the situation, we have to check lvm.conf & multipath.conf to avoid devices conflicts ?
 
THanks for your answers.@hotwired007 : could you give more detail about "To test it, go into to your array controller and swap the active controller for your virtual disk."
Click to expand...

Use the DELL Powervault Modular Disk manager - manage the array - Storage & Copy Services - find the disk in the Disk group - right click and change - ownership/preferred path
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back