Hello,
I have cluster 3 nodes with QSAN SAN. I have conected 2 LUNs iSCSI over multipatch to nodes, create LVM groups and connect them to each node.
Every thing works fine to moment when I create VM with two disks one on the one LUN second on the second LUN.
VM works properly, but when I try remove VM one multipath stop working and node try removing VM to infinity. Multipath with second LUN is failed. All VMs on this node are without status.
When I have two disks on the same LUN I can remove VM without problem.
Have You some idea what is wrong?
Part syslog from node:
May 12 14:31:02 zeus13 pvedaemon[952850]: destroy VM 311: UPID:zeus13:000E8A12:0ED735BE:5551F286:qmdestroy:311:root@pam:
May 12 14:31:02 zeus13 pvedaemon[872090]: <root@pam> starting task UPID:zeus13:000E8A12:0ED735BE:5551F286:qmdestroy:311:root@pam:
May 12 14:31:04 zeus13 multipathd: dm-19: remove map (uevent)
May 12 14:31:04 zeus13 multipathd: dm-19: devmap not registered, can't remove
May 12 14:31:04 zeus13 multipathd: dm-19: remove map (uevent)
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 00 80 07 ff 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 8390655
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 02 80 07 fb 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 41945083
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] CDB: Write same(16): 93 08 00 00 00 00 01 80 07 fd 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sde, sector 25167869
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 04 80 07 f7 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 75499511
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 00 00 08 00 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 2048
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 01 00 07 fe 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 16779262
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 02 00 07 fc 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 33556476
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 03 00 07 fa 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 50333690
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 04 00 07 f8 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 67110904
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 05 00 07 f6 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 83888118
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:80.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:176.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:32.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:64.
May 12 14:31:04 zeus13 multipathd: 8:32: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 3
May 12 14:31:04 zeus13 multipathd: 8:64: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 2
May 12 14:31:04 zeus13 multipathd: 8:80: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 1
May 12 14:31:04 zeus13 multipathd: 8:176: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 0
May 12 14:31:05 zeus13 multipathd: mp_data01: sde - directio checker reports path is up
May 12 14:31:05 zeus13 multipathd: 8:64: reinstated
multipath -ll result:
mp_nfs01 (32031001378919140) dm-3 Qsan,P600Q-D316
size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 7:0:0:4 sdb 8:16 active ready running
|- 14:0:0:4 sdi 8:128 active ready running
|- 15:0:0:4 sdj 8:144 active ready running
`- 16:0:0:4 sdk 8:160 active ready running
mp_vms01 (3202d001378919140) dm-4 Qsan,P600Q-D316
size=2.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 9:0:0:0 sdd 8:48 active ready running
|- 12:0:0:0 sdg 8:96 active ready running
|- 13:0:0:0 sdh 8:112 active ready running
`- 18:0:0:0 sdm 8:192 active ready running
mp_data01 (32034001378919140) dm-5 Qsan,P600Q-D316
size=3.7T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=enabled
|- 8:0:0:6 sdc 8:32 failed ready running
|- 10:0:0:6 sde 8:64 failed ready running
|- 11:0:0:6 sdf 8:80 failed ready running
`- 17:0:0:6 sdl 8:176 failed ready running
I have cluster 3 nodes with QSAN SAN. I have conected 2 LUNs iSCSI over multipatch to nodes, create LVM groups and connect them to each node.
Every thing works fine to moment when I create VM with two disks one on the one LUN second on the second LUN.
VM works properly, but when I try remove VM one multipath stop working and node try removing VM to infinity. Multipath with second LUN is failed. All VMs on this node are without status.
When I have two disks on the same LUN I can remove VM without problem.
Have You some idea what is wrong?
Part syslog from node:
May 12 14:31:02 zeus13 pvedaemon[952850]: destroy VM 311: UPID:zeus13:000E8A12:0ED735BE:5551F286:qmdestroy:311:root@pam:
May 12 14:31:02 zeus13 pvedaemon[872090]: <root@pam> starting task UPID:zeus13:000E8A12:0ED735BE:5551F286:qmdestroy:311:root@pam:
May 12 14:31:04 zeus13 multipathd: dm-19: remove map (uevent)
May 12 14:31:04 zeus13 multipathd: dm-19: devmap not registered, can't remove
May 12 14:31:04 zeus13 multipathd: dm-19: remove map (uevent)
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 00 80 07 ff 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 8390655
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 02 80 07 fb 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 41945083
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 10:0:0:6: [sde] CDB: Write same(16): 93 08 00 00 00 00 01 80 07 fd 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sde, sector 25167869
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 17:0:0:6: [sdl] CDB: Write same(16): 93 08 00 00 00 00 04 80 07 f7 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdl, sector 75499511
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 00 00 08 00 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 2048
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 01 00 07 fe 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 16779262
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 02 00 07 fc 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 33556476
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 03 00 07 fa 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 50333690
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 11:0:0:6: [sdf] CDB: Write same(16): 93 08 00 00 00 00 04 00 07 f8 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdf, sector 67110904
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Sense Key : Illegal Request [current]
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] Add. Sense: Invalid field in cdb
May 12 14:31:04 zeus13 kernel: sd 8:0:0:6: [sdc] CDB: Write same(16): 93 08 00 00 00 00 05 00 07 f6 00 7f ff ff 00 00
May 12 14:31:04 zeus13 kernel: end_request: critical target error, dev sdc, sector 83888118
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:80.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:176.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:32.
May 12 14:31:04 zeus13 kernel: device-mapper: multipath: Failing path 8:64.
May 12 14:31:04 zeus13 multipathd: 8:32: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 3
May 12 14:31:04 zeus13 multipathd: 8:64: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 2
May 12 14:31:04 zeus13 multipathd: 8:80: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 1
May 12 14:31:04 zeus13 multipathd: 8:176: mark as failed
May 12 14:31:04 zeus13 multipathd: mp_data01: remaining active paths: 0
May 12 14:31:05 zeus13 multipathd: mp_data01: sde - directio checker reports path is up
May 12 14:31:05 zeus13 multipathd: 8:64: reinstated
multipath -ll result:
mp_nfs01 (32031001378919140) dm-3 Qsan,P600Q-D316
size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 7:0:0:4 sdb 8:16 active ready running
|- 14:0:0:4 sdi 8:128 active ready running
|- 15:0:0:4 sdj 8:144 active ready running
`- 16:0:0:4 sdk 8:160 active ready running
mp_vms01 (3202d001378919140) dm-4 Qsan,P600Q-D316
size=2.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 9:0:0:0 sdd 8:48 active ready running
|- 12:0:0:0 sdg 8:96 active ready running
|- 13:0:0:0 sdh 8:112 active ready running
`- 18:0:0:0 sdm 8:192 active ready running
mp_data01 (32034001378919140) dm-5 Qsan,P600Q-D316
size=3.7T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=enabled
|- 8:0:0:6 sdc 8:32 failed ready running
|- 10:0:0:6 sde 8:64 failed ready running
|- 11:0:0:6 sdf 8:80 failed ready running
`- 17:0:0:6 sdl 8:176 failed ready running