Error "Read Capacity" on all nodes accessing iSCSI

[fips]

Hey,

I recently add a FreeNAS storage to my cluster (3 Nodes, newest 4.2, full updated) and connected the nodes via iSCSI to it.
Now I have tons of that error messages:
Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

I have seen, that other users have those errors too, but there was never a suggestion what to check or how to fix it.

Here the log when the error occurred on the nodes:
Node1:
Sep 7 11:11:29 vmbase1 pveproxy[18616]: starting 1 worker(s)

Sep 7 11:11:29 vmbase1 pveproxy[18616]: worker 6163 started

Sep 7 11:11:42 vmbase1 kernel: [44464.255833] scsi 4:0:0:1: Direct-Access FreeNAS iSCSI Disk 0123 PQ: 0 ANSI: 6

Sep 7 11:11:43 vmbase1 iscsid: Connection1:0 to [target: iqn.2016-08.it.fuerst.storage0.ctl:dctarget, portal: 172.30.2.100,3260] through [iface: default] is operational now

Sep 7 11:14:56 vmbase1 kernel: [44657.885889] blk_update_request: I/O error, dev sdc, sector 10695475184

Sep 7 11:15:02 vmbase1 kernel: [44663.474365] sd 4:0:0:0: [sdd] Write Protect is off

Sep 7 11:15:02 vmbase1 kernel: [44663.474368] sd 4:0:0:0: [sdd] Mode Sense: 73 00 10 08

Sep 7 11:15:02 vmbase1 kernel: [44663.474741] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA

Sep 7 11:16:14 vmbase1 kernel: [44735.960895] sd 4:0:0:1: [sdc] Unit Not Ready

Sep 7 11:16:14 vmbase1 kernel: [44735.961096] sd 4:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:16:14 vmbase1 kernel: [44735.961310] sd 4:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:17:01 vmbase1 CRON[8899]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Sep 7 11:17:51 vmbase1 pmxcfs[2647]: [status] notice: received log

Sep 7 11:17:51 vmbase1 pmxcfs[2647]: [status] notice: received log

Sep 7 11:19:33 vmbase1 kernel: [44934.326414] sd 4:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:19:33 vmbase1 kernel: [44934.326593] sd 4:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:19:33 vmbase1 kernel: [44934.326599] sd 4:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:19:33 vmbase1 kernel: [44934.327078] sd 4:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:19:52 vmbase1 kernel: [44954.162044] sd 4:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:19:52 vmbase1 kernel: [44954.162200] sd 4:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:20:32 vmbase1 kernel: [44994.042895] sd 4:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:20:32 vmbase1 kernel: [44994.043057] sd 4:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:21:39 vmbase1 systemd-timesyncd[1831]: interval/delta/delay/jitter/drift 2048s/+0.000s/0.016s/0.001s/+20ppm

Sep 7 11:21:57 vmbase1 pveproxy[6163]: worker exit

Sep 7 11:21:57 vmbase1 pveproxy[18616]: worker 6163 finished

Sep 7 11:21:57 vmbase1 pveproxy[18616]: starting 1 worker(s)

Sep 7 11:21:57 vmbase1 pveproxy[18616]: worker 10762 started

Sep 7 11:22:11 vmbase1 pveproxy[18618]: worker exit

Sep 7 11:22:11 vmbase1 pveproxy[18616]: worker 18618 finished

Sep 7 11:22:11 vmbase1 pveproxy[18616]: starting 1 worker(s)

Sep 7 11:22:11 vmbase1 pveproxy[18616]: worker 10825 started

Sep 7 11:22:12 vmbase1 kernel: [45094.195891] sd 4:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE


Node2:
Sep 7 10:56:03 vmbase2 systemd-timesyncd[1880]: interval/delta/delay/jitter/drift 2048s/-0.002s/0.009s/0.003s/-10ppm

Sep 7 11:09:43 vmbase2 pmxcfs[2689]: [status] notice: received log

Sep 7 11:11:41 vmbase2 kernel: [44748.774316] scsi host6: iSCSI Initiator over TCP/IP

Sep 7 11:11:41 vmbase2 kernel: [44749.029880] scsi 6:0:0:1: Direct-Access FreeNAS iSCSI Disk 0123 PQ: 0 ANSI: 6

Sep 7 11:11:41 vmbase2 kernel: [44749.030159] sd 6:0:0:1: Attached scsi generic sg2 type 0

Sep 7 11:11:41 vmbase2 kernel: [44749.030446] sd 6:0:0:1: [sdc] 10695475200 512-byte logical blocks: (5.48 TB/4.98 TiB)

Sep 7 11:11:41 vmbase2 kernel: [44749.030448] sd 6:0:0:1: [sdc] 131072-byte physical blocks

Sep 7 11:11:41 vmbase2 kernel: [44749.030824] sd 6:0:0:1: [sdc] Write Protect is off

Sep 7 11:11:41 vmbase2 kernel: [44749.030826] sd 6:0:0:1: [sdc] Mode Sense: 73 00 10 08

Sep 7 11:11:41 vmbase2 kernel: [44749.030929] sd 6:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA

Sep 7 11:11:41 vmbase2 kernel: [44749.034088] sd 6:0:0:1: [sdc] Attached SCSI disk

Sep 7 11:11:42 vmbase2 iscsid: Connection1:0 to [target: iqn.2016-08.it.fuerst.storage0.ctl:dctarget, portal: 172.30.2.100,3260] through [iface: default] is operational now

Sep 7 11:15:01 vmbase2 kernel: [44949.251024] sd 6:0:0:1: [sdc] Unit Not Ready

Sep 7 11:15:01 vmbase2 kernel: [44949.251027] sd 6:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:01 vmbase2 kernel: [44949.251029] sd 6:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:15:01 vmbase2 kernel: [44949.251216] sd 6:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:15:01 vmbase2 kernel: [44949.251218] sd 6:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:01 vmbase2 kernel: [44949.251219] sd 6:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:15:01 vmbase2 kernel: [44949.251425] sd 6:0:0:1: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:15:01 vmbase2 kernel: [44949.251427] sd 6:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:01 vmbase2 kernel: [44949.251428] sd 6:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:15:01 vmbase2 kernel: [44949.251997] sdc: detected capacity change from 5476083302400 to 0

Sep 7 11:15:01 vmbase2 kernel: [44949.252227] scsi 6:0:0:0: Direct-Access FreeNAS iSCSI Disk 0123 PQ: 0 ANSI: 6

Sep 7 11:15:01 vmbase2 kernel: [44949.252547] sd 6:0:0:0: Attached scsi generic sg3 type 0

Sep 7 11:15:01 vmbase2 kernel: [44949.252549] sd 6:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical


Node 3:
Sep 7 11:05:01 vmbase3 CRON[8684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Sep 7 11:09:43 vmbase3 pmxcfs[2922]: [status] notice: received log

Sep 7 11:11:37 vmbase3 kernel: [44937.298964] scsi 7:0:0:1: Direct-Access FreeNAS iSCSI Disk 0123 PQ: 0 ANSI: 6

Sep 7 11:11:37 vmbase3 kernel: [44937.299795] sd 7:0:0:1: [sdc] 131072-byte physical blocks

Sep 7 11:11:37 vmbase3 kernel: [44937.300265] sd 7:0:0:1: [sdc] Mode Sense: 73 00 10 08

Sep 7 11:11:38 vmbase3 iscsid: Connection1:0 to [target: iqn.2016-08.it.fuerst.storage0.ctl:dctarget, portal: 172.30.2.100,3260] through [iface: default] is operational now

Sep 7 11:13:27 vmbase3 systemd-timesyncd[2151]: interval/delta/delay/jitter/drift 2048s/-0.000s/0.016s/0.000s/-50ppm

Sep 7 11:14:56 vmbase3 kernel: [45136.722724] sd 7:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:14:56 vmbase3 kernel: [45136.723123] sd 7:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:14:56 vmbase3 kernel: [45136.723583] sd 7:0:0:1: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:14:56 vmbase3 kernel: [45136.723587] sd 7:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:14:56 vmbase3 kernel: [45136.725058] scsi 7:0:0:0: Direct-Access FreeNAS iSCSI Disk 0123 PQ: 0 ANSI: 6

Sep 7 11:14:56 vmbase3 kernel: [45136.725514] sd 7:0:0:0: Attached scsi generic sg3 type 0

Sep 7 11:14:56 vmbase3 kernel: [45136.725688] sd 7:0:0:0: [sdd] 131072-byte physical blocks

Sep 7 11:14:56 vmbase3 kernel: [45136.725990] sd 7:0:0:0: [sdd] Mode Sense: 73 00 10 08

Sep 7 11:14:56 vmbase3 kernel: [45136.729236] sd 7:0:0:0: [sdd] Attached SCSI disk

Sep 7 11:15:01 vmbase3 CRON[10696]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Sep 7 11:15:16 vmbase3 kernel: [45155.988743] sd 7:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:16 vmbase3 kernel: [45155.988909] sd 7:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Sep 7 11:15:16 vmbase3 kernel: [45155.988914] sd 7:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:15:16 vmbase3 kernel: [45155.989080] sd 7:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:27 vmbase3 kernel: [45167.342379] sd 7:0:0:1: [sdc] Unit Not Ready

Sep 7 11:15:27 vmbase3 kernel: [45167.342385] sd 7:0:0:1: [sdc] Sense Key : Illegal Request [current]

Sep 7 11:15:27 vmbase3 kernel: [45167.342389] sd 7:0:0:1: [sdc] Add. Sense: Logical unit not supported

Sep 7 11:15:27 vmbase3 kernel: [45167.342594] sd 7:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE






Thanks
Steve

 

[fips]

After rebooting node1, it doesn't bring those messages. node2 and node3 still annoying with it.
I ran lsscsi an all three nodes and saw that result:

node1:
root@vmbase1:~# lsscsi
[0:0:0:0] disk ATA WDC WD10EFRX-68J 1A01 /dev/sda
[1:0:0:0] disk ATA SAMSUNG HD103SJ 0001 /dev/sdb
[4:0:0:0] disk FreeNAS iSCSI Disk 0123 /dev/sdc

node2:
root@vmbase2:/var/log# lsscsi
[0:0:0:0] disk ATA WDC WD5002ABYS-0 3B03 /dev/sda
[1:0:0:0] disk ATA WDC WD5000ABYS-0 1C01 /dev/sdb
[6:0:0:0] disk FreeNAS iSCSI Disk 0123 /dev/sdd
[6:0:0:1] disk FreeNAS iSCSI Disk 0123 /dev/sdc

node3:
root@vmbase3:/etc/pve# lsscsi
[1:0:0:0] disk ATA TS256GSSD370S 9A /dev/sda
[3:0:0:0] disk ATA TS256GSSD370S 9A /dev/sdb
[7:0:0:0] disk FreeNAS iSCSI Disk 0123 /dev/sdd
[7:0:0:1] disk FreeNAS iSCSI Disk 0123 /dev/sdc


I guess that /dev/sdd makes those troubles.
Can I remove /dev/sdd without remove /dev/sdc and shutting down all running VMs?