iSCSI storage, VM crashing

[omegared77]

Hey all,

I noticed that I have sometimes VM with read only FS, or crashed.
A VM reset solves the issue, still I'd like to know why it happens.

I have a 4 node cluster, running v5.4.6 (the 5th node has a hardware issue)

The SAN storage is a iSCSI HPE MSA1040

Then, the LUN is mounted as a bare metal LVM :


Each cluster member herits the datacenter settings, and is correctly connected to the SAN and LUN.






The MSA is configured on 4 ethernet ports in a dedicated VLAN




On monday, I added a new node in the cluster.
Fresh install, no VM.
The iSCSI storage has been mounted automatically, no issue... expect I had many VMs unreachable, displaying this king of messages :





I had to reset some of them, because I couldn't login anymore (no password prompt after the login one).
On some tohers, the / FS was read only, a restart solved the issue.

Any idea about how to fix this issue ?

 

[Alwin]

Did you check the syslog/journal of the nodes to see if anything pops up?

 

[omegared77]

Alwin,

In journalctl -xe I can see this :

Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:0: alua: device naa.600c0ff0003be5490000000000000000 port group 0 rel port 2
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:0: [sdd] Sense Key : Illegal Request [current]
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:0: [sdd] Add. Sense: Invalid command operation code
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:1: alua: supports implicit TPGS
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:1: alua: device naa.600c0ff0003be549f1358a5c01000000 port group 0 rel port 2
Jul 05 13:03:48 btz-pve2301 kernel: sd 2:0:0:0: alua: port group 00 state A preferred supports tolusNA
Jul 05 13:03:48 btz-pve2301 kernel: sd 3:0:0:0: alua: supports implicit TPGS
Jul 05 13:03:48 btz-pve2301 kernel: sd 3:0:0:0: alua: device naa.600c0ff0003be4db0000000000000000 port group 1 rel port 6
Jul 05 13:03:48 btz-pve2301 kernel: sd 3:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 05 13:03:48 btz-pve2301 kernel: sd 3:0:0:0: [sde] Sense Key : Illegal Request [current]
Jul 05 13:03:48 btz-pve2301 kernel: sd 3:0:0:0: [sde] Add. Sense: Invalid command operation code

 

[omegared77]

I also got his :

[9659567.693226] sd 2:0:0:0: alua: supports implicit TPGS
[9659567.693232] sd 2:0:0:0: alua: device naa.600c0ff0003be5490000000000000000 port group 0 rel port 2
[9659567.694547] sd 2:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[9659567.694550] sd 2:0:0:0: [sdd] Sense Key : Illegal Request [current]
[9659567.694552] sd 2:0:0:0: [sdd] Add. Sense: Invalid command operation code