Help! only one node lost connection to ISCSI target, how to recover?

[m.ardito]

I had some ISCSI problems recently, and found one node had ISCSI troubles and VM with disks on it suffered.
Now I moved all VM to other node, and restarted that failing one, but ISCSI cannot be reconnected...
(see this other post http://forum.proxmox.com/threads/20752-Found-duplicate-PV)

How can I recover this situation?
I have left there some VMs (OFF) and they can't be restarted...
tried a bunch of ISCSI commands but there seem to be a major problem...

ISCSI host is a NAS, that can be reached by both nodes and shows no other problems

"good node"
# ping ts879
PING ts879 (192.168.3.249) 56(84) bytes of data.
64 bytes from ts879 (192.168.3.249): icmp_req=1 ttl=64 time=0.196 ms

#pvs
  PV         VG                 Fmt  Attr PSize    PFree
  /dev/sda2  pve                lvm2 a--    66.55g  8.37g
  /dev/sdb   pve_vm_disks_ts879 lvm2 a--  1000.00g 22.81g

# iscsiadm -m session
tcp: [1] 192.168.3.249:3260,1 iqn.2004-04.com.qnap:ts-879u-rp:iscsi.pve.d4e6fc



"bad node"
# ping ts879
PING ts879 (192.168.3.249) 56(84) bytes of data.
64 bytes from ts879 (192.168.3.249): icmp_req=1 ttl=64 time=0.115 ms

# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda2  pve  lvm2 a--  66.55g 8.37g

# iscsiadm -m session
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session50
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session51
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session52
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session53
iscsiadm: No active sessions.

the only difference I can see in /etc/iscsi (apart initiator name, obviously) seems to be that

"good node" has 
# ls -la /etc/iscsi/send_targets/
total 28
drw------- 7 root root 4096 Jul 10  2014 .
drwxr-xr-x 5 root root 4096 Nov  7  2013 ..
drw------- 2 root root 4096 Nov 21  2013 172.16.0.3,3260
drw------- 2 root root 4096 Jan 15 15:09 192.168.3.249,3260
drw------- 2 root root 4096 Jul 11  2014 192.168.3.30,3260
drw------- 2 root root 4096 Nov 20  2013 192.168.3.78,3260
drw------- 2 root root 4096 Jul 10  2014 ts879,3260


while "bad node" has
~# ls -la /etc/iscsi/send_targets/
total 24
drw------- 6 root root 4096 Nov 27  2013 .
drwxr-xr-x 5 root root 4096 Nov 19  2013 ..
drw------- 2 root root 4096 Nov 21  2013 172.16.0.3,3260
drw------- 2 root root 4096 Jun 14  2014 192.168.3.30,3260
drw------- 2 root root 4096 Nov 20  2013 192.168.3.78,3260
drw------- 2 root root 4096 Sep 15 20:17 ts879,3260

Thanks, Marco

 

[m.ardito]

after issuing a discovery command

# iscsiadm --mode discovery --type sendtargets --portal 192.168.3.249
iscsiadm: Login response timeout. Waited 30 seconds and did not get reponse PDU.

iscsiadm: discovery login to 192.168.3.249 failed, giving up 2
iscsiadm: Could not perform SendTargets discovery: encountered non-retryable iSCSI login failure

now I got (as on the "good" node)

# ls -la /etc/iscsi/send_targets
total 28
drw------- 7 root root 4096 Jan 16 16:58 .
drwxr-xr-x 5 root root 4096 Nov 19  2013 ..
drw------- 2 root root 4096 Nov 21  2013 172.16.0.3,3260
drw------- 2 root root 4096 Jan 16 16:58 192.168.3.249,3260
drw------- 2 root root 4096 Jun 14  2014 192.168.3.30,3260
drw------- 2 root root 4096 Nov 20  2013 192.168.3.78,3260
drw------- 2 root root 4096 Sep 15 20:17 ts879,3260

but nothing changes...

any suggestion?

Marco

 

[udo]

...

...
"bad node"
# ping ts879
PING ts879 (192.168.3.249) 56(84) bytes of data.
64 bytes from ts879 (192.168.3.249): icmp_req=1 ttl=64 time=0.115 ms

# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda2  pve  lvm2 a--  66.55g 8.37g

# iscsiadm -m session
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session50
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session51
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session52
iscsiadm: could not read session targetname: 5
iscsiadm: could not find session info for session53
iscsiadm: No active sessions.

...

Hi Marco,
looks for me that an old session hang on the "bad" node, or that the privileges on the NAS forbit the access from the "bad" node.

Have you try an

/etc/init.d/open-iscsi restart

Udo

 

[m.ardito]

Hi Marco,
looks for me that an old session hang on the "bad" node, or that the privileges on the NAS forbit the access from the "bad" node.

Have you try an

/etc/init.d/open-iscsi restart

Udo

Thanks udo, trying to connect that "bad" node to another NAS ISCSI is possible, so I think the problem could be in the NAS.
I am currently (slowly) moving all running VM disk to another NAS ISCSI LUN, then will try to restart NAS ISCSI service, or if all fails, reboot it.

will report results.

Marco