[SOLVED] iSCSI issue - node down - help please

hotwired007

Member
Sep 19, 2011
533
4
16
UK
came into work this morning to find on of my nodes was down and unable to connect to the iSCSI but the other node can connect fine.

my setup is that i ahve a readynas 2100 with two 512GB iSCSI shares - IMAGES-SCSI and IMAGES-TWO-SCSI, both with LVM ontop (IMAGES-LVM & IMAGES-TWO-LVM).

On the node where the iSCSI has failed, the system log shows:

Code:
Aug 24 09:21:21 bellerophon kernel: connection16:0: detected conn error (1020)
Aug 24 09:21:21 bellerophon iscsid: Kernel reported iSCSI connection 16:0 error (1020) state (2)
Aug 24 09:21:25 bellerophon pvedaemon[3497]: starting vnc proxy UPID:bellerophon:00000DA9:0002DE3F:50373985:vncproxy:123:root@pam:
Aug 24 09:21:25 bellerophon pvedaemon[2472]: <root@pam> starting task UPID:bellerophon:00000DA9:0002DE3F:50373985:vncproxy:123:root@pam:
Aug 24 09:21:25 bellerophon pvedaemon[2471]: <root@pam> successful auth for user 'root@pam'
Aug 24 09:21:26 bellerophon pvedaemon[3497]: command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 123 2>/dev/null'' failed: exit code 2
Aug 24 09:21:26 bellerophon pvedaemon[2472]: <root@pam> end task UPID:bellerophon:00000DA9:0002DE3F:50373985:vncproxy:123:root@pam: command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 123 2>/dev/null'' f$
Aug 24 09:21:31 bellerophon pvedaemon[3500]: start VM 123: UPID:bellerophon:00000DAC:0002E042:5037398B:qmstart:123:root@pam:
Aug 24 09:21:31 bellerophon pvedaemon[2471]: <root@pam> starting task UPID:bellerophon:00000DAC:0002E042:5037398B:qmstart:123:root@pam:
Aug 24 09:21:32 bellerophon pvedaemon[3504]: starting vnc proxy UPID:bellerophon:00000DB0:0002E0B0:5037398C:vncproxy:123:root@pam:
Aug 24 09:21:32 bellerophon pvedaemon[2472]: <root@pam> starting task UPID:bellerophon:00000DB0:0002E0B0:5037398C:vncproxy:123:root@pam:
Aug 24 09:21:32 bellerophon pvedaemon[3504]: command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 123 2>/dev/null'' failed: exit code 2
Aug 24 09:21:32 bellerophon pvedaemon[2472]: <root@pam> end task UPID:bellerophon:00000DB0:0002E0B0:5037398C:vncproxy:123:root@pam: command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 123 2>/dev/null'' f$
Aug 24 09:22:44 bellerophon pvedaemon[3500]: WARNING: command '/usr/bin/iscsiadm --mode discovery --type sendtargets --portal 192.168.100.6' failed: exit code 255
Aug 24 09:22:58 bellerophon pvestatd[2681]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images --login' failed: exit code 255
Aug 24 09:22:58 bellerophon pvestatd[2681]: status update time (272.512 seconds)
Aug 24 09:23:00 bellerophon kernel: scsi23 : iSCSI Initiator over TCP/IP
Aug 24 09:23:00 bellerophon kernel: connection17:0: detected conn error (1020)
Aug 24 09:23:00 bellerophon pvestatd[2681]: WARNING: storage 'IMAGES-TWO-ISCSI' is not online
Aug 24 09:23:01 bellerophon iscsid: Kernel reported iSCSI connection 17:0 error (1020) state (2)
Aug 24 09:23:28 bellerophon pvedaemon[2471]: <root@pam> successful auth for user 'root@pam'
Aug 24 09:24:19 bellerophon pvedaemon[2474]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images-two --login' failed: exit code 255
Aug 24 09:24:20 bellerophon iscsid: session [iqn.2012-03.ReadyNAS2:images-two,192.168.100.6,3260] already running.
Aug 24 09:24:21 bellerophon pvedaemon[2474]: WARNING: storage 'IMAGES-ISCSI' is not online
Aug 24 09:24:23 bellerophon pvedaemon[2474]: WARNING: storage 'IMAGES-ISCSI' is not online
Aug 24 09:24:48 bellerophon pvestatd[2681]: WARNING: command '/usr/bin/iscsiadm --mode discovery --type sendtargets --portal 192.168.100.6' failed: exit code 255
Aug 24 09:24:50 bellerophon kernel: scsi24 : iSCSI Initiator over TCP/IP
Aug 24 09:24:50 bellerophon kernel: connection18:0: detected conn error (1020)
Aug 24 09:24:51 bellerophon iscsid: Kernel reported iSCSI connection 18:0 error (1020) state (2)
Aug 24 09:24:54 bellerophon pvedaemon[3500]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images-two --login' failed: exit code 255
Aug 24 09:24:54 bellerophon pvedaemon[3500]: can't activate LV '/dev/IMAGES-TWO-LVM/vm-123-disk-1':   Skipping volume group IMAGES-TWO-LVM
Aug 24 09:24:54 bellerophon pvedaemon[2471]: <root@pam> end task UPID:bellerophon:00000DAC:0002E042:5037398B:qmstart:123:root@pam: can't activate LV '/dev/IMAGES-TWO-LVM/vm-123-disk-1':   Skipping volume group I$
Aug 24 09:26:03 bellerophon kernel: connection18:0: detected conn error (1020)
Aug 24 09:26:04 bellerophon iscsid: Kernel reported iSCSI connection 18:0 error (1020) state (2)
Aug 24 09:27:03 bellerophon pvestatd[2681]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images --login' failed: exit code 255
Aug 24 09:27:35 bellerophon kernel: scsi25 : iSCSI Initiator over TCP/IP
Aug 24 09:28:13 bellerophon kernel: connection19:0: detected conn error (1020)
Aug 24 09:28:14 bellerophon iscsid: Kernel reported iSCSI connection 19:0 error (1020) state (2)
Aug 24 09:28:37 bellerophon pvedaemon[2474]: <root@pam> successful auth for user 'root@pam'
Aug 24 09:29:17 bellerophon pvestatd[2681]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images-two --login' failed: exit code 255
Aug 24 09:29:17 bellerophon pvestatd[2681]: status update time (378.261 seconds)
Aug 24 09:29:36 bellerophon kernel: scsi26 : iSCSI Initiator over TCP/IP
Aug 24 09:31:01 bellerophon pmxcfs[1497]: [dcdb] notice: data verification successful
any suggestions what i should check?
 
Last edited:

hotwired007

Member
Sep 19, 2011
533
4
16
UK
Re: iSCSI issue - node down - help please

tried restarting the iscsi service on the node but no difference - i have also looked at the logs and found that my 'primary' node (first node i configured) is having no issues at all and all the others are having issues - i created to new nodes and they are both having issues connecting to the ReadyNAS.
 

hotwired007

Member
Sep 19, 2011
533
4
16
UK
Re: iSCSI issue - node down - help please

Aug 24 14:40:21 enterprise pvedaemon[1743]: WARNING: command '/usr/bin/iscsiadm --mode node --targetname iqn.2012-03.ReadyNAS2:images-two --login' failed: exit code 255

what does this mean? there is no username or password configured on my readynas :S
 

spirit

Well-Known Member
Apr 2, 2010
3,434
147
63
www.odiso.com
Re: iSCSI issue - node down - help please

iscsi login is a basic command to establish connection to your san. (user/password no mandatory).

Can you ping your san from the non working proxmox node?
 

hotwired007

Member
Sep 19, 2011
533
4
16
UK
Re: iSCSI issue - node down - help please

yes.

Code:
root@enterprise:~# ping 192.168.100.6
PING 192.168.100.6 (192.168.100.6) 56(84) bytes of data.
64 bytes from 192.168.100.6: icmp_req=1 ttl=64 time=0.239 ms
64 bytes from 192.168.100.6: icmp_req=2 ttl=64 time=0.216 ms
64 bytes from 192.168.100.6: icmp_req=3 ttl=64 time=0.224 ms
64 bytes from 192.168.100.6: icmp_req=4 ttl=64 time=0.214 ms
64 bytes from 192.168.100.6: icmp_req=5 ttl=64 time=0.226 ms
64 bytes from 192.168.100.6: icmp_req=6 ttl=64 time=0.239 ms
64 bytes from 192.168.100.6: icmp_req=7 ttl=64 time=0.214 ms
64 bytes from 192.168.100.6: icmp_req=8 ttl=64 time=0.217 ms
64 bytes from 192.168.100.6: icmp_req=9 ttl=64 time=0.225 ms
64 bytes from 192.168.100.6: icmp_req=10 ttl=64 time=0.218 ms
64 bytes from 192.168.100.6: icmp_req=11 ttl=64 time=0.216 ms
64 bytes from 192.168.100.6: icmp_req=12 ttl=64 time=0.218 ms
64 bytes from 192.168.100.6: icmp_req=13 ttl=64 time=0.220 ms
64 bytes from 192.168.100.6: icmp_req=14 ttl=64 time=0.225 ms
^C
--- 192.168.100.6 ping statistics ---
14 packets transmitted, 14 received, 0% packet loss, time 12999ms
rtt min/avg/max/mdev = 0.214/0.222/0.239/0.012 ms
 

hotwired007

Member
Sep 19, 2011
533
4
16
UK
Re: iSCSI issue - node down - help please

just had a look at the logs on the readynas box:

Code:
Aug 24 15:02:47 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:02:48 ReadyNAS2 last message repeated 3 times
Aug 24 15:02:52 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:02:57 ReadyNAS2 last message repeated 3 times
Aug 24 15:03:06 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:03:06 ReadyNAS2 last message repeated 2 times
Aug 24 15:03:31 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:03:33 ReadyNAS2 last message repeated 9 times
Aug 24 15:03:39 ReadyNAS2 kernel: net_ratelimit: 26 callbacks suppressed
Aug 24 15:03:39 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:03:39 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:03:43 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
Aug 24 15:03:48 ReadyNAS2 last message repeated 7 times
Aug 24 15:03:55 ReadyNAS2 kernel: TCP: Possible SYN flooding on port 3260. Sending cookies.
its a closed network!
 

spirit

Well-Known Member
Apr 2, 2010
3,434
147
63
www.odiso.com
Re: iSCSI issue - node down - help please

"kernel: TCP: Possible SYN flooding on port 3260. Sending cookies."
is because your proxmox host try to established too much connection I think.
(you can try to add net.ipv4.tcp_syncookies = 1 in your /etc/sysctl.conf , then sysctl -p /etc/sysctl.conf to reload conf).

not sure it help, but it should mitigate the syn flood.
 

hotwired007

Member
Sep 19, 2011
533
4
16
UK
Re: iSCSI issue - node down - help please

took the cluster down to do a firmware update on the ReadyNAS 2100 to .21 - had various fixes for iSCSI.

Looks like adding a second iSCSI volume caused an issue also configured each iSCSI with the IQNs from each of the Proxmox hosts.

had an issue with an old iSCSI connection causing flooding of the log files and an error message everytime i migrated a machine but rebooting each node seems to have fixed it.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!