iSCSI Reconnecting every 10 seconds to FreeNAS solution

D

derek

Renowned Member
Jan 3, 2015
6
0
66
Hi folks,

Proxmox seems to be connecting and reconnection every 10 seconds to my FreeNAS machine

Feb 28 10:26:11 nas ctld[57937]: 10.0.0.1: read: connection lost
Feb 28 10:26:11 nas ctld[2904]: child process 57937 terminated with exit status 1
Feb 28 10:26:21 nas ctld[57947]: 10.0.0.1: read: connection lost
Feb 28 10:26:21 nas ctld[2904]: child process 57947 terminated with exit status 1
Feb 28 10:26:31 nas ctld[57957]: 10.0.0.1: read: connection lost
Feb 28 10:26:31 nas ctld[2904]: child process 57957 terminated with exit status 1
Feb 28 10:26:41 nas ctld[57965]: 10.0.0.1: read: connection lost
Feb 28 10:26:41 nas ctld[2904]: child process 57965 terminated with exit status 1
Feb 28 10:26:51 nas ctld[57974]: 10.0.0.1: read: connection lost
Feb 28 10:26:51 nas ctld[2904]: child process 57974 terminated with exit status 1
Feb 28 10:27:01 nas ctld[57984]: 10.0.0.1: read: connection lost
Feb 28 10:27:01 nas ctld[2904]: child process 57984 terminated with exit status 1
Feb 28 10:27:11 nas ctld[57996]: 10.0.0.1: read: connection lost
Feb 28 10:27:11 nas ctld[2904]: child process 57996 terminated with exit status 1


I've opened up a ticket with the FreeNAS folks (https://bugs.freenas.org/issues/7891) and they seem to think the issue on the Proxmox side.

Does anyone have any insight into what is actually happening?

Thanks.

- Derek
 
FreeNas connection to proxmox with scsi is pretty stable here. Even synology to proxmox.
Some suggestions:
Replace the network cable
Check the switch - use another port

That currently came to my mind.
 
Thanks. I've replaced the cable (it's a crossover cable over 10Gbe) - same result. switched the port, same result. The connection is actually perfectly stable, it's not crashing, those messages are just showing up on the freenas console every 10 seconds.
 
Hello to all
yes. I can confirm the same behaviour like Derek
and do not know hot to solve this.
Is something telling us that FREENAS iscsi wouldn't be right choice ?

But performance seem to be correct and reliable ..

Best regards

Tonci
 
I have perfectly functioning iSCSI connections to both Qnap, Debian, and Solaris. To me this indicates a problem on the FreeNAS side.

PS. whenever a fault is detected involving FreeNAS the FreeNAS people always blames others. In there mind FreeNAS is always free of fault and all other solutions are error prone!!
 
I had the same issue when using lvm via iscsi on FreeNAS. What helped was:
1. Remove lvm storage via proxmox gui
2. Remove iscsi via proxmox gui, (The iscsi config on the system stays intact)
- Now the ctld messages should stop
3. Verify that the iscsi-connection is still active and the VG is there:
# iscsiadm -m session -o show
# vgs
Click to expand...
4. Add LVM via proxmox gui (it is shown as a local lvm)

This worked for me. So I think it must have something to do with a proxmox service.
 
Had same issue...
pokerface said:
I had the same issue when using lvm via iscsi on FreeNAS. What helped was:
1. Remove lvm storage via proxmox gui
2. Remove iscsi via proxmox gui, (The iscsi config on the system stays intact)
- Now the ctld messages should stop
3. Verify that the iscsi-connection is still active and the VG is there:

4. Add LVM via proxmox gui (it is shown as a local lvm)

This worked for me. So I think it must have something to do with a proxmox service.
Click to expand...

Remark to step 2: it's not necessary - just disable it (when node restarted - iscsi session will dissapear, so simply enable and disable it again).
I'm using FreeBSD with ctld as iscsi server.
 
I'm having the same issues. PVE 3.4 and FreeNAS 9.3

Tried everthing. Nothing seems to solve this issue. Help!!!!!!!!!!!!!!!!!!!!!!
 
Just a quick question. What hardware do you use?
According to the FreeNAS devs they only really support Intel hardware in that sense that tests are never performed on AMD hardware.
 
There are timeout values in this file:
Code: 
/usr/share/perl5/PVE/Storage/LunCmd/Istgt.pm
search for $timeout (line 67, 519) and you can see it's set to 10 seconds. Line 533 is set to 15 seconds.

that coincide directly with the messages.

The file references:
Current SIGHUP reload limitations (http://www.peach.ne.jp/archives/istgt/):

As far as hardware, I'm running FreeNAS on a Xeon (intel) and I'm using iSCSI and LVM on top of that.

This is starting to feel like the right direction now.
 
I updated the FreeNAS bug. Hopefully someone will respond, perhaps it's a limitation on the FreeNAS side, or maybe we can configure the Proxmox side to stop the spamming. With a cluster, I can't read anything in the FreeNAS /var/log/messages. It's spammed every few seconds.
 
Did anyone ever solve this? I'm experiencing it on a new FreeNAS 9.3 install with Proxmox 3. Any indication of which side it's on?
 
Between the 10-second log spam for iSCSI here and the 10-second log spam for NFS in this thread, I'm inclined to believe this is a Proxmox issue.
I'm connecting from Proxmox 4.0 to FreeNAS 9.3 with both iSCSI and NFS, so I see both of these messages in my logs:

Code: 
Jan 17 16:11:37 freenas ctld[11424]: 172.17.253.1: read: connection lost
Jan 17 16:11:37 freenas ctld[2091]: child process 11424 terminated with exit status 1
Jan 17 16:11:37 freenas mountd[2187]: export request succeeded from 172.17.253.1
Jan 17 16:11:47 freenas mountd[2187]: export request succeeded from 172.17.253.1
Jan 17 16:11:47 freenas ctld[11443]: 172.17.253.1: read: connection lost
Jan 17 16:11:47 freenas ctld[2091]: child process 11443 terminated with exit status 1
Jan 17 16:11:57 freenas mountd[2187]: export request succeeded from 172.17.253.1
Jan 17 16:11:57 freenas ctld[11447]: 172.17.253.1: read: connection lost
Jan 17 16:11:57 freenas ctld[2091]: child process 11447 terminated with exit status 1
 
Has anybody figured this out? This is still happening with the latest ProxMox VE 4.4 and FreeNAS 9.10.2-U1, and it renders my daily log emails completely useless. It definitely sounds like a similar issue to the NFS connection, with ProxMox initiating a new connection every 10 seconds just to check for availability. In fact, I just confirmed it by running service pvestatd stop on proxmox and the log spam immediately stopped, service pvestatd start, logspam starts again. This is definitely a ProxMox issue, caused by pvestatd.
 
pvestatd checks every 10 second if the storages are still available, and for iscsi and nfs, there seem to be no better check

e.g. for nfs we simply do:
showmount -e <ip>
to verify that the nfs service on the server still works
 
Ok, well whatever the iscsi equivalent of showmount is should gracefully close the connection when it's done, instead of just abandoning it to timeout, or force-closing it, whichever the case may be.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom