no automatic target login after port reinitialize

[RBretzel]

Hi,
I have the following challenge.
If I show the active session with this command "iscsiadm --mode session --print=1"

During regular operations my iSCSI connection looks like this.


When a port reinitialize is done by the target I have this state.



I have already made the following changes in iscsid.conf
node.startup = automatic
node.leading_login = Yes
node.session.timeo.replacement_timeout = 15

I had already tried the commands
iscsiadm -m node --op=update -n node.conn[0].startup -v automatic
iscsiadm -m node --op=update -n node.startup -v automatic

The solution is to provied a logout and login of the port or the tough method to restart iscsid

Can anyone tell me how I can solve this without manual intervention?
I am grateful for any solutions.

Regards
Rainer

 

[RBretzel]

Hi,

I also used these two commands, but they did not solve the problem.
#iscsiadm --mode node --target <IQN> --portal <IP-Adr.>:3260 -n discovery.sendtargets.use_discoveryd -v Yes
#iscsiadm --mode node --target <IQN> --portal <IP-Adr.>:3260 -n discovery.sendtargets.discoveryd_poll_inval -v 30

I would be pleased about any new ideas for solving the problem

best
Rainer

 

[fweber]

Hi, how long does a port reinitialization usually take?
Can you check the journal of the iscsid unit and find a time at which you triggered a port reinitialization? I suppose there will be some Kernel reported iSCSI connection [...] error at that time. Can you post these messages up until the port reinitialization is finished?
You can see the iscsid journal of the current boot with the following command:

journalctl -u iscsid -b

 

[RBretzel]

Hi Friedrich,
I have performed the reinitialization on one port and these messages.

Apr 30 14:18:58 de-prox02 iscsid[1413]: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Apr 30 14:19:00 de-prox02 iscsid[1413]: Received iferror -107: Transport endpoint is not connected.
Apr 30 14:19:00 de-prox02 iscsid[1413]: can't set operational parameter 18 for connection 1:0, retcode -107 (0)
Apr 30 14:19:00 de-prox02 iscsid[1413]: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (1)
Apr 30 14:19:02 de-prox02 iscsid[1413]: can't bind conn 1:0 to session 1, retcode -107 (115)
Apr 30 14:19:04 de-prox02 iscsid[1413]: can't bind conn 1:0 to session 1, retcode -107 (115)
Apr 30 14:19:06 de-prox02 iscsid[1413]: can't bind conn 1:0 to session 1, retcode -107 (115)
Apr 30 14:19:08 de-prox02 iscsid[1413]: can't bind conn 1:0 to session 1, retcode -107 (115)
Apr 30 14:19:10 de-prox02 iscsid[1413]: can't bind conn 1:0 to session 1, retcode -107 (115)

I hope this information is helpful.

Regards
Rainer

 

[bbgeek17]

Hi @RBretzel,

The core software components involved in this flow are iscsid and your vendor's iSCSI implementation. Given that Proxmox is using standard versions of iscsid that are widely deployed, this is more likely related to a bug or quirk in your vendor's iSCSI implementation. That said, we've seen and debugged iscsid issues in the past!

Based on your screenshot, this could be an issue specific to Datacore. Can you reach out to them? They should be able to help you with both components of the equation.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[fweber]

Hi, in addition to what @bbgeek17 wrote: There is a resolved open-iscsi upstream bug report [1] with a very similar problem description and error message. The patch that fixes the error (iscsid: stop connection for recovery if error is not timeout in iscsi_login_eh (#388)) is apparently included in open-iscsi 2.1.9 [2]. Debian Bookworm (and thus Proxmox VE 8) only ships 2.1.8, which does not include the fix. It might be worth a try to check if open-iscsi 2.1.9, or specifically the patch [3], fixes the issue you're seeing. The most straightforward way to test that might be to build your own patched open-iscsi 2.1.8 package with the fix [3] included. Note that we strongly recommend against running such custom packages on production systems, but trying it out on a test system would be helpful to see whether the patch indeed fixes the issue you're seeing.

[1] https://github.com/open-iscsi/open-iscsi/issues/387
[2] https://github.com/open-iscsi/open-iscsi/blob/4a45825fd1251c6cc7d343cc33ec7004c818f5b1/Changelog
[3] https://github.com/open-iscsi/open-iscsi/commit/9f2074568e6c39f85c9d948cb3b869f4fc774695.diff

 

[RBretzel]

Hi Friedrich,

thanks for the information about the open-iscsi version 2.1.9 and the included fix.
I have installed this version and the necessary additional packages for compilation in my test environment.
During the test it became clear to me that the reinitialization now works in principle but not reliably. The 100% function can be achieved by changing the parameter “node.session.initial_login_retry_max” in the iscsid.conf to the value 128.

Best regards,
Rainer Bretzel
DataCore Software GmbH

 

[bbgeek17]

Hi Rainer, thank you for doing the testing!

@fiona I definitely +1 including 2.1.9 into PVE8. It turns out we saw this bug in the wild once in a deployment that uses encrypted storage transports (i.e., iSCSI over TLS).

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[fweber]

Thanks for testing open-iscsi 2.1.9! Since the upstream fix [0] seemed straightforward enough, we have built a open-iscsi package with the fix applied (version 2.1.8-1.pve1) The package is provided in pvetest [2]. @RBretzel could you please check whether open-iscsi 2.1.8-1.pve1 (in combination with the configuration change you mentioned) fixes the issue for you?

[0] https://github.com/open-iscsi/open-iscsi/commit/9f2074568e6c39f85c9d948cb3b869f4fc774695.diff
[1] https://git.proxmox.com/?p=package-rebuilds.git;a=commit;h=a98b564d9a68d5be238544e449e114ac3f269a97
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_test_repo

 

[RBretzel]

Hello Friedrich,

as you wished, I have tested it with version 2.1.8-1.pve1.
My result of the test is that it sometimes works that the port logs in again. I also can't find a scheme or rule when it works and when it doesn't, it's a random one.
I would ask you to include the open-iscsi version 2.1.9 in one of the next releases or updates of your software.
Manual installation of the version 2.1.9 is a challenge for the standard administrator. Furthermore, it can also have a negative impression on administrators who are considering a migration.

Best regards,
Rainer Bretzel
DataCore Software GmbH

 

[fweber]

Hello Rainer,

Thanks for testing open-iscsi 2.1.8-1.pve1. If I understand correctly, it does not completely fix the issue in your tests, whereas open-iscsi 2.1.9 (plus increasing node.session.initial_login_retry_max) reliably fixes the issue. If so, this sounds like 2.1.9 includes a patch that improves the situation and which is currently missing in our 2.1.8-1.pve1. I had another look at the open-iscsi 2.1.9 Changelog [1] again, but did not see anything that sounds related. Can you try again with 2.1.8-1.pve1 (make sure to do so on a clean machine without a previous 2.1.9 install, and reboot after installing the open-iscsi package)? If the automatic login after port reinitialization still fails occasionally, could you provide the portion of the journal when it fails? You can generate a journal of the current boot with the following command:

journalctl -b

I would ask you to include the open-iscsi version 2.1.9 in one of the next releases or updates of your software.
Manual installation of the version 2.1.9 is a challenge for the standard administrator. Furthermore, it can also have a negative impression on administrators who are considering a migration.

Yes, I agree it would be good to ship an open-iscsi package that fixes the issue from this thread. However, to keep the impact on the rest of the PVE packages as small as possible, we would prefer to ship a modified open-iscsi 2.1.8 package with the necessarily patches applied, instead of shipping a new minor version open-iscsi 2.1.9. Hence, it would be great if you could provide the logs from above, so we can maybe identify the upstream patches that are currently still missing in our modified 2.1.8 package.

[1] https://github.com/open-iscsi/open-iscsi/blob/b38f272e81326eabf3f5a9e49c249bbf24409229/Changelog