Cannot reconnect to iSCSI

[jasonoldy]

Hi all

Having an issue with Proxmox 6.4-4 connecting to iSCSI storage

It has been working beautifully for about a year or more, and one day I found proxmox running with a question mark over this storage. Cause could have been a reboot or power outage, but can't say for sure as everything auto powers on after loss.

To test the storage is ok, I have fired up Ubuntu Live and tested a connection the same target - works fine, I can see all the disk images

Started Proxmox again, removed the iSCSI storage, and tried to add new - results below



First attempt results in "Timed Out", 2nd attempt results in "cfs-lock". The 2nd attempt does add the storage anyway, but again the question mark

I'm not sure how to force Proxmox to connect

I'm also very anxious that any other steps I take will put the disks at risk

 

[bbgeek17]

You are probably getting a timeout for some reason. Check your MTUs.
Try "pvesm scan iscsi <HOST[ORT]>"
Check "journalctl -n 500" for iscsi related errors.
Try this command and see if returns successfully in reasonable time: /usr/bin/iscsiadm --mode discovery --type sendtargets --portal IP:3260

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jasonoldy]

Hi bbgeek17

This is the NIC responsible for iSCSI



root@proxmox2:~# pvesm scan iscsi 192.168.189.2
iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.189.2,3260,1 iqn.2022518.tnas:storage.ce7l1e]
iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.188.64,3260,1 iqn.2022518.tnas:storage.ce7l1e]
iqn.2022518.tnas:storage.ce7l1e 192.168.189.2:3260,192.168.188.64:3260,169.254.252.146:3260



May 05 21:16:00 proxmox2 systemd[1]: Starting Proxmox VE replication runner...
May 05 21:16:00 proxmox2 systemd-logind[1015]: Session 3 logged out. Waiting for processes to exit.
May 05 21:16:00 proxmox2 systemd[1]: session-3.scope: Succeeded.
May 05 21:16:00 proxmox2 systemd-logind[1015]: Removed session 3.
May 05 21:16:00 proxmox2 pvedaemon[1402]: <root@pam> end task UPIDroxmox2:00004559:000B7625:64556286:vncshell::root@pam: OK
May 05 21:16:01 proxmox2 systemd[1]: pvesr.service: Succeeded.
May 05 21:16:01 proxmox2 systemd[1]: Started Proxmox VE replication runner.
May 05 21:16:10 proxmox2 systemd[1]: Stopping User Manager for UID 0...
May 05 21:16:10 proxmox2 systemd[17772]: Stopped target Default.
May 05 21:16:10 proxmox2 systemd[17772]: Stopped target Basic System.
May 05 21:16:10 proxmox2 systemd[17772]: Stopped target Sockets.
May 05 21:16:10 proxmox2 systemd[17772]: gpg-agent-extra.socket: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
May 05 21:16:10 proxmox2 systemd[17772]: gpg-agent-ssh.socket: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Closed GnuPG cryptographic agent (ssh-agent emulation).
May 05 21:16:10 proxmox2 systemd[17772]: gpg-agent.socket: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Closed GnuPG cryptographic agent and passphrase cache.
May 05 21:16:10 proxmox2 systemd[17772]: dirmngr.socket: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Closed GnuPG network certificate management daemon.
May 05 21:16:10 proxmox2 systemd[17772]: gpg-agent-browser.socket: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
May 05 21:16:10 proxmox2 systemd[17772]: Stopped target Paths.
May 05 21:16:10 proxmox2 systemd[17772]: Reached target Shutdown.
May 05 21:16:10 proxmox2 systemd[17772]: systemd-exit.service: Succeeded.
May 05 21:16:10 proxmox2 systemd[17772]: Started Exit the Session.
May 05 21:16:10 proxmox2 systemd[17772]: Reached target Exit the Session.
May 05 21:16:10 proxmox2 systemd[17772]: Stopped target Timers.
May 05 21:16:10 proxmox2 systemd[1]: user@0.service: Succeeded.
May 05 21:16:10 proxmox2 systemd[1]: Stopped User Manager for UID 0.
May 05 21:16:10 proxmox2 systemd[1]: Stopping User Runtime Directory /run/user/0...
May 05 21:16:10 proxmox2 systemd[1]: run-user-0.mount: Succeeded.
May 05 21:16:10 proxmox2 systemd[1]: user-runtime-dir@0.service: Succeeded.
May 05 21:16:10 proxmox2 systemd[1]: Stopped User Runtime Directory /run/user/0.
May 05 21:16:10 proxmox2 systemd[1]: Removed slice User Slice of UID 0.
May 05 21:16:12 proxmox2 pvestatd[1375]: command '/usr/bin/iscsiadm --mode node --targetname iqn.2022518.tnas:storage.ce7l1e --login' failed: exit code 8
May 05 21:16:12 proxmox2 pvestatd[1375]: status update time (120.351 seconds)
May 05 21:16:13 proxmox2 iscsid[1160]: Connection-1:0 to [target: iqn.2022518.tnas:storage.ce7l1e, portal: 169.254.252.146,3260] through [iface: default] is shutdown.
May 05 21:17:00 proxmox2 systemd[1]: Starting Proxmox VE replication runner...
May 05 21:17:01 proxmox2 systemd[1]: pvesr.service: Succeeded.
May 05 21:17:01 proxmox2 systemd[1]: Started Proxmox VE replication runner.
May 05 21:17:01 proxmox2 CRON[18816]: pam_unix(cron:session): session opened for user root by (uid=0)
May 05 21:17:01 proxmox2 CRON[18817]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
May 05 21:17:01 proxmox2 CRON[18816]: pam_unix(cron:session): session closed for user root
May 05 21:17:29 proxmox2 pvedaemon[1400]: <root@pam> starting task UPIDroxmox2:000049C3:000C2C8E:64556459:vncshell::root@pam:
May 05 21:17:29 proxmox2 pvedaemon[18883]: starting termproxy UPIDroxmox2:000049C3:000C2C8E:64556459:vncshell::root@pam:
May 05 21:17:29 proxmox2 pvedaemon[1401]: <root@pam> successful auth for user 'root@pam'
May 05 21:17:29 proxmox2 login[18888]: pam_unix(login:session): session opened for user root by root(uid=0)
May 05 21:17:29 proxmox2 systemd[1]: Created slice User Slice of UID 0.
May 05 21:17:29 proxmox2 systemd[1]: Starting User Runtime Directory /run/user/0...
May 05 21:17:29 proxmox2 systemd-logind[1015]: New session 6 of user root.
May 05 21:17:29 proxmox2 systemd[1]: Started User Runtime Directory /run/user/0.
May 05 21:17:29 proxmox2 systemd[1]: Starting User Manager for UID 0...
May 05 21:17:29 proxmox2 systemd[18900]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
May 05 21:17:29 proxmox2 systemd[18900]: Reached target Paths.
May 05 21:17:29 proxmox2 systemd[18900]: Listening on GnuPG network certificate management daemon.
May 05 21:17:29 proxmox2 systemd[18900]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
May 05 21:17:29 proxmox2 systemd[18900]: Listening on GnuPG cryptographic agent and passphrase cache.
May 05 21:17:29 proxmox2 systemd[18900]: Reached target Timers.
May 05 21:17:29 proxmox2 systemd[18900]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
May 05 21:17:29 proxmox2 systemd[18900]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
May 05 21:17:29 proxmox2 systemd[18900]: Reached target Sockets.
May 05 21:17:29 proxmox2 systemd[18900]: Reached target Basic System.
May 05 21:17:29 proxmox2 systemd[18900]: Reached target Default.
May 05 21:17:29 proxmox2 systemd[18900]: Startup finished in 43ms.
May 05 21:17:29 proxmox2 systemd[1]: Started User Manager for UID 0.
May 05 21:17:29 proxmox2 systemd[1]: Started Session 6 of user root.
May 05 21:17:29 proxmox2 login[18913]: ROOT LOGIN on '/dev/pts/0'



root@proxmox2:~# /usr/bin/iscsiadm --mode discovery --type sendtargets --portal 192.168.189.2:3260
iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.189.2,3260,1 iqn.2022518.tnas:storage.ce7l1e]
iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.188.64,3260,1 iqn.2022518.tnas:storage.ce7l1e]
192.168.189.2:3260,1 iqn.2022518.tnas:storage.ce7l1e
192.168.188.64:3260,1 iqn.2022518.tnas:storage.ce7l1e
169.254.252.146:3260,1 iqn.2022518.tnas:storage.ce7l1e



I might need help deciphering all this

 

[bbgeek17]

Is your /etc/iscsi/ directory OK? perhaps it suffered in that power outage that you suspect took your host out?

It should look something like this:

root@proxmox-1:~# ls -al /etc/iscsi/
total 40
drwxr-xr-x   6 root root  4096 Mar 22  2022 .
drwxr-xr-x 104 root root  4096 May  3 06:40 ..
drw-------   2 root root  4096 Nov 16  2020 ifaces
-rw-------   1 root root   349 Nov 16  2020 initiatorname.iscsi
-rw-------   1 root root 11913 Sep 29  2016 iscsid.conf
drw-------   3 root root  4096 May  5 11:42 nodes
drw-------   3 root root  4096 Nov 16  2020 send_targets
drw-------   2 root root  4096 May  5 11:42 static

If /etc/iscsi is not healthy then chances are there are other issues on the system too.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

https://forum.proxmox.com/threads/lvm-luns-suddenly-missing.111070/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jasonoldy]

https://forum.proxmox.com/threads/lvm-luns-suddenly-missing.111070/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Good find, thank you

Default file was 0B, so it is now deleted.

Removed and added the iSCSI storage again - no errors - no question marks

Tried to start a VM and now there is an error for 'iscsilvm' does not exist

I can see a 15TB LVM as 'sdh' but can't recall where in the gui I would have named a LVM as 'iscsilvm'

 

[jasonoldy]

Ignore me, I remembered. Storage > Add LVM

Its not a case of just adding iSCSI storage and expecting vm-disks to appear

Time to invest in a backup routine! I don't want to lose sleep like that again

Thanks for your help