iscsi not coming up properly after host reboot

[Omaha2002]

We have hosts with multiple iSCSI datastores. After updating and rebooting a host, the host status shows a green check, but the iSCSI datastores display a grey question mark.

Running `systemctl restart pvestatd` resolves the issue and the iSCSI datastores become available again.

Before creating a cron job to automatically restart `pvestatd`, I would like to explore if there is another solution to this problem.

 

[bbgeek17]

Hello @Omaha2002
You should not have to restart pvestatd manually or via cron job. Although its a quick way to move with business, it does not really address your underlying issue.
Given that many people are using native PVE iSCSI successfully and there are currently no similar reports in the channel, chances are high that this is related to your environment.

The first thing to do is to carefully examine your log from reboot time : journalctl -b0

Providing information below might also be helpful:
- what storage system you are using
- what version of PVE you are using
- how did you configure your storage (cat /etc/pve/storage.cfg)
- what does each of the nodes see? (pvesm status) (preferably in "bad" state and "good" state)
- what does the OS of each of the node see? (iscsiadm -m node;iscsiadm -m session;lsscsi;lsblk;multipath -ll;pvs;vgs;lvs)
- are there any errors in the journal on the node? (journalctl -n 500)
- are there any errors that happen during the operation execution (journalctl -f)
- have you already rebooted all of the nodes?

There may be other questions.

I would also recommend repeating a reboot on at least one node to check if the problem is reproducible.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Omaha2002]

Thanks for the reply.

We have hosts in an IBM datacenter and are using the iSCSI storage from IBM, it's Netapp storage. Latest PVE, version 8.2.4 with kernel 6.8.8-1

Spoiler: /etc/pve/storage.cfg

dir: local
path /var/lib/vz
content backup,vztmpl,iso

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

iscsi: BNC_Prod_DIensten_B
portal 10.3.142.211
target iqn.1992-08.com.netapp:stfams0301
content none

lvm: BNC
vgname VG_BNC
content images,rootdir
saferemove 0
shared 1

esxi: ESX01
server 1X.XX.XX.XX.XX5
username root
content import
skip-cert-verification 1

pbs: PBS_XXXXXX
datastore Backup_pool
server 192.168.2.250
content backup
fingerprint 7a:bf:f4:a8:64:b4:XX:XX:XX:XX:XX:XX:XX:XX:f1:1e:a7
prune-backups keep-all=1
username IBMProxmox@pbs

iscsi: Ecommerce
portal 10.3.142.212
target iqn.1992-08.com.netapp:stfams0301
content images

lvm: BXC-Ecom
vgname VG-Ecom
base Ecommerce:0.0.4.scsi-3600a0XXXXXXXXX4d3f4d64326a466a
content images,rootdir
saferemove 0
shared 1

iscsi: BXC_Prod_Diensten_A
portal 10.3.142.215
target iqn.1992-08.com.netapp:stfams0301
content images

lvm: BXC_Prod_A
vgname VG_BNC_A
base BXC_Prod_Diensten_A:0.0.1.scsi-3600a09XXXXXXXX24d3f4d64326a4668
content images,rootdir
saferemove 0
shared 1

Need to check the bad status because all working now, will do that in the evening

Spoiler: pvesm status

Name Type Status Total Used Available %
BXC lvm active 7340027904 5894381568 1445646336 80.30%
BXC-Ecom lvm active 6291451904 4041015296 2250436608 64.23%
BXC_Prod_A lvm active 8388603904 1300238336 7088365568 15.50%
BXC_Prod_DIensten_B iscsi active 0 0 0 0.00%
BXC_Prod_Diensten_A iscsi active 0 0 0 0.00%
ESX01 esxi active 0 0 0 0.00%
Ecommerce iscsi active 0 0 0 0.00%
PBS_XXXXXXa pbs active 7665311104 4779210368 2886100736 62.35%
local dir active 98497780 11758764 81689468 11.94%
local-lvm lvmthin active 1792532480 0 1792532480 0.00%

Will have to reboot a node to check for your other questions, so far every time we rebooted a node it comes up green with grey questionsmark on the iSCSI storage and after a restart of pvestatd iscsi storage becomes available and works. As I stated above.