iscsi LVM fails after boot of the host

[rgproxmox1]

Hi,

I have the following scenario:

I have a cluster (non-HA) that has an iSCSI LVM that contains VMs. One of the Proxmox hosts had to be reinstalled due to disk errors. After I reinstalled it (3.2.2/82599a65 version), I had it re-join the domain with the "pvecm add <existing cluster member IP>" command. The node seemed to join successfully and I can see the iSCSI LVM mentioned in the storage for the node that rejoined the cluster. The problem is that whenever I restart the node I see the error:

No volume groups found
No volume groups found
No volume groups found
Volume group "nas-iscsi-vg" not found
TASK ERROR: can't activate LV '/dev/nas-iscsi-vg/vm-112-disk-1': Skipping volume group nas-iscsi-vg


The weird thing is that after a min or so if I ask to start the VM that failed, it doesn't have a problem. It's just at boot time.
If I examine the storage, it can see the LVM, it can detect correctly the available space, etc.
I've compared as many settings as I can between the other nodes and this one (vgscan, lvscan, /etc/pve/storage.cfg) and I can't find out what might be the issue

Any ideas?

Thanks in advance

 

[dswartz]

Sounds like the iscsi initiator is not getting the LUN activated in time?

 

[rgproxmox1]

After the boot of the Proxmox host, if I execute "lvscan" all the LVMs are marked as "inactive". I can click on the web icon for the specific VM I want to start and start it. After that, lvscan shows the VM in question (at the moment I'm only playing with one VM, given this behavior) as "ACTIVE". When does Proxmox try to activate the LVM? When the VM needs to be started? I'd imagine Proxmox tries to start the VM at boot time of the host at the very end of the boot sequence, right?

 

[rgproxmox1]

That's what you'd think. I can play putting "artificial" delays, but I don't know which file is responsible for "waking up" the LVMs at boot time

 

[dswartz]

Dunno, sorry. All I can say is that I've seen other distros where iscsi LUNs are found fairly late in the game. What you are describing sounds like that, but hard to say for sure.

 

[rgproxmox1]

Comparing with another cluster where things are working correctly, I noticed that the storage.cfg looks different. In the "good" cluster, I have these entries related to iscsi/lvm:

iscsi: nas-iscsi
target iqn.2000-01.net.psisoft.prodnas2:mxiscsivms2
portal 10.87.83.10
content none

lvm: nas-iscsi-lvm
vgname nas-iscsi-vg
base nas-iscsi:0.0.0.scsi-3600144f090958569000053349ee00004
shared
content images


in the "sick" cluster, I have similar entries, but I don't have a "base nas-iscsi:0.0...." entry

 

[rgproxmox1]

I have a workaround for the problem. I went ahead and modified /etc/init.d/pve-manager and just before the line that has: "pvesh --nooutput create /nodes/localhost/startall", I added "sleep 30". With this arbitrary delay, it works (at least 2 out of 2 boots, so far).

Maybe someone can have the background of why is it not working w/o that delay

 

[rgproxmox1]

I found out that the other 2 Proxmox hosts in the cluster had the same issue and the same workaround worked for both. The other cluster I have doesn't have the same issues.

 

[rgproxmox1]

I'm kind of concerned with the workaround, since it implies that if I upgrade a package that contains the init script pve-manager, I'll have to make the same change.

 

[mylesw]

I have the identical problem with PM 3.1 on a cluster with 3 nodes. Using FreeNAS iSCSI as my server, any storage that has PM as the initiator, fails on restart of the hypervisor. However it is definitely a timing issue - like you, if I manually umount and mount -a the devices as listed in /etc/fstab (which includes the iSCSI mount to /dev/sd??), it comes up fine. I can only assume that the iSCSI software has not completed at boot time. Whether this will mean that /etc/fstab gets to mount is unknown. I'm thinking that I might need to extend your workaround to force remounting the iSCSI volume as well, but we'll see....