live migration fails on shared LVM

[Andreas Pflug]

I've setup LVM with lvmlockd and sanlock, backed by iscsi storage via multipath. AFAICS multipath can't be configured via GUI, so the actual backing of the volume group is unknown to PVE which should be fine since multipathd will handle everything.

storage.cfg looks like this:

lvm: santest
vgname santest
content images
shared 1

Running vms off that storage works fine, but live migration will fail because pve can't activate the volume on the target machine. Apparently, volumes are activated using "lvchange -aey" instead of using "lvchange -asy" to enable sharing temporarily.

Am I missing something?

 

[dcsapak]

Am I missing something?

the pve stack uses its own locking which does not rely on lvmlockd, so this is not necessary, downside is you can only use such a volume group within one pve cluster

 

[Andreas Pflug]

My storage is shared with non-PVE initiators as well. In addition, command line lvm actions won't lock against pve either. So PVE locking is mostly non-locking...

Activating the volume to migrate with share option for the period of migration, and locking it exclusively afterwards would fix this.

 

[dcsapak]

that is sadly a setup that is not really supported... but you can create a seperate vg just for your pve cluster, to not conflict with the other users...

 

[Andreas Pflug]

Actually, the VGs need to be shared... Some VMs will be migrated to PVE, with TB of data on the SAN.

 

[dcsapak]

i see the issue, but i guess we will not change that behaviour anytime soon...
you could patch the code for now for yourself (be aware that every update will overwrite this) and when you are finished migrating, do not use lvmlockd anymore

 

[speter]

How on earth should I configure shared VG without lvmlockd? I have successfully created shared vgs with lvmlockd, everything works except live migration between hosts. But when I change VGS not to be shared (in OS, proxmox config stays same), deconfigure lvmlockd - I have no possibility to have VG accessible from both cluster nodes... What am I missing?

 

[dcsapak]

How on earth should I configure shared VG without lvmlockd? I have successfully created shared vgs with lvmlockd, everything works except live migration between hosts. But when I change VGS not to be shared (in OS, proxmox config stays same), deconfigure lvmlockd - I have no possibility to have VG accessible from both cluster nodes... What am I missing?

can you post a bit more info on your setup and what are you trying to do and possibly error messages?

PVE uses it's own locking for LVM that does not rely on lvmlockd but that means you cannot use such a vg for other things concurrently

 

[speter]

I simply cannot create shared VG in OS without lockd. So the VG is active on one node only, and second node (OS) will not pick up the vg because it is already open by other node. I have set up same hostname in lvmlocal.conf, so now both nodes can see the VG, but I am not sure this is correct and safe config

 

[dcsapak]

for pve you don't have to create a 'shared' vg, just a normal one.

can you post the output of 'vgs' on both nodes?

EDIT: also on what underlying storage did you create the LVM ?

 

[speter]

root@pve1:~# vgs -ao+systemid
  VG          #PV #LV #SN Attr   VSize   VFree   System ID
vg1             1   3   0 wz--n- 930.68g 630.68g pve1
vg2             1  10   0 wz--n- 930.68g 120.68g pve1


root@pve2:~# vgs -ao+systemid
*empty*

When I create normal vg, it is visible only from one node. When i set up *the same* hostname in /etc/lvm/lvmlocal.conf on both nodes, then VG is shown on both nodes, and proxmox picks them normally. But I am not sure this is safe config.

VGS are on multipathed FC LUNs.

 

[dcsapak]

how did you create the volume groups? (the exact command would be interesting)

edit, also you should be able to remove the systemid from the vg with:

vgchange --systemid "" <VGNAME>