Only 1 node of 3 node cluster can see LVM Group data

[greenvomit8]

Hi
We are looking at replacing our Hyper-V structure with Proxmox. Hence, I'm a newbie to both proxmox and Linux but loving my mac and am hoping there is something simple that i have missed.

I'm setting up a test environment with 3 nodes (proxmox1, promox2, proxmox3) configured as cluster.
We are using a SAN.
I've setup LVM Groups with Network Backing as per the wiki doc.
Proxmox 1 can see the ISCSI target and the LVM group. Including the contents of these.
Proxmox 2 and 3 can see the same ISCSI target and LVM group but cannot see the contents.

I can create VMs on Proxmox1 but cannot do the same on Proxmox2 or Proxmox3.
I've migrated 2 VMs to proxmox 2 but cannot start the VMs nor migrate them back.

Error on VM startup from proxmox2:
TASK ERROR: can't activate LV '/dev/proxmox_test_LVM/vm-101-disk-1': Skipping volume group proxmox_test_LVM

Error trying to migrate vm from proxmox2 to proxmox1:
Volume group "proxmox_test_LVM" not found
Jul 31 15:10:09 ERROR: migration aborted (duration 00:00:00): can't activate LV '/dev/proxmox_test_LVM/vm-101-disk-1': Skipping volume group proxmox_test_LVM
TASK ERROR: migration aborted

When I look at the status of the LVM group via proxmos1 it is Active
When I look at the status of the same LVM group via proxmox2 it is not active.

Can someone please steer me in the right direction so I can tell my manager how good proxmox is so we can dump hyper-v? I've inserted a screen capture of the Datacenter Storage items.
Thank you

Lorenzo

 

[dietmar]

Error on VM startup from proxmox2:
TASK ERROR: can't activate LV '/dev/proxmox_test_LVM/vm-101-disk-1': Skipping volume group proxmox_test_LVM

What is the output of

# vgs

 

[greenvomit8]

Hi Dietmar
Thank you for the quick reply, here is the output from the 3 machines.
For some reason proxmox3 can now see the LVM as active. Perhaps because I rebuilt it after I got Apache errors but Proxmox2 still cannot.

root@proxmox1:~# vgs
VG #PV #LV #SN Attr VSize VFree
proxmox_test_LVM 1 3 0 wz--n- 931.32g 821.32g
pve 1 3 0 wz--n- 167.18g 16.00g

root@proxmox2:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 465.26g 15.99g

root@proxmox3:~# vgs
VG #PV #LV #SN Attr VSize VFree
proxmox_test_LVM 1 3 0 wz--n- 931.32g 821.32g
pve 1 3 0 wz--n- 74.03g 9.25g

 

[dietmar]

And what is the output of (on proxmox2):

# pvesm status

 

[greenvomit8]

Hi Dietmar
Here is the output as requested:

root@proxmox2:~# pvesm status
Promise_DA0 iscsi 1 0 0 0 100.00%
Promise_NFS nfs 1 937723808 7591696 920605392 1.32%
local dir 1 360483468 199268 360284200 0.56%
proxmox_test_LVM lvm 0 0 0 0 100.00%

Thank you

Lorenzo

 

[dietmar]

Can it see the iscsi device?

# pvs

also try to rescan

# vgscan

 

[greenvomit8]

Hi Dietmar
It should be able to.
When I check from the SAN (10.0.10.1), Proxmox2 has an active iSCSI session with the SAN and Proxmox2 can ping the SAN iSCSI target. Here are the requested outputs:
root@proxmox2:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 pve lvm2 a-- 465.26g 15.99g
root@proxmox2:~# vgscan
Reading all physical volumes. This may take a while...
Found volume group "pve" using metadata type lvm2
root@proxmox2:~# ping 10.0.10.1
PING 10.0.10.1 (10.0.10.1) 56(84) bytes of data.
64 bytes from 10.0.10.1: icmp_req=1 ttl=64 time=0.101 ms.
64 bytes from 10.0.10.1: icmp_req=2 ttl=64 time=0.180 ms
Attached is a screen capture for proxmox2 iscsi session status

 

[greenvomit8]

Hi Dietmar
The 2 vms on proxmox2 can be trashed if needed as this is only a test environment for the moment.
Is there a quick workaround such as removing and readding Proxmox2 to the cluster?
Will I be able to delete these 2 vms from LVM?
Thanks

Lorenzo

 

[greenvomit8]

ok i think i know what caused the issue. proxmox2 was removed from the cluster and then re-added after a rebuild.
I subsequently added another host called proxmox4. I added the node to the same cluster, removed it and then readded it to the cluster using the -force switch because it would not add to the cluster without the -force switch. Proxmox4 is now having exactly the same issue that proxmox2 is having.
I think the same thing happened with proxmox2. Unless I'm missing something obvious (which I'll admit is most likely the case as i am new to proxmox) it seems to be a flaw that cluster nodes cannot be removed and readded back to the cluster.

 

[kenp]

I am having a similar issue where my VMs (CentOS) cannot see the lvm groups. I don't have a proxmox cluster though. Just a single instance of proxmox VE. The proxmox host can see the vg's and lv's but not the VM's.