proxmox cluster - LVM issue

  • Thread starter Thread starter raven007
  • Start date Start date
R

raven007

Guest
Hell.o

The setup:
- 5 nodes (proxmox pve 1.9)
- 1 dual-head (multipath) HBA - per node
- SAN (HP P2000 G3)

Deployment of one VM (standard scenario):
- new volume @ SAN
- rescan/udev
- pvcreate (cli/ssh on node) on top of the device mapper (linux kernel) - DM contains 4 block-devices
- vgcreate on top of (above) pv
- add to PVE (webinterface/pvesm CLI)
- install VM

The problem:
- some PVs/VGs are only available on the master node
- this means that the migration part will fail on the target-nodes that don't "know" about the storage pv/vg used for the migrated VM

Example (legend : node128 - master)
node128:~# pvdisplay | grep back
VG Name back
node128:~#
node128:~# vgdisplay | grep back
VG Name back

node130:~# pvdisplay | grep back
node130:~#
node130:~# vgdisplay | grep back
node130:~#

node128:~# pvesm list | grep back
back lvm 0 1 877977600 877658112 100%
node130:~# pvesm list | grep back
back lvm 0 0 0 0 100%

node128:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : 10.11.1.128 M A 1 day 18:29 2.35 44% 2%
2 : 10.11.1.130 N S 8 days 10:50 2.12 15% 2%
3 : 10.11.1.132 N S 8 days 10:37 1.29 81% 4%
5 : 10.11.1.129 N S 8 days 10:44 1.71 48% 1%
6 : 10.11.1.135 N S 8 days 05:28 1.00 3% 1%
7 : 10.11.1.136 N S 8 days 10:31 1.06 3% 1%
8 : 10.11.1.131 N S 8 days 10:39 1.00 50% 3%
node130:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : 10.11.1.128 M A 1 day 18:29 2.29 44% 2%
2 : 10.11.1.130 N S 8 days 10:50 2.10 15% 2%
3 : 10.11.1.132 N S 8 days 10:37 1.33 81% 4%
5 : 10.11.1.129 N S 8 days 10:44 1.76 48% 1%
6 : 10.11.1.135 N S 8 days 05:29 1.00 3% 1%
7 : 10.11.1.136 N S 8 days 10:31 1.05 3% 1%
8 : 10.11.1.131 N S 8 days 10:39 1.07 50% 3%
Click to expand...

Is there a way to "sync" all PVs/VGs from master to the slave nodes so that "all storage" is visible on all nodes (not only on master), thus making the migration process a reality?

Thanks a bunch.
 
raven007 said:
Is there a way to "sync" all PVs/VGs from master to the slave nodes so that "all storage" is visible on all nodes (not only on master), thus making the migration process a reality?
Click to expand...

Well, 'pvesm list' should re-read all data. Is the SAN volume active on all nodes?
 
tried that too, and more:
- pvscan
- vgscan
- udev restart
- triggering rescan of the SCSI bus
- restarted multipath(d) (even multipath -F)

no go...
 
100% sure (double checked the LUN maskings).

by the way, one important/interesting piece of info is that the volume that's not "seen" on the other members of the proxmox cluster is not "constant".
to be more clear, in the above example, node130 does not "see" the "back" SAN volume. but on other nodes "back" volume is accessible, but other(s) are not.

basically, what the problem is that I can migrate VMs only between a specific pair(s) of nodes (the ones that do have the PVs/VGs used by that VM).

thanks you.
 
Please check if you really see the same data on all nodes. I guess you can use

# md5sum /dev/XYZ

to compare.
 
I have found a "workaround" - steps :
- add LUN @ SAN
- trigger udev rescan (udevadm trigger)
- trigger scsi/fc bus rescan (/sys/class/fc_host ; /sys/class/scsi_host)
- pvscan/vgscan

That's it. Enjoy.
 
You must log in or register to reply here.
Top Bottom