VM locked in a node after upgrading to PVE 7.

dominix · Feb 7, 2025

hi everyone,
a very strange thing happened.
I got a cluster on 2 HPE DL830 with a fiber channel disk bay MSA family, originally in PVE 6
I carefully read about https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0
I migrated all VM from node 1 to node 2
upgraded node 1 to pve 7, rebooted node1, waited that it come back in line and rejoined the cluster.
then moved all VM from node2 to node1, all went fine, upgraded node2 to PVE7.
Now I got a cluster in VPE 7, that seems to work well, all VM are on node1, except I can not move back any VM to node2 because it issue en error every time I try.

Code:

root@vms-infomox-1:~# qm migrate 320 vms-infomox-2 --online
2025-02-07 09:51:32 starting migration of VM 320 to node 'vms-infomox-2' (192.168.25.121)
2025-02-07 09:51:32 starting VM 320 on remote node 'vms-infomox-2'
2025-02-07 09:51:35 [vms-infomox-2] can't activate LV '/dev/MSA/vm-320-disk-0':   Cannot activate LVs in VG MSA while PVs appear on duplicate devices.
2025-02-07 09:51:35 ERROR: online migrate failure - remote command failed with exit code 255
2025-02-07 09:51:35 aborting phase 2 - cleanup resources
2025-02-07 09:51:35 migrate_cancel
2025-02-07 09:51:37 ERROR: migration finished with problems (duration 00:00:06)
migration problems

what I see in the syslog

Code:

Feb  7 09:51:31 vms-infomox-1 qm[122837]: <root@pam> starting task UPID:vms-infomox-1:0001E00B:00176F1B:67A66443:qmigrate:320:root@pam:
Feb  7 09:51:34 vms-infomox-1 pmxcfs[1493]: [status] notice: received log
Feb  7 09:51:35 vms-infomox-1 pmxcfs[1493]: [status] notice: received log
Feb  7 09:51:37 vms-infomox-1 pmxcfs[1493]: [status] notice: received log
Feb  7 09:51:37 vms-infomox-1 pmxcfs[1493]: [status] notice: received log
Feb  7 09:51:37 vms-infomox-1 qm[122891]: migration problems
Feb  7 09:51:37 vms-infomox-1 qm[122837]: <root@pam> end task UPID:vms-infomox-1:0001E00B:00176F1B:67A66443:qmigrate:320:root@pam: migration problems

I'm stuck here. anybody could give me hint or direction to get out of this nightmare.
I don't got the duplicate device thing, as there not seems to be any duplicate.

dominix · Feb 7, 2025

may be this could be related to this difference on how volume are seen on different node but I dont known how to deal with

Code:

root@vms-infomox-2:~# pvs
  WARNING: Not using device /dev/sdd1 for PV uftjdL-8Kk1-26tB-9Aqp-nYLr-K1aV-IEfjN6.
  WARNING: Not using device /dev/sde1 for PV Aa6lLW-oK9c-UF2F-kQQd-ogUh-fDWK-flPv0d.
  WARNING: PV uftjdL-8Kk1-26tB-9Aqp-nYLr-K1aV-IEfjN6 prefers device /dev/sdb1 because device is used by LV.
  WARNING: PV Aa6lLW-oK9c-UF2F-kQQd-ogUh-fDWK-flPv0d prefers device /dev/sdc1 because device is used by LV.
  PV         VG  Fmt  Attr PSize    PFree
  /dev/sda3  pve lvm2 a--  <279.11g      0
  /dev/sdb1  MSA lvm2 a--    <2.73t  <2.63t
  /dev/sdc1  MSA lvm2 a--    <2.73t 704.01g
root@vms-infomox-2:~# lsblk
NAME                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                        8:0    0 279.4G  0 disk
├─sda1                     8:1    0     1M  0 part
├─sda2                     8:2    0   256M  0 part
└─sda3                     8:3    0 279.1G  0 part
  ├─pve-swap             253:0    0    10G  0 lvm  [SWAP]
  ├─pve-root             253:1    0    50G  0 lvm  /
  └─pve-data             253:2    0 219.1G  0 lvm  /var/lib/vz
sdb                        8:16   0   2.7T  0 disk
└─sdb1                     8:17   0   2.7T  0 part
  └─MSA-vm--338--disk--0 253:13   0   100G  0 lvm
sdc                        8:32   0   2.7T  0 disk
└─sdc1                     8:33   0   2.7T  0 part
  ├─MSA-vm--330--disk--0 253:4    0   200G  0 lvm
  ├─MSA-vm--147--disk--0 253:6    0    40G  0 lvm
  ├─MSA-vm--134--disk--0 253:7    0    15G  0 lvm
  ├─MSA-vm--336--disk--0 253:10   0   100G  0 lvm
  ├─MSA-vm--324--disk--0 253:14   0    50G  0 lvm
  ├─MSA-vm--602--disk--0 253:15   0    40G  0 lvm
  ├─MSA-vm--601--disk--0 253:16   0    40G  0 lvm
  ├─MSA-vm--346--disk--0 253:17   0    80G  0 lvm
  ├─MSA-vm--171--disk--0 253:18   0   100G  0 lvm
  ├─MSA-vm--110--disk--0 253:19   0    40G  0 lvm
  └─MSA-vm--100--disk--0 253:20   0    50G  0 lvm
sdd                        8:48   0   2.7T  0 disk
└─sdd1                     8:49   0   2.7T  0 part
sde                        8:64   0   2.7T  0 disk
└─sde1                     8:65   0   2.7T  0 part


root@vms-infomox-1:~# pvs
  PV                                                  VG  Fmt  Attr PSize    PFree
  /dev/mapper/3600c0ff0001be7bddcd8e25b01000000-part1 MSA lvm2 a--    <2.73t 704.01g
  /dev/mapper/3600c0ff0001be801b9d8e25b01000000-part1 MSA lvm2 a--    <2.73t  <2.63t
  /dev/sda3                                           pve lvm2 a--  <279.11g      0
root@vms-infomox-1:~# lsblk
NAME                                        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                           8:0    0 279.4G  0 disk
├─sda1                                        8:1    0     1M  0 part
├─sda2                                        8:2    0   256M  0 part
└─sda3                                        8:3    0 279.1G  0 part
  ├─pve-swap                                253:0    0    10G  0 lvm   [SWAP]
  ├─pve-root                                253:1    0    50G  0 lvm   /
  └─pve-data                                253:2    0 219.1G  0 lvm   /var/lib/vz
sdb                                           8:16   0   2.7T  0 disk
├─sdb1                                        8:17   0   2.7T  0 part
└─3600c0ff0001be801b9d8e25b01000000         253:3    0   2.7T  0 mpath
  └─3600c0ff0001be801b9d8e25b01000000-part1 253:5    0   2.7T  0 part
    └─MSA-vm--338--disk--0                  253:17   0   100G  0 lvm
sdc                                           8:32   0   2.7T  0 disk
├─sdc1                                        8:33   0   2.7T  0 part
└─3600c0ff0001be7bddcd8e25b01000000         253:4    0   2.7T  0 mpath
  └─3600c0ff0001be7bddcd8e25b01000000-part1 253:6    0   2.7T  0 part
    ├─MSA-vm--331--disk--0                  253:7    0   200G  0 lvm
    ├─MSA-vm--330--disk--0                  253:8    0   200G  0 lvm
    ├─MSA-vm--142--disk--0                  253:9    0   332G  0 lvm
    ├─MSA-vm--147--disk--0                  253:10   0    40G  0 lvm
    ├─MSA-vm--134--disk--0                  253:11   0    15G  0 lvm
    ├─MSA-vm--311--disk--0                  253:12   0   200G  0 lvm
    ├─MSA-vm--335--disk--0                  253:13   0   250G  0 lvm
    ├─MSA-vm--336--disk--0                  253:14   0   100G  0 lvm
    ├─MSA-vm--337--disk--0                  253:15   0    50G  0 lvm
    ├─MSA-vm--337--disk--1                  253:16   0   200G  0 lvm
    ├─MSA-vm--324--disk--0                  253:18   0    50G  0 lvm
    ├─MSA-vm--602--disk--0                  253:19   0    40G  0 lvm
    ├─MSA-vm--601--disk--0                  253:20   0    40G  0 lvm
    ├─MSA-vm--346--disk--0                  253:21   0    80G  0 lvm
    ├─MSA-vm--171--disk--0                  253:22   0   100G  0 lvm
    ├─MSA-vm--110--disk--0                  253:23   0    40G  0 lvm
    ├─MSA-vm--100--disk--0                  253:24   0    50G  0 lvm
    └─MSA-vm--320--disk--0                  253:25   0   100G  0 lvm

bbgeek17 · Feb 7, 2025

One of your nodes is properly configured to use Multipath, while the other is not.
How it happened is up to you to find out.

The solution is to install/configure Multipath.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

dominix · Feb 10, 2025

thanks bbgeek17.

I'm trying to understand what is missing. setup is the same on both node. however I got on node2 this specific error

Code:

systemctl --failed

  UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
● lvm2-pvscan@8:49.service loaded failed failed LVM event activation on device 8:49
● lvm2-pvscan@8:65.service loaded failed failed LVM event activation on device 8:65

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed.

Does that sound to anyone ?
I don't know what to search from now on.

dominix · Feb 11, 2025

In desperation I edited every possible option in /etc/multipath.conf and ended up with a working configuration.
not en easy one. My guess is that multipath combined with LVM need really a good article on the technique to make a clear understanding on how it works.

Search

Search

VM locked in a node after upgrading to PVE 7.

dominix

Renowned Member

dominix

Renowned Member

bbgeek17

Distinguished Member

dominix

Renowned Member

dominix

Renowned Member

We value your privacy