PVE Upgrade 6.4 to 7.1 destroys LVM (thin provision)

[reinsle]

Hi Forum,

The System was a Installation of Proxmox PVE 6.x on 4 * 4 TB Raid 5. The System was updated to last 6.4 Version of Proxmox.

During the Upgrade to 7.1 i was unable to reach the VMs. On Console i found (See attached Screenshot)

pvs:

--- >B ---
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 10.91t 0
/dev/sdb pve lvm2 a-- <3.64t <3.61t
--- >B ---

/dev/sdb is a additional harddisk to provide more free space

vgs:

--- >B ---
VG #PV #LV #SN Attr VSize VFree
pve 2 46 0 wz--n- 14.55t <3.61t
--- >B ---

lvs -a

--- >B ---
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi---tz-- <10.77t
data_meta0 pve -wi-a----- 16.00g
data_meta1 pve -wi-a----- 16.00g
data_meta2 pve -wi-a----- 16.00g
[data_tdata] pve Twi-a----- <10.77t
[data_tmeta] pve ewi-a----- 16.00g
[lvol1_pmspare] pve ewi------- 16.00g
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 8.00g
vm-111-disk-0 pve Vwi---tz-- 50.00g data
vm-111-disk-1 pve Vwi---tz-- 1.00t data
vm-112-disk-0 pve Vwi---tz-- 500.00g data
vm-112-disk-1 pve Vwi---tz-- 1.00t data
vm-113-disk-0 pve Vwi---tz-- 100.00g data
vm-113-disk-1 pve Vwi---tz-- 500.00g data
vm-113-disk-2 pve Vwi---tz-- 500.00g data
vm-113-disk-3 pve Vwi---tz-- 500.00g data
vm-113-disk-4 pve Vwi---tz-- 500.00g data
vm-114-disk-0 pve Vwi---tz-- 50.00g data
vm-115-disk-0 pve Vwi---tz-- 50.00g data
vm-115-disk-1 pve Vwi---tz-- 100.00g data
vm-118-disk-0 pve Vwi---tz-- 50.00g data
vm-121-disk-0 pve Vwi---tz-- 50.00g data
vm-132-disk-0 pve Vwi---tz-- 50.00g data
vm-134-disk-0 pve Vwi---tz-- 100.00g data
vm-136-disk-0 pve Vwi---tz-- 100.00g data
vm-137-disk-0 pve Vwi---tz-- 100.00g data
vm-138-disk-0 pve Vwi---tz-- 200.00g data
vm-141-disk-0 pve Vwi---tz-- 100.00g data
vm-142-disk-0 pve Vwi---tz-- 250.00g data
vm-151-disk-0 pve Vwi---tz-- 75.00g data
vm-161-disk-0 pve Vwi---tz-- 100.00g data
vm-161-disk-1 pve Vwi---tz-- 250.00g data
vm-162-disk-0 pve Vwi---tz-- 50.00g data
vm-172-disk-0 pve Vwi---tz-- 100.00g data
vm-181-disk-0 pve Vwi---tz-- 50.00g data
vm-182-disk-0 pve Vwi---tz-- 50.00g data
vm-183-disk-0 pve Vwi---tz-- 50.00g data
vm-184-disk-0 pve Vwi---tz-- 50.00g data
vm-185-disk-0 pve Vwi---tz-- 50.00g data
vm-186-disk-0 pve Vwi---tz-- 50.00g data
vm-2000-disk-0 pve Vwi---tz-- 50.00g data
vm-2001-disk-0 pve Vwi---tz-- 50.00g data
vm-2011-disk-0 pve Vwi---tz-- 50.00g data
vm-2012-disk-0 pve Vwi---tz-- 50.00g data
vm-2013-disk-0 pve Vwi---tz-- 100.00g data
vm-2021-disk-0 pve Vwi---tz-- 50.00g data
vm-2022-disk-0 pve Vwi---tz-- 50.00g data
vm-471-disk-0 pve Vwi---tz-- 150.00g data
--- >B ---

while executing vgchange -ay i got following messages:

--->B ---
Activation of logical volume pve/data is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-136-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-161-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-161-disk-1 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-132-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-114-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-113-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-113-disk-1 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-113-disk-2 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-113-disk-3 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-111-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-111-disk-1 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-137-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-471-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-134-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-162-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-115-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-181-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-182-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-183-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-184-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-185-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-186-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-112-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-118-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-112-disk-1 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-141-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-142-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-151-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-121-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-113-disk-4 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-172-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-115-disk-1 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2012-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2022-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2011-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2021-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2013-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2000-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-2001-disk-0 is prohibited while logical volume pve/data_tmeta is active.
Activation of logical volume pve/vm-138-disk-0 is prohibited while logical volume pve/data_tmeta is active.
5 logical volume(s) in volume group "pve" now active
--- >B ---

Same errors on activating virtual machines:

--- >B ---
root@horst:~# qm start 112
activating LV 'pve/data' failed: Activation of logical volume pve/data is prohibited while logical volume pve/data_tmeta is active.
--- >B ---

I don't knew if the problem occures as result of updating proxmox to 7.1 or has other source.

Any hints to resolve this?

Thanks a lot

 

[Pride1922]

Did you ever find a solution? I have the same problem.

 

[TechQI]

Same, too

 

[reinsle]

No, sorry, no one replied and right now i don't find a solution.

 

[Kostya Bezsmertnyi]

Same problem. I can fix it with commands below, but after node reboot the problem returns.

lvchange -a n pve/data_tmeta
lvchange -a n pve/data_tdata
vgchange -a y pve

 

[fiona]

Hi,
could you try the workaround mentioned here? Although it might not be the same issue, because of the device mapper warnings (the original reports didn't have those IIRC). So if the workaround doesn't help or if you want to further debug the issue, please follow the instructions here and provide the log and the output of pveversion -v.

 

[edisonlee55]

I solved this issue by doing lvconvert --repair pve/data (Remember to backup your volumes first).
After node reboot, the problem doesn't return anymore.

# Disable the storage pools which is using the volumes
pvesm set local-lvm --disable 1

# Deactivate the volumes which we want to repair
lvchange -an pve/data

# Repair the volumes
lvconvert --repair pve/data

# Reactivate the volumes
lvchange -ay pve/data

# Re-enable the storage pools
pvesm set local-lvm --disable 0

Worth mentioning, some warnings do appear after repair, but the volumes on my side still work fine (Again, remember to backup your volumes first):

lvconvert --repair pve/data
  Volume groups "pve" has insufficient free space (128 extents): 4048 required.
  WARNING: LV pve/data_meta0 holds a backup of the unrepaired metadata. Use lvremove when no longer required.

References:
https://forum.proxmox.com/threads/after-power-outage-pve-7-1-pve-data-failed.101432/
https://forum.proxmox.com/threads/corrupted-drive-how-to-repair.93907/
https://pve.proxmox.com/wiki/Storage

Similar issues:
https://forum.proxmox.com/threads/l...t-and-require-to-be-manually-activated.94455/