inactive and invisible thin-pool after 7to8 upgrade

[guff666]

I was updating a server from 7.4 to 8.3 and after rebooting, it's come up without an `lvm-thin` pool. running `lvscan` shows the `pve/data` pool is inactive

root@pve14:~# lvscan
  inactive          '/dev/pve/data' [<7.13 TiB] inherit
  ACTIVE            '/dev/pve/swap' [7.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  inactive          '/dev/pve/vm-1400-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-1402-disk-0' [516.00 GiB] inherit
  inactive          '/dev/pve/vm-1400-disk-1' [4.88 TiB] inherit
  inactive          '/dev/pve/base-1401-disk-0' [8.00 GiB] inherit
  inactive          '/dev/pve/vm-1403-disk-0' [8.00 GiB] inherit

The journal shows:

Feb 14 15:36:04 pve14 vgchange[894]: cannot perform fix without a full examination
Feb 14 15:36:04 pve14 vgchange[894]: Usage: thin_check [options] {device|file}
Feb 14 15:36:04 pve14 vgchange[894]: Options:
Feb 14 15:36:04 pve14 vgchange[894]:   {-q|--quiet}
Feb 14 15:36:04 pve14 vgchange[894]:   {-h|--help}
Feb 14 15:36:04 pve14 vgchange[894]:   {-V|--version}
Feb 14 15:36:04 pve14 vgchange[894]:   {-m|--metadata-snap}
Feb 14 15:36:04 pve14 vgchange[894]:   {--auto-repair}
Feb 14 15:36:04 pve14 vgchange[894]:   {--override-mapping-root}
Feb 14 15:36:04 pve14 vgchange[894]:   {--clear-needs-check-flag}
Feb 14 15:36:04 pve14 vgchange[894]:   {--ignore-non-fatal-errors}
Feb 14 15:36:04 pve14 vgchange[894]:   {--skip-mappings}
Feb 14 15:36:04 pve14 vgchange[894]:   {--super-block-only}
Feb 14 15:36:04 pve14 vgchange[692]:   Check of pool pve/data failed (status:1). Manual repair required!

Unfortunately, I can't run `thin_check` on `pve/data` because it doesn't appear in `/dev/pve`

root@pve14:~# ls -l /dev/pve
total 0
lrwxrwxrwx 1 root root 7 Feb 14 15:36 root -> ../dm-1
lrwxrwxrwx 1 root root 7 Feb 14 15:36 swap -> ../dm-0
[CODE]

If I reboot with an Ubuntu 20 live disk, all volumes show as available, but I get errors about a corrupt superblock on pve/data.

I've seen lots of other similar threads, but in all of them /dev/pve/data exists.

 

[waltar]

Try this steps:
pvscan
vgscan
vgcfgrestore --list
vgcfgrestore pve
vgscan
vgs
lvscan
lvs
thin_check /dev/pve/data
thin_check --auto-repair /dev/pve/data
thin_check /dev/pve/data
maybe this also if not ok after the above cmd's: vgchange -ay pve
lvscan
lvs

 

[guff666]

Thanks, @waltar, however, I'd need to understand better what vgcfgrestore does before I commit anything that changes the state. From what I can find, it uses the files in /etc/lvm/archive. There are many files in there because I have had multiple attempts at repairing the thin-pool. There's nothing in the info from vgcfgrestore --list pve that indicates a good point to restore to:

File:         /etc/lvm/archive/pve_00396-1533845931.vg/pve_00396-1533845931.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove /dev/pve/snap_vm-1402-disk-0_vzdump'
  Backup Time:  Tue Feb 11 13:00:14 2025

  
  File:         /etc/lvm/archive/pve_00397-2099267154.vg/pve_00397-2099267154.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-1402-disk-0_vzdump -pr -s pve/vm-1402-disk-0'
  Backup Time:  Tue Feb 11 13:00:43 2025

  
  File:         /etc/lvm/archive/pve_00398-306271771.vg/pve_00398-306271771.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvremove -f pve/snap_vm-1402-disk-0_vzdump'
  Backup Time:  Tue Feb 11 13:34:00 2025

  
  File:         /etc/lvm/archive/pve_00399-822780001.vg/pve_00399-822780001.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-1402-disk-0_vzdump -pr -s pve/vm-1402-disk-0'
  Backup Time:  Wed Feb 12 06:04:20 2025

  
  File:         /etc/lvm/archive/pve_00400-632526696.vg/pve_00400-632526696.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvremove -f pve/snap_vm-1402-disk-0_vzdump'
  Backup Time:  Wed Feb 12 11:28:52 2025

  
  File:         /etc/lvm/archive/pve_00401-1626614872.vg/pve_00401-1626614872.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-1403-disk-0_vzdump -pr -s pve/vm-1403-disk-0'
  Backup Time:  Wed Feb 12 11:29:01 2025

  
  File:         /etc/lvm/archive/pve_00402-1743971854.vg/pve_00402-1743971854.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvremove -f pve/snap_vm-1403-disk-0_vzdump'
  Backup Time:  Wed Feb 12 11:36:36 2025

  
  File:         /etc/lvm/archive/pve_00403-545232378.vg/pve_00403-545232378.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-1402-disk-0_vzdump -pr -s pve/vm-1402-disk-0'
  Backup Time:  Thu Feb 13 06:04:34 2025

  
  File:         /etc/lvm/archive/pve_00404-1603621844.vg/pve_00404-1603621844.vg
  VG name:      pve
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-1403-disk-0_vzdump -pr -s pve/vm-1403-disk-0'
  Backup Time:  Thu Feb 13 12:05:05 2025

  
  File:         /etc/lvm/archive/pve_00405-1134549171.vg/pve_00405-1134549171.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove /dev/pve//snap_vm-1402-disk-0_vzdump'
  Backup Time:  Thu Feb 13 13:42:06 2025

  
  File:         /etc/lvm/archive/pve_00406-923233398.vg/pve_00406-923233398.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:09:44 2025

  
  File:         /etc/lvm/archive/pve_00407-1918809782.vg/pve_00407-1918809782.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:11:58 2025

  
  File:         /etc/lvm/archive/pve_00408-1636882381.vg/pve_00408-1636882381.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:17:28 2025

  
  File:         /etc/lvm/archive/pve_00409-80370912.vg/pve_00409-80370912.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/data_meta0'
  Backup Time:  Thu Feb 13 14:51:41 2025

  
  File:         /etc/lvm/archive/pve_00410-1236734815.vg/pve_00410-1236734815.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/data_meta1'
  Backup Time:  Thu Feb 13 14:51:43 2025

  
  File:         /etc/lvm/archive/pve_00411-565185542.vg/pve_00411-565185542.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:51:47 2025

  
  File:         /etc/lvm/archive/pve_00412-1295053456.vg/pve_00412-1295053456.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:54:00 2025

  
  File:         /etc/lvm/archive/pve_00413-1643939630.vg/pve_00413-1643939630.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:57:20 2025

  
  File:         /etc/lvm/archive/pve_00414-337716381.vg/pve_00414-337716381.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Thu Feb 13 14:59:33 2025

  
  File:         /etc/lvm/archive/pve_00415-912134038.vg/pve_00415-912134038.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/snap_vm-1403-disk-0_vzdump'
  Backup Time:  Thu Feb 13 17:58:54 2025

  
  File:         /etc/lvm/archive/pve_00416-1153799467.vg/pve_00416-1153799467.vg
  VG name:      pve
  Description:  Created *before* executing '/usr/sbin/vgchange -ay'
  Backup Time:  Fri Feb 14 15:36:03 2025

  
  File:         /etc/lvm/archive/pve_00417-2145268757.vg/pve_00417-2145268757.vg
  VG name:      pve
  Description:  Created *before* executing '/usr/sbin/vgchange -ay'
  Backup Time:  Fri Feb 14 15:36:03 2025

  
  File:         /etc/lvm/archive/pve_00418-1864459304.vg/pve_00418-1864459304.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Fri Feb 14 16:42:59 2025

  
  File:         /etc/lvm/archive/pve_00419-324987228.vg/pve_00419-324987228.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair pve/data'
  Backup Time:  Fri Feb 14 16:45:13 2025

  
  File:         /etc/lvm/archive/pve_00420-72099545.vg/pve_00420-72099545.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 16:50:20 2025

  
  File:         /etc/lvm/archive/pve_00421-850431647.vg/pve_00421-850431647.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/data_meta0'
  Backup Time:  Fri Feb 14 17:00:13 2025

  
  File:         /etc/lvm/archive/pve_00422-856657675.vg/pve_00422-856657675.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/data_meta1'
  Backup Time:  Fri Feb 14 17:01:55 2025

  
  File:         /etc/lvm/archive/pve_00423-124086473.vg/pve_00423-124086473.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:02:36 2025

  
  File:         /etc/lvm/archive/pve_00424-519054210.vg/pve_00424-519054210.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:04:49 2025

  
  File:         /etc/lvm/archive/pve_00425-1651354562.vg/pve_00425-1651354562.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:08:07 2025

  
  File:         /etc/lvm/archive/pve_00426-1467260246.vg/pve_00426-1467260246.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:10:20 2025

  
  File:         /etc/lvm/archive/pve_00427-1323402246.vg/pve_00427-1323402246.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/data_meta0'
  Backup Time:  Fri Feb 14 17:13:01 2025

  
  File:         /etc/lvm/archive/pve_00428-934949293.vg/pve_00428-934949293.vg
  VG name:      pve
  Description:  Created *before* executing 'lvremove pve/vm-1402-disk-0'
  Backup Time:  Fri Feb 14 17:15:36 2025

  
  File:         /etc/lvm/archive/pve_00429-724412191.vg/pve_00429-724412191.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:32:19 2025

  
  File:         /etc/lvm/archive/pve_00430-863467780.vg/pve_00430-863467780.vg
  VG name:      pve
  Description:  Created *before* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:34:33 2025

  
  File:         /etc/lvm/backup/pve/pve
  VG name:      pve
  Description:  Created *after* executing 'lvconvert --repair -v pve/data'
  Backup Time:  Fri Feb 14 17:34:33 2025

`lvs -a` shows:

LV               VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  base-1401-disk-0 pve Vri---tz-k  8.00g data                                               
  data             pve twi---tz-- <7.13t                                                    
  data_meta0       pve -wi-a----- 15.81g                                                    
  [data_tdata]     pve Twi------- <7.13t                                                    
  [data_tmeta]     pve ewi------- 15.81g                                                    
  [lvol2_pmspare]  pve ewi------- 15.81g                                                    
  root             pve -wi-ao---- 96.00g                                                    
  swap             pve -wi-ao----  7.00g                                                    
  vm-1400-disk-0   pve Vwi---tz-- 32.00g data                                               
  vm-1400-disk-1   pve Vwi---tz--  4.88t data                                               
  vm-1403-disk-0   pve Vwi---tz--  8.00g data

`lvscan` shows

inactive          '/dev/pve/data' [<7.13 TiB] inherit
  ACTIVE            '/dev/pve/swap' [7.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  inactive          '/dev/pve/vm-1400-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-1400-disk-1' [4.88 TiB] inherit
  inactive          '/dev/pve/base-1401-disk-0' [8.00 GiB] inherit
  inactive          '/dev/pve/vm-1403-disk-0' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [15.81 GiB] inherit

`pvscan` shows

PV /dev/sda3   VG pve             lvm2 [<7.28 TiB / 572.00 MiB free]
  Total: 1 [<7.28 TiB] / in use: 1 [<7.28 TiB] / in no VG: 0 [0   ]

and `vgscan'

Found volume group "pve" using metadata type lvm2

The fact that everything appears to be OK under Ubuntu 20 makes me wonder if the problem is not the thin-pool itself, but rather something with proxmox.

 

[waltar]

vgcfgrestore check and restore last (or even older if selected) the definiton of volumes in the lvm pool so not restore data by itself. This is useful when volumes cannot be found anymore or the regular metadata of volumes cannot be read.