Grey question mark over storage - No such logical volume

[cyberspace-cat]

After a reboot (installed a video card) on my single node pve 7.3-3 box, I got a gray question mark on one of my lvm-thin local storage disks.

lsblk looks like it's /dev/sda without any partitions, just /dev/sda. Storage name is "local-5" which has been used for a few years as lvm-thin, no issues. Not showing anything related to local-5 in the output besides the syslog error(s).

Output of pvs, vgs and syslog error

root@pve:/var/log# pvs
  PV         VG      Fmt  Attr PSize    PFree
  /dev/sdc1  local4  lvm2 a--  <465.76g <135.76g
  /dev/sdd3  local-3 lvm2 a--  <750.00g <281.00g
  /dev/sde1  local-2 lvm2 a--  <465.76g  <87.76g
  /dev/sdf1  ssd3    lvm2 a--  <931.51g       0


root@controller:/var/log# vgs
  VG      #PV #LV #SN Attr   VSize    VFree
  local-2   1  23   0 wz--n- <465.76g  <87.76g
  local-3   1  18   0 wz--n- <750.00g <281.00g
  local4    1  17   0 wz--n- <465.76g <135.76g
  ssd3      1   2   0 wz--n- <931.51g       0

# Keeps repeating this below
Dec 13 01:01:37 controller pvestatd[3006]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Dec 13 01:01:37 controller pvestatd[3006]: no such logical volume local-5/local-5
Dec 13 01:01:38 controller pvedaemon[3049]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Dec 13 01:01:38 controller pvedaemon[3049]: no such logical volume local-5/local-5

cfdisk is showing it (/dev/sda) as 1 partition, but as "Filesystem: LMV2_Member".
fdisk -l is showing same, no partitions and nothing else related to "local-5" in the output.

storage.cfg:

lvmthin: local-5
        thinpool local-5
        vgname local-5
        content rootdir,images

Any help is appreciated. Had a ton of data on there

 

[fiona]

Hi,
I'd check for any errors/warnings from LVM or about /dev/sda during boot in /var/log/syslog. You can also try running pvscan -vv to get verbose output during scanning (you can use up to four v's, but it might get very long).

 

[cyberspace-cat]

Hi,
I'd check for any errors/warnings from LVM or about /dev/sda during boot in /var/log/syslog. You can also try running pvscan -vv to get verbose output during scanning (you can use up to four v's, but it might get very long).

Hmm still nothing of interest running those. No errors, other partitions or known vm's in reference to local-5 or /dev/sda. Have about 70% on pv backup server which I can restore.

Are there any recommendations on how to recover the vm files off "local-5" aka the disk /dev/sda. I can pull it out and hook it up to a desktop windows or linux box. That way the extra 30% of vm's can be pulled, and moved over to another local storage. Then I can just reformat /dev/sda. Thank you.

EDIT: Data was completely empty on that disk. Bit the bullet and restored from pbs, then created a new lvm thin on the empty ssd. Not sure what the root cause was.

 

[fiona]

Hmm still nothing of interest running those. No errors, other partitions or known vm's in reference to local-5 or /dev/sda. Have about 70% on pv backup server which I can restore.

Are there any recommendations on how to recover the vm files off "local-5" aka the disk /dev/sda. I can pull it out and hook it up to a desktop windows or linux box. That way the extra 30% of vm's can be pulled, and moved over to another local storage. Then I can just reformat /dev/sda. Thank you.

EDIT: Data was completely empty on that disk. Bit the bullet and restored from pbs, then created a new lvm thin on the empty ssd. Not sure what the root cause was.

So the pvscan command also didn't mention /dev/sda? Strange, as you said it still showed as an LVM2_Member with cfdisk. How did you verify that "data was completely empty on that disk"? Hopefully it's not a hardware issue with the disk itself.

 

[cyberspace-cat]

So the pvscan command also didn't mention /dev/sda? Strange, as you said it still showed as an LVM2_Member with cfdisk. How did you verify that "data was completely empty on that disk"? Hopefully it's not a hardware issue with the disk itself.

I'm pretty sure it was a hardware issue. Must've just wiped it, maybe electro static or something during the video card install. Did some other maintenance too like removing old drives that fried due to a power outage/surge awhile back (prior to getting an UPS). Showing 20% wear, so not a fried ssd. Open debian gui disk utility and showed nothing, like it was empty. pvscan showed no vm's associated with it.

Anyways, re-set it as lvm-thin storage via the new proxmox 7 feature and just migrating backups back onto it. Home lab use so not a business impact or anything. Thank tho for your help.