Missing local-lvm volume after reboot

[BillyElliot]

Hey everyone,
im at a loss here, on my own.
I just restarted my main Proxmox node and since after some time no container was reachable I looked in Webinterface. No Container and VM was running. Trying to start them results in: "TASK ERROR: no such logical volume pve/data".
I think I found the local path to the original storage to be /dev/pve, since there are all the disks of my containers. I was just about to migrate from this machine to my new server but now I can't since the storage is apparently missing.
I have not modified the Hardware of this node since reboot.
Please let me know how to approach this issue, since my research online has not jet helped me fixing this.
I already tried running: lvs, pvscan, vgscan, but with no output?
And i noticed, running lsblk, that there are new (so I think; at least new since clustering) subdirectories and "drives" called: pve-data_tdata; pve-data_data; pve-data_tmeta.
Something else I remember was, that recently I sometimes saw a warning, that some pool was running out of space. Sadly I did not have time to investigate at these times...
I am happy to provide any info you need, I just don't know where to start in terms of providing information ahead of time, since im out of knowledge here.
Thank you for your time reading this and for you help!
Best regards
BillyElliot

 

[bbgeek17]

I already tried running: lvs, pvscan, vgscan, but with no output?

This would be very strange, are you sure? If so, perhaps you lost the disk that held your pvedata? Was it separate than you root disk?
What does lsblk show? lsscsi? nvme list?
Are all the disks listed in above commands that you expected? Is the disk where LVM supposed to be there?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[BillyElliot]

Hey @bbgeek17 and thank you for your reply.
I just reran those mentioned commands and pvscan outputs:
"No matching physical volumes found".
But other than that I get no output from those commands.
I only have one SSD installed right now, since this machine was my starting point into proxmox. So there only ever was this local ssd. Aside from that there is a external HDD, that I did not use for Proxmox, but was mounted, as you can see below, in lsblk output.
Here the output:

lsblk:
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 119.2G  0 disk
├─sda1                         8:1    0  1007K  0 part
├─sda2                         8:2    0   512M  0 part /boot/efi
└─sda3                         8:3    0 118.7G  0 part
  ├─pve-swap                 253:0    0     4G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0  29.5G  0 lvm  /
  ├─pve-data_tmeta           253:2    0     1G  0 lvm 
  │ └─pve-data-tpool         253:4    0  68.5G  0 lvm 
  │   ├─pve-data             253:5    0  68.5G  1 lvm 
  │   ├─pve-vm--100--disk--0 253:6    0    10G  0 lvm 
  │   ├─pve-vm--104--disk--0 253:7    0    10G  0 lvm 
  │   ├─pve-vm--101--disk--0 253:8    0     4M  0 lvm 
  │   ├─pve-vm--101--disk--1 253:9    0    32G  0 lvm 
  │   ├─pve-vm--110--disk--0 253:10   0    10G  0 lvm 
  │   ├─pve-vm--105--disk--0 253:11   0     5G  0 lvm 
  │   └─pve-vm--102--disk--0 253:12   0     8G  0 lvm 
  └─pve-data_tdata           253:3    0  68.5G  0 lvm 
    └─pve-data-tpool         253:4    0  68.5G  0 lvm 
      ├─pve-data             253:5    0  68.5G  1 lvm 
      ├─pve-vm--100--disk--0 253:6    0    10G  0 lvm 
      ├─pve-vm--104--disk--0 253:7    0    10G  0 lvm 
      ├─pve-vm--101--disk--0 253:8    0     4M  0 lvm 
      ├─pve-vm--101--disk--1 253:9    0    32G  0 lvm 
      ├─pve-vm--110--disk--0 253:10   0    10G  0 lvm 
      ├─pve-vm--105--disk--0 253:11   0     5G  0 lvm 
      └─pve-vm--102--disk--0 253:12   0     8G  0 lvm 
sdb                            8:16   0   3.6T  0 disk
└─sdb1                         8:17   0   3.6T  0 part /mnt/usb

both other commands are not found. But I don't have a nvme SSD, root drive is sata SSD.

Thank you again for you fast response and for your time helping me!
Best regards,
BillyElliot

 

[bbgeek17]

Run " pvscan -vvvv" and study output carefully, there is a good chance you will find there why the disk is being ignored.
You can use same options with pvs, vgs and lvs.
Are you sure you didnt play with lvm filter and forgot to remove your changes?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[BillyElliot]

Thank you for your input, @bbgeek17
I ran the command and saw some "Skipping (regex)" lines, on /dev/pve, as well as "filter caching bad".
The output of pvs, vgs and lvs seem identical in content as pvscan.
Im quite sure I did not play with lvm filer, as I don't even recognize this term.
As mentioned before, I recently (as in 1 week/3-4 reboots) added a second node to the (then) new cluster. I also was successful before in migrating a Nextcloud VM.
I just tried to locate "filters/filter-regex.c" to see what's in there, but could not do so.

Again thank you for your answer, time and patience!
BillyElliot

 

[bbgeek17]

The messages about "filter cache" are normal debug at that level of verbosity.

So you have a single disk, the disk still appears to contain some LVM data based on "lsblk" output, yet LVM commands dont show the information. It appears you have some sort of data corruption. Could be from unflushed write, could be due to bad hardware, or phase of the moon.

Next steps are either restore from backup, or plug this into google "lvm repair" and carefully read through the articles.
Dont run the commands that look data modifying that you dont fully understand. Measure twice, run once.
Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

Something else I remember was, that recently I sometimes saw a warning, that some pool was running out of space. Sadly I did not have time to investigate at these times...

Just saw this, which I skipped over initially. Running out of space is a bad idea for any data managing system (volume or file). Either way you are in "recovery/repair" land.
Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox