[SOLVED] No such logical volume pve/data

Mio3000 · Nov 15, 2022

Hello all,

during the night my backup hanged and i was forced to reboot my proxmox host using hw button. after reboot my local-lvm dissapeard and i lost access to my virtual machines. could someone help me to restore it? thank you.

this is what i have in log:
no such logical volume pve/data

Code:

root@proxmox:~#  pvesm lvmthinscan pve
  Volume group "pve" not found
  Cannot process volume group pve
command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --config 'report/time_format="%s"' --options vg_name,lv_name,lv_size,lv_attr,pool_lv,data_percent,metadata_percent,snap_percent,uuid,tags,metadata_size,time pve' failed: exit code 5

and this is my lsblk:

Code:

root@proxmox:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0   2.7T  0 disk
sdb                            8:16   0   2.7T  0 disk
sdc                            8:32   0 298.1G  0 disk
└─sdc1                         8:33   0 298.1G  0 part
sdd                            8:48   0   1.8T  0 disk
└─sdd1                         8:49   0   1.8T  0 part
sde                            8:64   0 476.9G  0 disk
├─sde1                         8:65   0  1007K  0 part
├─sde2                         8:66   0   512M  0 part
└─sde3                         8:67   0 476.4G  0 part
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta           253:2    0   3.6G  0 lvm
  │ └─pve-data-tpool         253:4    0 349.3G  0 lvm
  │   ├─pve-data             253:5    0 349.3G  1 lvm
  │   ├─pve-vm--103--disk--0 253:6    0    70G  0 lvm
  │   ├─pve-vm--106--disk--0 253:7    0    70G  0 lvm
  │   ├─pve-vm--106--disk--1 253:8    0     4M  0 lvm
  │   ├─pve-vm--105--disk--0 253:9    0   200G  0 lvm
  │   ├─pve-vm--107--disk--0 253:10   0    70G  0 lvm
  │   ├─pve-vm--104--disk--0 253:11   0     4M  0 lvm
  │   ├─pve-vm--104--disk--1 253:12   0    32G  0 lvm
  │   ├─pve-vm--100--disk--0 253:13   0    32G  0 lvm
  │   └─pve-vm--108--disk--0 253:14   0    32G  0 lvm
  └─pve-data_tdata           253:3    0 349.3G  0 lvm
    └─pve-data-tpool         253:4    0 349.3G  0 lvm
      ├─pve-data             253:5    0 349.3G  1 lvm
      ├─pve-vm--103--disk--0 253:6    0    70G  0 lvm
      ├─pve-vm--106--disk--0 253:7    0    70G  0 lvm
      ├─pve-vm--106--disk--1 253:8    0     4M  0 lvm
      ├─pve-vm--105--disk--0 253:9    0   200G  0 lvm
      ├─pve-vm--107--disk--0 253:10   0    70G  0 lvm
      ├─pve-vm--104--disk--0 253:11   0     4M  0 lvm
      ├─pve-vm--104--disk--1 253:12   0    32G  0 lvm
      ├─pve-vm--100--disk--0 253:13   0    32G  0 lvm
      └─pve-vm--108--disk--0 253:14   0    32G  0 lvm

fiona · Nov 15, 2022

Hi,
what is the output of the following?

Code:

pvs
vgs
lvs
pveversion -v

Are there any related error messages during boot in the syslog?

Mio3000 · Nov 15, 2022

@fiona - Hello,

after couple of reboots it started working again...

Code:

root@proxmox:~# pvs
  PV         VG  Fmt  Attr PSize    PFree 
  /dev/sdd3  pve lvm2 a--  <476.44g <16.00g
root@proxmox:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree 
  pve   1  12   0 wz--n- <476.44g <16.00g
root@proxmox:~# lvs
  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotz-- <349.31g             43.38  2.13                           
  root          pve -wi-ao----   96.00g                                                   
  swap          pve -wi-ao----    8.00g                                                   
  vm-100-disk-0 pve Vwi-aotz--   32.00g data        7.49                                   
  vm-103-disk-0 pve Vwi-a-tz--   70.00g data        37.75                                 
  vm-104-disk-0 pve Vwi-aotz--    4.00m data        0.00                                   
  vm-104-disk-1 pve Vwi-aotz--   32.00g data        45.69                                 
  vm-105-disk-0 pve Vwi-aotz--  200.00g data        14.37                                 
  vm-106-disk-0 pve Vwi-a-tz--   70.00g data        41.13                                 
  vm-106-disk-1 pve Vwi-a-tz--    4.00m data        3.12                                   
  vm-107-disk-0 pve Vwi-a-tz--   70.00g data        67.03                                 
  vm-108-disk-1 pve Vwi-aotz--   32.00g data        11.34                                 
root@proxmox:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-12
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
root@proxmox:~#

i will try another reboot or shutdown in one hour, if the issue still persist... for now no errors in syslog

Mio3000 · Nov 15, 2022

@fiona - hello, so the issue persist after reboot:

Code:

Linux proxmox 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov 15 13:55:42 2022 from 192.168.123.109
root@proxmox:~# pvs
root@proxmox:~# vgs
root@proxmox:~# lvs
root@proxmox:~#

EDIT: and only hw reboot will bring back the local-lvm, reboot trough terminal will not fix the missing local-lvm

fiona · Nov 15, 2022

Mio3000 said:

@fiona - hello, so the issue persist after reboot:

Code:

Linux proxmox 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov 15 13:55:42 2022 from 192.168.123.109
root@proxmox:~# pvs
root@proxmox:~# vgs
root@proxmox:~# lvs
root@proxmox:~#

EDIT: and only hw reboot will bring back the local-lvm

Please check the syslog during boot. There really should be some hints why LVM can't initialize properly.

Mio3000 · Nov 15, 2022

i didnt found anything, here is the syslog, please check it, you maybe find something, thank you

bbgeek17 · Nov 15, 2022

Nov 15 13:54:29 proxmox lvm[887]: pvscan[887] /dev/sde3 excluded by filters: device is rejected by filter config.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Mio3000 · Nov 15, 2022

@bbgeek17 - i reverted changes in the lvm.conf, and now it is working, i dont know why it did not popup when i tested it couple of weeks ago, and did also the reboot, all was ok... thank you for finiding this

mishanw · Apr 10, 2024

What did you have to do? im having the same issue.

fiona · Apr 10, 2024

Hi,

mishanw said:
What did you have to do? im having the same issue.

please share the exact error message you get and the output of the following:

Code:

pvs
vgs
lvs
cat /etc/pve/storage.cfg
pveversion -v

The system log/journal during boot might also contain more information (e.g. to get full journal since the last boot journalctl -b > /tmp/journal.txt).

mishanw · Apr 10, 2024

fiona said:
Hi,

please share the exact error message you get and the output of the following:

Code:

pvs vgs lvs cat /etc/pve/storage.cfg pveversion -v

The system log/journal during boot might also contain more information (e.g. to get full journal since the last boot journalctl -b > /tmp/journal.txt).

pvs
vgs
lvs
cat /etc/pve/storage.cfg
pveversion -v
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

dir: backup-small
path /home/backup-small
content backup
prune-backups keep-all=1
shared 0

dir: usb-backup
path /home/usb-backup
content images,backup,iso
prune-backups keep-all=1
shared 0

proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve)
pve-manager: 8.1.10 (running version: 8.1.10/4b06efb5db453f29)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve1
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.10-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1

fiona · Apr 11, 2024

mishanw said:
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

This is the only LVM(thin) storage that is defined. Is the error about this one? Please share the output of

Code:

pvs
vgs
lvs

and check the boot log.

mishanw · Apr 11, 2024

fiona said:
This is the only LVM(thin) storage that is defined. Is the error about this one? Please share the output of

Code:

pvs vgs lvs

and check the boot log.

I think I figured it out.

2 of the nodes are using xfs for boot drives. On the server I added to the cluster, it has a ZFS mirrored raid for Proxmox as the boot volume, This was the thing that I belive caused the issue, I went to the datacenter storage and removed the new node from the LVM thin. No more errors, I mean I cant use the LVM-thin anymore, but hopefully thats not a deal breaker.

fiona · Apr 12, 2024

mishanw said:
I think I figured it out.

2 of the nodes are using xfs for boot drives. On the server I added to the cluster, it has a ZFS mirrored raid for Proxmox as the boot volume, This was the thing that I belive caused the issue, I went to the datacenter storage and removed the new node from the LVM thin. No more errors, I mean I cant use the LVM-thin anymore, but hopefully thats not a deal breaker.

Okay, that explains it. Nodes in a cluster all share the same storage configuration, new nodes will inherit it when joining. You could also add back the local-zfs which should be present on the new node (going to Datacenter > Storage > Add > ZFS) and restrict that to just the new node.

mishanw · Apr 12, 2024

fiona said:
Okay, that explains it. Nodes in a cluster all share the same storage configuration, new nodes will inherit it when joining. You could also add back the local-zfs which should be present on the new node (going to Datacenter > Storage > Add > ZFS) and restrict that to just the new node.

This is how it currently looks, Do I still need to add a local-zfs ? I realized that if one of my nodes goes down, VMs cant migrate to this node to provide HA since the nodes missing the local-lvm

fiona · Apr 15, 2024

mishanw said:
View attachment 66247

This is how it currently looks, Do I still need to add a local-zfs ? I realized that if one of my nodes goes down, VMs cant migrate to this node to provide HA since the nodes missing the local-lvm

You can still add back local-zfs if you want to use that space. You can't use HA with a local, non-replicated storage, like already answered in the other thread.

Search

Search

[SOLVED] No such logical volume pve/data

Mio3000

New Member

fiona

Proxmox Staff Member

Mio3000

New Member

Mio3000

New Member

fiona

Proxmox Staff Member

Mio3000

New Member

Attachments

bbgeek17

Distinguished Member

Mio3000

New Member

mishanw

New Member

fiona

Proxmox Staff Member

mishanw

New Member

fiona

Proxmox Staff Member

mishanw

New Member

fiona

Proxmox Staff Member

mishanw

New Member

fiona

Proxmox Staff Member