Local lvm systems won't start.

ukhost4u · Apr 10, 2018

Hello.

All my VM’s on the local-lvm has stopped working. When I try to start one I get:

Code:

root@vps15:~# qm start 101
kvm: -drive file=/dev/pve/vm-101-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/pve/vm-101-disk-1': No such file or directory
start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=c9165e46-091e-4be5-a786-734e1d9f5f3c' -name server.cloudstix.com -smp '4,sockets=2,cores=2,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 16000 -k en-gb -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:22fc9b8e76e' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/pve/vm-101-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=3A:38:39:63:33:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1

Hopefully this also helps:

Code:

root@vps15:~# lvs
  LV            VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve  twi-aotz--   3.59t             0.00   0.05                           
  lvol0_pmspare pve  -wi-a----- 116.00m                                                   
  repaired_01   pve  -wi-a-----   4.00g                                                   
  root          pve  -wi-ao----  96.00g                                                   
  swap          pve  -wi-ao----   8.00g                                                   
  vm-101-disk-1 pve  Vwi---tz-- 250.00g data                                               
  vm-102-disk-1 pve  Vwi---tz-- 402.00g data                                               
  vm-103-disk-1 pve  Vwi---tz-- 100.00g data                                               
  vm-105-disk-1 pve  Vwi---tz-- 100.00g data                                               
  vm-107-disk-1 pve  Vwi---tz-- 260.00g data                                               
  vm-110-disk-1 pve  Vwi---tz--   1.82t data                                               
  vm-113-disk-1 pve  Vwi---tz--  50.00g data                                               
  vm-114-disk-1 pve  Vwi---tz--  50.00g data                                               
  vm-117-disk-1 pve  Vwi---tz--  50.00g data                                               
  vm-118-disk-1 pve  Vwi---tz--  32.00g data                                               
root@vps15:~#

Code:

root@vps15:~# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  pve    1  15   0 wz--n- 3.71t 10.66g
root@vps15:~#

Code:

root@vps15:~# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                     8:0    0  5.2T  0 disk
├─sda1                  8:1    0 1007K  0 part
├─sda2                  8:2    0  127M  0 part
├─sda3                  8:3    0  3.7T  0 part
│ ├─pve-root          251:0    0   96G  0 lvm  /
│ ├─pve-swap          251:1    0    8G  0 lvm  [SWAP]
│ ├─pve-data_tmeta    251:2    0  1.1G  0 lvm 
│ │ └─pve-data-tpool  251:4    0  3.6T  0 lvm 
│ │   └─pve-data      251:5    0  3.6T  0 lvm 
│ ├─pve-data_tdata    251:3    0  3.6T  0 lvm 
│ │ └─pve-data-tpool  251:4    0  3.6T  0 lvm 
│ │   └─pve-data      251:5    0  3.6T  0 lvm 
│ ├─pve-lvol0_pmspare 251:6    0  116M  0 lvm 
│ └─pve-repaired_01   251:7    0    4G  0 lvm 
└─sda4                  8:4    0  1.5T  0 part /storage

Code:

root@vps15:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                BocxEB-IScl-DpEx-Da5J-N9Fa-p1p2-0aoMuX
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-12-08 10:25:44 +0000
  LV Status              available
  # open                 2
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:1
  
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                4u3qQm-DEog-PxFZ-V6Ik-41hK-wLQl-JlJFBY
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-12-08 10:25:44 +0000
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0
  
  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                007FAl-GnEF-1O1k-aNqd-pvOl-R1G5-P3ttQy
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-12-08 10:25:45 +0000
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              available
  # open                 1
  LV Size                3.59 TiB
  Allocated pool data    0.00%
  Allocated metadata     0.05%
  Current LE             942048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:4
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-101-disk-1
  LV Name                vm-101-disk-1
  VG Name                pve
  LV UUID                1o7FHR-dWMw-5v1o-0e4P-qEas-XC5q-2cSdS9
  LV Write Access        read/write
  LV Creation host, time vps15, 2016-12-08 14:02:42 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                250.00 GiB
  Current LE             64000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-102-disk-1
  LV Name                vm-102-disk-1
  VG Name                pve
  LV UUID                HoASUg-L9rK-Wvcb-xfgS-YgEH-ehfu-7ChgeO
  LV Write Access        read/write
  LV Creation host, time vps15, 2016-12-09 20:35:29 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                402.00 GiB
  Current LE             102912
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-103-disk-1
  LV Name                vm-103-disk-1
  VG Name                pve
  LV UUID                fciUVO-0JAb-Elaf-6NZ4-hNm9-208d-EoSaDr
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2017-01-06 15:02:02 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                100.00 GiB
  Current LE             25600
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-107-disk-1
  LV Name                vm-107-disk-1
  VG Name                pve
  LV UUID                lpXN5J-EZ32-5bZ0-OdYm-BU9k-TWkQ-yq3ymb
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2017-01-19 23:04:06 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                260.00 GiB
  Current LE             66560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-110-disk-1
  LV Name                vm-110-disk-1
  VG Name                pve
  LV UUID                MslLt4-53xG-Ep17-g9Bk-xUGV-OReV-T4aVR7
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2017-03-26 09:40:41 +0100
  LV Pool name           data
  LV Status              NOT available
  LV Size                1.82 TiB
  Current LE             476599
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-105-disk-1
  LV Name                vm-105-disk-1
  VG Name                pve
  LV UUID                zSGt5r-eEMY-rcqz-cRe9-oWEg-sUaA-nRe0Zk
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-01-30 14:40:16 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                100.00 GiB
  Current LE             25600
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-113-disk-1
  LV Name                vm-113-disk-1
  VG Name                pve
  LV UUID                iSNIwV-Nb2r-nX59-0KOy-2jmd-QUSz-gK14lT
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-03-21 11:54:00 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-117-disk-1
  LV Name                vm-117-disk-1
  VG Name                pve
  LV UUID                mSy1hx-HQKB-dgXc-5CHj-Zelw-3S55-3sKwwU
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-03-22 17:29:24 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-114-disk-1
  LV Name                vm-114-disk-1
  VG Name                pve
  LV UUID                i9WqSj-JaH0-jfXf-CqMU-1W8R-kHjG-5C6lC2
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-03-22 17:47:51 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-118-disk-1
  LV Name                vm-118-disk-1
  VG Name                pve
  LV UUID                bxaP5Y-EsVB-LlLi-fKuu-3No8-F0H0-jEDTkz
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-03-23 16:55:41 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                32.00 GiB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/lvol0_pmspare
  LV Name                lvol0_pmspare
  VG Name                pve
  LV UUID                XxT37U-DhdE-NFrw-EHJj-e6Ff-k57f-3PZFv0
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-12-08 10:25:45 +0000
  LV Status              available
  # open                 0
  LV Size                116.00 MiB
  Current LE             29
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:6
  
  --- Logical volume ---
  LV Path                /dev/pve/repaired_01
  LV Name                repaired_01
  VG Name                pve
  LV UUID                jFixHd-Yvzf-GTyA-ujJx-pya5-ZfQz-uWVv86
  LV Write Access        read/write
  LV Creation host, time vps15.ukhost4u.com, 2018-04-10 04:37:00 +0100
  LV Status              available
  # open                 0
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:7

I understand the issue is because they are at status: NOT available but when I try to active one I get:

Code:

root@vps15:~# vgchange -a y
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  device-mapper: reload ioctl on  failed: No data available
  5 logical volume(s) in volume group "pve" now active
root@vps15:~#

Can anyone help?

charlesg · Jul 30, 2018

Were you ever able to get this fixed?

I just ran into a similar issue after filling up my metadata pool. I started seeing LVM issues and I tried to grow the pool size. Since I suspected the metadata was corrupted I tried the below which brought me to your start.

Code:

lvconvert --repair

I cannot activate or find any of the thin provisioned volumes. I am trying to establish if this is something I could recover from or I should just blow ProxMox away.

Code:

root@pve1:/# lvchange -ay pve/vm-100-disk-5
  device-mapper: reload ioctl on  failed: No data available

Regards,

charlesg · Aug 18, 2018

It turns out filing up the metadatapool and running `lvconvert --repair` corrupted everything. I used the thin-provisioning-tools (https://github.com/jthornber/thin-provisioning-tools) and some advice from knowledgeable folks to recover.

I documented the process here:

https://www.unixrealm.com/?p=12000

Search

Search

Local lvm systems won't start.

ukhost4u

Member

charlesg

Active Member

charlesg

Active Member

We value your privacy