Power failure -> LV not available -> mount fail -> Failed to initialize container

[ThomasBlock]

Hi. I had an power failure. System could reboot from bootdisk (ssd). But one drive is now broken. I am not a lvm expert. Which Things and commands could i learn to fix the problem? Mount commands? repair commands?

VM 101 needs disk "hdd". everything seems to be available. but GUI shows that there is now "0% usage" on the disk. what can i do?

There are no (hardware) errors in syslog etc.. The Disks are RAID6 and the Raid Controller says all disks are fine, redundancy = 2.

If i recall correctly, these commands were used to create the disk:
pvcreate /dev/sdb1
vgcreate hdd /dev/sdb1
GUI: datacenter > storage > add lvm ( or lvm-thin? not sure.. )

lxc-start -n 101 -F
lxc-start: 101: conf.c: run_buffer: 316 Script exited with status 2

lxc-start: 101: start.c: lxc_init: 816 Failed to run lxc.hook.pre-start for container "101"

lxc-start: 101: start.c: __lxc_start: 2007 Failed to initialize container "101"

lxc-start: 101: conf.c: run_buffer: 316 Script exited with status 1

lxc-start: 101: start.c: lxc_end: 962 Failed to run lxc.hook.post-stop for container "101"

lxc-start: 101: tools/lxc_start.c: main: 308 The container failed to start

lxc-start: 101: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options

lxc-start -o lxc-start.log -lDEBUG -F -n 101
lxc-start 101 20220906081253.660 INFO lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver AppArmor
lxc-start 101 20220906081253.660 INFO conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "101", config section "lxc"
lxc-start 101 20220906081254.370 DEBUG conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: failed to get device path
lxc-start 101 20220906081254.379 ERROR conf - conf.c:run_buffer:316 - Script exited with status 2
lxc-start 101 20220906081254.379 ERROR start - start.c:lxc_init:816 - Failed to run lxc.hook.pre-start for container "101"
lxc-start 101 20220906081254.379 ERROR start - start.c:__lxc_start:2007 - Failed to initialize container "101"
lxc-start 101 20220906081254.379 INFO conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "101", config section "lxc"
lxc-start 101 20220906081254.882 INFO conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "101", config section "lxc"
lxc-start 101 20220906081255.578 DEBUG conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 101 lxc post-stop produced output: umount: /var/lib/lxc/101/rootfs: not mounted
lxc-start 101 20220906081255.578 DEBUG conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 101 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/101/rootfs' failed: exit code 1
lxc-start 101 20220906081255.587 ERROR conf - conf.c:run_buffer:316 - Script exited with status 1
lxc-start 101 20220906081255.587 ERROR start - start.c:lxc_end:962 - Failed to run lxc.hook.post-stop for container "101"
lxc-start 101 20220906081255.588 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 101 20220906081255.588 ERROR lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options

pct mount 101

mounting container failed

failed to get device path

pvs
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 vgnvme0 lvm2 a-- 2.91t 0
/dev/sda3 pve lvm2 a-- 3.49t <16.38g
/dev/sdb hdd lvm2 a-- 130.97t 512.00m

pvdisplay
--- Physical volume ---
PV Name /dev/sdb
VG Name hdd
PV Size 130.97 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 34333055
Free PE 128
Allocated PE 34332927
PV UUID mD0Tp8-BxkT-sezn-5Lic-Cv6e-Ry3t-NrcrBo

vgs
VG #PV #LV #SN Attr VSize VFree
hdd 1 6 0 wz--n- 130.97t 512.00m
pve 1 14 0 wz--n- 3.49t <16.38g
vgnvme0 1 4 0 wz--n- 2.91t 0

vgdisplay
--- Volume group ---
VG Name hdd
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 30
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 6
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 130.97 TiB
PE Size 4.00 MiB
Total PE 34333055
Alloc PE / Size 34332927 / <130.97 TiB
Free PE / Size 128 / 512.00 MiB
VG UUID SrqL0n-pr8C-DG1l-ywhr-4H7o-4xgb-nGsAah

lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
hdd hdd twi---tz-- <130.94t
snap_vm-101-disk-0_leer hdd Vri---tz-k <127.93t hdd vm-101-disk-0
snap_vm-101-disk-0_neu202206 hdd Vri---tz-k <127.93t hdd vm-101-disk-0
snap_vm-101-disk-0_store hdd Vri---tz-k <127.93t hdd vm-101-disk-0
vm-101-disk-0 hdd Vwi---tz-- <127.93t hdd
vm-103-disk-0 hdd Vwi---tz-- 64.00g hdd
data pve twi-aotz-- 3.10t 8.25 0.90
root pve -wi-ao---- 96.00g
snap_vm-100-disk-0_eins pve Vri---tz-k 128.00g data vm-100-disk-0
snap_vm-100-disk-0_vortraffic pve Vri---tz-k 128.00g data
snap_vm-100-disk-1_eins pve Vri---tz-k 32.00g data vm-100-disk-1
swap pve -wi-a----- 256.00g
vm-100-disk-0 pve Vwi-a-tz-- 128.00g data snap_vm-100-disk-0_vortraffic 94.85
vm-100-disk-1 pve Vwi-a-tz-- 32.00g data 21.22
vm-100-state-eins pve Vwi-a-tz-- <31.74g data 9.62
vm-100-state-vortraffic pve Vwi-a-tz-- <31.74g data 7.00
vm-102-disk-0 pve Vwi-a-tz-- 8.00g data 11.88
vm-104-state-ok pve Vwi-a-tz-- <312.99g data 1.26
vm-105-disk-1 pve Vwi-aotz-- 64.00g data 7.50
vm-106-disk-0 pve Vwi-aotz-- 64.00g data 12.95
root vgnvme0 twi-aotz-- 2.66t 81.91 48.88
swap vgnvme0 -wi-ao---- 256.00g
vm-104-disk-0 vgnvme0 Vwi-aotz-- 1.95t root 87.32

lvdisplay
--- Logical volume ---
LV Name hdd
VG Name hdd
LV UUID nywYDf-zSXH-AN3K-6fHV-YKMl-IUp5-sjb0K9
LV Write Access read/write
LV Creation host, time apfel, 2021-07-07 11:45:36 +0200
LV Pool metadata hdd_tmeta
LV Pool data hdd_tdata
LV Status NOT available
LV Size <130.94 TiB
Current LE 34324831
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/hdd/vm-101-disk-0
LV Name vm-101-disk-0
VG Name hdd
LV UUID oFZSJL-bXun-QPch-1RE2-LtBK-1wUa-2Zgfux
LV Write Access read/write
LV Creation host, time apfel, 2021-07-07 17:03:50 +0200
LV Pool name hdd
LV Status NOT available
LV Size <127.93 TiB
Current LE 33536000
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/hdd/snap_vm-101-disk-0_leer
LV Name snap_vm-101-disk-0_leer
VG Name hdd
LV UUID 04ZCl4-8lDl-DbFV-vQiU-Mrbq-xXLd-Jt1iYd
LV Write Access read only
LV Creation host, time apfel, 2021-07-09 15:41:52 +0200
LV Pool name hdd
LV Thin origin name vm-101-disk-0
LV Status NOT available
LV Size <127.93 TiB
Current LE 33536000
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/hdd/snap_vm-101-disk-0_store
LV Name snap_vm-101-disk-0_store
VG Name hdd
LV UUID 5CU4Au-LVcK-iiIc-pCwT-cudt-lYOo-hi0NGu
LV Write Access read only
LV Creation host, time apfel, 2021-07-11 17:29:12 +0200
LV Pool name hdd
LV Thin origin name vm-101-disk-0
LV Status NOT available
LV Size <127.93 TiB
Current LE 33536000
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/hdd/vm-103-disk-0
LV Name vm-103-disk-0
VG Name hdd
LV UUID JyVieK-0he6-Va2m-6oEz-3o6R-nf0m-A9s9KF
LV Write Access read/write
LV Creation host, time apfel, 2022-02-21 09:12:16 +0100
LV Pool name hdd
LV Status NOT available
LV Size 64.00 GiB
Current LE 16384
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/hdd/snap_vm-101-disk-0_neu202206
LV Name snap_vm-101-disk-0_neu202206
VG Name hdd
LV UUID V5cS1f-jFay-vnY8-vnVE-2TMG-c1sl-JKy37e
LV Write Access read only
LV Creation host, time apfel, 2022-05-27 11:34:40 +0200
LV Pool name hdd
LV Thin origin name vm-101-disk-0
LV Status NOT available
LV Size <127.93 TiB
Current LE 33536000
Segments 1
Allocation inherit
Read ahead sectors auto

fdisk -l
Disk /dev/nvme0n1: 2.91 TiB, 3200631791616 bytes, 6251233968 sectors
Disk model: SAMSUNG MZPLJ3T2HBJR-00007
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sda: 3.49 TiB, 3840103415808 bytes, 7500201984 sectors
Disk model: PERC H710P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 33EE0C1A-2C18-44D3-9F31-3B6A4418F32F

Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 1050623 1048576 512M EFI System
/dev/sda3 1050624 7500201950 7499151327 3.5T Linux LVM


Disk /dev/sdb: 130.97 TiB, 144003274113024 bytes, 281256394752 sectors
Disk model: PERC H710P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

dmesg | grep sdb
[ 3.227985] sd 0:2:1:0: [sdb] 281256394752 512-byte logical blocks: (144 TB/131 TiB)
[ 3.228003] sd 0:2:1:0: [sdb] Write Protect is off
[ 3.228005] sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
[ 3.228043] sd 0:2:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3.238079] sd 0:2:1:0: [sdb] Attached SCSI disk