no local-lvm after deleting snapshot with failure and following reboot of the host

[Klaus H]

Ich habe am 29.3. gegen 9:12 Uhr einen Snapshot gelöscht um Platz zu schaffen, dabei gab es einen Fehler, diesen habe ich nicht weiter beachtet, nur gesehen dass in der WebGUI wo ich diesen Löschen wollte, dieser noch vorhanden war. Daraufhin beschloss ich einen Tag später den Host neuzustarten. Danach war local-lvm 0 Byte groß! Und ich benötige dringend die aktuellen Daten aus der enthaltenen Datenbank (vm-100-disk-2) da das letzte externe Backup zu alt ist.

Hoffe mir kann jemand helfen!
Danke für eure Zeit

root@kps-px0:~# pvs -v
Wiping internal VG cache
Wiping cache of LVM-capable devices
PV VG Fmt Attr PSize PFree DevSize PV UUID
/dev/sda3 pve lvm2 a-- 409.85g 15.84g 409.85g e2UAwY-3wDC-bcII-ALIX-UiGp-7Ifb-6kX01f


root@kps-px0:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi---tz-- 293.86g
root pve -wi-ao---- 50.00g
snap_vm-100-disk-1_snap1sql pve Vri---tz-k 32.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180202T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180223T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180228T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180302T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180305T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180309T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180312T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180315T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180319T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180322T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180326T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180327T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap20180328T0320 pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-1_snap2sql pve Vri---tz-k 53.00g data vm-100-disk-1
snap_vm-100-disk-2_snap1sql pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180202T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180219T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180223T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180228T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180302T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180305T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180309T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180312T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180315T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180319T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180322T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180326T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180327T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap20180328T0320 pve Vri---tz-k 100.00g data vm-100-disk-2
snap_vm-100-disk-2_snap2sql pve Vri---tz-k 100.00g data vm-100-disk-2
swap pve -wi-ao---- 50.00g
vm-100-disk-1 pve Vwi---tz-- 53.00g data
vm-100-disk-2 pve Vwi---tz-- 100.00g data

root@kps-px0:~# lvscan
ACTIVE '/dev/pve/swap' [50.00 GiB] inherit
ACTIVE '/dev/pve/root' [50.00 GiB] inherit
inactive '/dev/pve/data' [293.86 GiB] inherit
inactive '/dev/pve/vm-100-disk-1' [53.00 GiB] inherit
inactive '/dev/pve/vm-100-disk-2' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap1sql' [32.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap1sql' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap2sql' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap2sql' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180202T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180202T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180219T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180223T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180223T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180228T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180228T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180302T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180302T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180305T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180305T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180309T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180309T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180312T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180312T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180315T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180315T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180319T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180319T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180322T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180322T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180326T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180326T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180327T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180327T0320' [100.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-1_snap20180328T0320' [53.00 GiB] inherit
inactive '/dev/pve/snap_vm-100-disk-2_snap20180328T0320' [100.00 GiB] inherit


root@kps-px0:~# cat /etc/lvm/archive/pve_00188-1218386259.vg
siehe: h t t p : / / server2.klaushansch.de/pve_00188-1218386259.vg

root@kps-px0:~# pvck -d -v /dev/sda3
Scanning /dev/sda3
Found label on /dev/sda3, sector 1, type=LVM2 001
Found text metadata area: offset=4096, size=1044480
Found LVM2 metadata record at offset=98304, size=303104, offset2=0 size2=0
Found LVM2 metadata record at offset=75776, size=22528, offset2=0 size2=0
Found LVM2 metadata record at offset=73216, size=2560, offset2=0 size2=0
Found LVM2 metadata record at offset=66048, size=7168, offset2=0 size2=0
Found LVM2 metadata record at offset=65536, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=65024, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=64000, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=62976, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=61440, size=1536, offset2=0 size2=0
Found LVM2 metadata record at offset=60416, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=59392, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=58880, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=57856, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=56832, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=56320, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=55296, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=54272, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=53248, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=52736, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=52224, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=51712, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=48640, size=3072, offset2=0 size2=0
Found LVM2 metadata record at offset=47616, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=46592, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=46080, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=45056, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=44032, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=42496, size=1536, offset2=0 size2=0
Found LVM2 metadata record at offset=41472, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=40448, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=39936, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=38912, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=37888, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=36864, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=36352, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=35840, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=35328, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=32768, size=2560, offset2=0 size2=0
Found LVM2 metadata record at offset=32256, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=31232, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=30208, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=28672, size=1536, offset2=0 size2=0
Found LVM2 metadata record at offset=27648, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=26624, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=26112, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=25088, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=24064, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=23552, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=22528, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=21504, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=20480, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=19968, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=19456, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=18944, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=17920, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=17408, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=15872, size=1536, offset2=0 size2=0
Found LVM2 metadata record at offset=14848, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=13824, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=13312, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=12288, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=11264, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=9728, size=1536, offset2=0 size2=0
Found LVM2 metadata record at offset=8704, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=7680, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=7168, size=512, offset2=0 size2=0
Found LVM2 metadata record at offset=6144, size=1024, offset2=0 size2=0
Found LVM2 metadata record at offset=5120, size=1024, offset2=0 size2=0

Alle meine Versuche schlagen fehl, Sicherungskopien dieses Standes existieren!
Vielen Dank für eure Hilfe, wenn noch Infos fehlen, reiche ich diese natürlich nach

 

[Klaus H]

Hier noch was zu dem Zeitpunkt passiert ist:
....... here the thin pool pve-data-tpool is running full (100%)
Mar 29 09:13:07 kps-px0 kernel: [2574055.672984] device-mapper: space map common: ref count insert failed
Mar 29 09:13:07 kps-px0 lvm[765]: WARNING: Thin pool pve-data-tpool metadata is now 100.00% full.
Mar 29 09:13:08 kps-px0 kernel: [2574055.676085] device-mapper: space map common: ref count insert failed
Mar 29 09:13:08 kps-px0 kernel: [2574055.676777] device-mapper: thin: 253:4: metadata operation 'dm_pool_commit_metadata' failed: error = -28
Mar 29 09:13:08 kps-px0 kernel: [2574055.677478] device-mapper: thin: 253:4: aborting current metadata transaction
Mar 29 09:13:09 kps-px0 kernel: [2574056.985386] device-mapper: thin: 253:4: switching pool to read-only mode
Mar 29 09:13:09 kps-px0 kernel: [2574056.985547] device-mapper: thin: 253:4: unable to service pool target messages in READ_ONLY or FAIL mode
Mar 29 09:13:09 kps-px0 pvedaemon[7954]: lvremove snapshot 'pve/snap_vm-100-disk-1_snap20180219T0320' error: Failed to update pool pve/data.
Mar 29 09:13:09 kps-px0 pvedaemon[11205]: end task UPID:kps-px0:00001F12:0F57BA4F:5ABC91FB:qmdelsnapshot:100:root@pam: lvremove snapshot 'pve/snap_vm-100-disk-1_snap20180219T0320' error: Failed to update pool pve/data.