[SOLVED] Host Unknown Filesystem, rescue mode on boot

H

HiTeckGrenwick

Member
Aug 10, 2020
2
0
6
36
So I believe I'm in the same boat as this thread:
https://forum.proxmox.com/threads/grub-rescue-recover-data.55084/
However I am stuck at mounting the logical volume in order to recover the lvm meta data.

Cause of problem;
Yesterday we had several successive power cuts and at some point my APC UPS couldn't keep up and just switched off causing an abrupt shutdown of my PVE box. Upon boot, connection to the web panel was timing out and my router showed no connection from the assigned IP. I connected a monitor and was greeted with:
Code: 
error: unknown filesystem
Entering rescue mode..

grub rescue>
Running 'ls' showed all drives but ls on the drives and gpts yielded unknown filesystem every time.

Current status;
I loaded up Debian 10.5 Live on a Ventoy USB and started following various steps towards recovering volume metadata. I am at a point where I think I need to mount the /dev/pve/root but I receive
Code: 
user@debian:/mnt/pvelvm$ sudo mount /dev/pve/root /mnt/pvelvm/root
mount: /mnt/pvelvm/root: wrong fs type, bad option, bad superblock on /dev/mapper/pve-root, missing codepage or helper program, or other error.


Pertinent info and steps taken so far;
All progress as been taken in the Debian 10.5 Live shell as gui wouldn't load on my system.

I used 'vgscan' to check the lvm name, I then used 'vgchange -ay pve' which returned "5 logical volumes in volume group "pve" no active. Next I checked if I could list the contents of pve, 'sudo ls -l /dev/pve/ which returned:
Code: 
lrwxrwxrwx 1 root root 7 Aug 10 18:32 root -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 10 18:32 swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 10 18:32 vm-101-disk-0 -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 10 18:32 vm-200-disk-0 -> ../dm-7
I then tried to mount root as above but received that error.

From what I've gathered from searching I think I need to recover an archived lvm metadata from the /dev/pve/root/etc/lvm/archive and restore it with pvcreate but I am stuck just trying to recover the archive. Am I on the right track, is there any other way I could recover my Proxmox environment or fix the boot/grub?

Thank you for your time and assistance,
-Brian


Specs:
Code: 
CPU:        AMD Ryzen 5 3400G
Chipset:    A300
System:     ASRock DeskMini A300
Boot drive: Sabrent Rocket 256GB m.2 NVME

Additional drive:
ZFS RAID - 2x WD Blue Mobile 2TB
ext4 - WD SN750 500GB m.2 NVME

First I wanted to determine if the physical dives showed up correctly;
Code: 
sudo fdisk -l

Disk /dev/nvme0n1: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Disk model: Sabrent                                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A255AEDD-9653-4D36-B7C3-15C3C2A71AAB

Device           Start       End   Sectors  Size Type
/dev/nvme0n1p1      34      2047      2014 1007K BIOS boot
/dev/nvme0n1p2    2048   1050623   1048576  512M EFI System
/dev/nvme0n1p3 1050624 500118158 499067535  238G Linux LVM


Disk /dev/nvme1n1: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: WDS500G3X0C-00SJG0                     
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20SPZX-22U
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 4B9609CF-1D5E-E34F-8664-D474A62017CE

Device          Start        End    Sectors  Size Type
/dev/sda1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sda9  3907012608 3907028991      16384    8M Solaris reserved 1


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20SPZX-22U
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 96231F2F-D193-C84F-B8BF-5222356D5E35

Device          Start        End    Sectors  Size Type
/dev/sdb1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdb9  3907012608 3907028991      16384    8M Solaris reserved 1


Disk /dev/loop0: 2.1 GiB, 2221899776 bytes, 4339648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

/dev/nvme0n1 is the target drive and the partition info appears correct to me. Next I checked physical and logical volumes
Code: 
sudo pvdisplay /dev/nvme0n1p3

  --- Physical volume ---
  PV Name               /dev/nvme0n1p3
  VG Name               pve
  PV Size               237.97 GiB / not usable <1.32 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              60921
  Free PE               4094
  Allocated PE          56827
  PV UUID               OS0F3w-drWC-eqYN-uF6G-2Qux-Qt1j-6ZyXVO
  
  --- Physical Segments ---
  Physical extent 0 to 2047:
    Logical volume    /dev/pve/swap
    Logical extents    0 to 2047
  Physical extent 2048 to 17215:
    Logical volume    /dev/pve/root
    Logical extents    0 to 15167
  Physical extent 17216 to 56032:
    Logical volume    /dev/pve/data_tdata
    Logical extents    0 to 38816
  Physical extent 56033 to 56429:
    Logical volume    /dev/pve/data_tmeta
    Logical extents    0 to 396
  Physical extent 56430 to 56826:
    Logical volume    /dev/pve/lvol0_pmspare
    Logical extents    0 to 396
  Physical extent 56827 to 60920:
    FREE



sudo lvdisplay
 
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                1GTwIs-5qr8-J1Rv-ptg4-XGhx-wakw-2rXp8T
  LV Write Access        read/write
  LV Creation host, time proxmox, 2020-06-24 21:34:43 +0000
  LV Status              NOT available
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                AMFVsO-4l69-O8Zv-ILFm-EgLR-q6p2-Fjf9pf
  LV Write Access        read/write
  LV Creation host, time proxmox, 2020-06-24 21:34:43 +0000
  LV Status              NOT available
  LV Size                59.25 GiB
  Current LE             15168
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                nmi1DK-kUs3-UjoS-Gwc1-t3w1-Th9O-yogP9K
  LV Write Access        read/write
  LV Creation host, time proxmox, 2020-06-24 21:34:43 +0000
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              NOT available
  LV Size                <151.63 GiB
  Current LE             38817
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-200-disk-0
  LV Name                vm-200-disk-0
  VG Name                pve
  LV UUID                lsJUHe-u1t4-uLuN-RKBf-G7PU-gJjy-QYpjFw
  LV Write Access        read/write
  LV Creation host, time ltspve, 2020-06-28 20:43:10 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                12.00 GiB
  Current LE             3072
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  
  --- Logical volume ---
  LV Path                /dev/pve/vm-101-disk-0
  LV Name                vm-101-disk-0
  VG Name                pve
  LV UUID                ZlExPW-fQsp-pDqJ-8bp9-ova2-RtzJ-eY3tmc
  LV Write Access        read/write
  LV Creation host, time ltspve, 2020-07-16 00:08:57 +0000
  LV Pool name           data
  LV Status              NOT available
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  


sudo lvs

  LV                              VG    Attr       LSize    Pool  Origin        Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                                                
  data                            pve   twi---tz-- <151.63g                                                             data_tdata(0)       
  [data_tdata]                    pve   Twi------- <151.63g                                                             /dev/nvme0n1p3(17216)
  [data_tmeta]                    pve   ewi-------    1.55g                                                             /dev/nvme0n1p3(56033)
  [lvol0_pmspare]                 pve   ewi-------    1.55g                                                             /dev/nvme0n1p3(56430)
  root                            pve   -wi-------   59.25g                                                             /dev/nvme0n1p3(2048)
  swap                            pve   -wi-------    8.00g                                                             /dev/nvme0n1p3(0)   
  vm-101-disk-0                   pve   Vwi---tz--    8.00g data                                                                             
  vm-200-disk-0                   pve   Vwi---tz--   12.00g data

Some info redacted to meet character limit, full outputs attached.
 

Attachments

Turned out my solution was simpler than the track I was on. Since my most recent system backup was only three weeks ago I wasn't going to be set back to far if things went completely arye. So I found a post today about receiving "wrong fs type..." when mounting an LVM and the suggestion was to run fsck. So I loaded up Debian, installed lvm2, set the volume group to active and ran:
Code: 
fsck.ext3 /dev/pve/root
Gave an 'a' for yes to all and I am back up and running.
Hopefully my quest for boot recovery can help someone else. ^_^
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom