Grub Rescue Recover Data

Discussion in 'Proxmox VE: Installation and configuration' started by Jonpaulh, Jun 12, 2019.

  1. Jonpaulh

    Jonpaulh New Member

    Joined:
    Nov 16, 2017
    Messages:
    12
    Likes Received:
    0
    Yesterday my electric was cut and my UPS's ran down, finally my proxmox server lost power. Once the power was restored all other server came back online except one. Currently when booting I get to the grub rescue shell.

    Code:
    error: unknown filesystem
    Entering rescue mode...
    
    I see the following with ls

    Code:
    grub rescue> ls
    (hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (lvm/pve-root) (lvm/pve-swap)
    
    I have tried doing an ls with each one and I get the same result regardless
    Code:
    grub rescue> ls (lvm/pve-root)
    (lvm/pve-root): Filesystem is unknown
    grub rescue> ls (lvm/pve-root)/
    error: unknown filesystem
    
    I have booted into a live ubuntu CD, checking fdisk -l

    Code:
    # fdisk -l
    Disk /dev/sda: 931.5GiB, 1000204886016 bytes, 1953525168 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physcial): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disklabel type: gpt
    Disk identifier: 7C6C8CF5-63CC-497D-A1D9-A3108D2E36DA
    
    Device        Start        End        Sectors        Size        Type
    /dev/sda1    34        2047        2014        10007K        BIOS boot
    /dev/sda2    2048        262143        260096        127M        EFI System
    /dev/sda3    262144        1953525134    1953262991    931.4G        Linux LVM
    
    Partition 1 does not start on a physical sector boundary
    
    Disk /dev/mapper/pve-swap: 8GiB, 8589934592 bytes, 16777216 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum,optimal): 4096 bytes / 4096 bytes
    
    Disk /dev/mapper/pve-root: 96 GiB, 103079215104 bytes, 201326592 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum,optimal): 4096 bytes / 4096 bytes
    
    Code:
    # parted /dev/sda p
    Model: ATA TOSHIBA HDWJ110 (scsi)
    Disk /dev/sda: 1000GB
    Sector size (logical/physical): 512B/4096B
    Partition Table: gpt
    Disk Flags:
    
    Number        start         End        size         File system     Name    Flags
    1             17.4kB        1049kB    1031kB                                bios_grub
    2             1049kB        134MB    133MB        fat32                    boot, esp
    3             134MB        1000GB    1000GB                                lvm
    
    There is some more info below, but I include the main parts
    Code:
    # pvs;lvs
    PV             VG         Fmt        Attr         PSize         PFree
    /dev/sda3    pve     lvm2     a--         <931.39g    15.79g
    LV             VG         Fmt        Attr         PSize         PFree
    data
    root
    swap
    vm-103-disk-1
    vm-104-disk-1
    vm-113-disk-1
    vm-134-disk-1
    
    It appears the disks are there, what would be the best way to backup the vm's or repair the disk?
     
    #1 Jonpaulh, Jun 12, 2019
    Last edited: Jun 12, 2019
  2. lhorace

    lhorace Member

    Joined:
    Oct 17, 2015
    Messages:
    127
    Likes Received:
    14
    Hello there,

    Have you actually try mounting the volumes with the Live ISO and physically check if the data is there?

    Regards
     
  3. Jonpaulh

    Jonpaulh New Member

    Joined:
    Nov 16, 2017
    Messages:
    12
    Likes Received:
    0


    I have tried a few more things and I include the outputs here

    Code:
    root@ubuntu-server:~# pvs
      WARNING: Not using lvmetad because a repair command was run.
      PV         VG  Fmt  Attr PSize    PFree
      /dev/sda3  pve lvm2 a--  <931.39g 15.79g
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvs
      WARNING: Not using lvmetad because a repair command was run.
      LV            VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      data          pve twi---tz-- 811.39g
      root          pve -wi-a-----  96.00g
      swap          pve -wi-a-----   8.00g
      vm-103-disk-1 pve Vwi---tz-- 200.00g data
      vm-104-disk-1 pve Vwi---tz-- 120.00g data
      vm-113-disk-1 pve Vwi---tz-- 200.00g data
      vm-134-disk-1 pve Vwi---tz-- 100.00g data
    root@ubuntu-server:~#
    root@ubuntu-server:~# mkdir /media/test
    root@ubuntu-server:~# mount /dev/v
    vcs          vcs2         vcs4         vcs6         vcsa1        vcsa3        vcsa5        vfio/        vhci         vhost-vsock
    vcs1         vcs3         vcs5         vcsa         vcsa2        vcsa4        vcsa6        vga_arbiter  vhost-net
    root@ubuntu-server:~# mount /dev/sda /media/test/
    mount: /media/test: /dev/sda already mounted or mount point busy.
    root@ubuntu-server:~# mount /dev/sda3 /media/test/
    mount: /media/test: unknown filesystem type 'LVM2_member'.
    root@ubuntu-server:~#
    root@ubuntu-server:~# vgchange -a y pve
      WARNING: Not using lvmetad because a repair command was run.
      /usr/sbin/thin_check: execvp failed: No such file or directory
      Check of pool pve/data failed (status:2). Manual repair required!
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      2 logical volume(s) in volume group "pve" now active
    root@ubuntu-server:~#
    root@ubuntu-server:~# pvscan
      PV /dev/sda3   VG pve             lvm2 [<931.39 GiB / 15.79 GiB free]
      Total: 1 [<931.39 GiB] / in use: 1 [<931.39 GiB] / in no VG: 0 [0   ]
    root@ubuntu-server:~#
    root@ubuntu-server:~# pvdisplay
      --- Physical volume ---
      PV Name               /dev/sda3
      VG Name               pve
      PV Size               <931.39 GiB / not usable 1.69 MiB
      Allocatable           yes
      PE Size               4.00 MiB
      Total PE              238435
      Free PE               4043
      Allocated PE          234392
      PV UUID               ie3JBl-wEPn-2bOM-1W22-mNyv-nhp2-SbIBCM
    
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvscan
      ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
      ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
      inactive          '/dev/pve/data' [811.39 GiB] inherit
      inactive          '/dev/pve/vm-134-disk-1' [100.00 GiB] inherit
      inactive          '/dev/pve/vm-104-disk-1' [120.00 GiB] inherit
      inactive          '/dev/pve/vm-103-disk-1' [200.00 GiB] inherit
      inactive          '/dev/pve/vm-113-disk-1' [200.00 GiB] inherit
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvdisplay
      --- Logical volume ---
      LV Path                /dev/pve/swap
      LV Name                swap
      VG Name                pve
      LV UUID                aXsqn2-6nBs-n2u5-s40U-n09N-2wSr-c6jvzs
      LV Write Access        read/write
      LV Creation host, time proxmox, 2016-11-10 09:08:42 +0000
      LV Status              available
      # open                 0
      LV Size                8.00 GiB
      Current LE             2048
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           253:0
    
      --- Logical volume ---
      LV Path                /dev/pve/root
      LV Name                root
      VG Name                pve
      LV UUID                kHE9Oj-ST01-jcyy-UqwJ-ZpXN-Oaon-vr1VYO
      LV Write Access        read/write
      LV Creation host, time proxmox, 2016-11-10 09:08:43 +0000
      LV Status              available
      # open                 0
      LV Size                96.00 GiB
      Current LE             24576
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           253:1
    
      --- Logical volume ---
      LV Name                data
      VG Name                pve
      LV UUID                AC6UqS-o0ZC-UwUy-MDRZ-BFYc-3OfO-7VV35K
      LV Write Access        read/write
      LV Creation host, time proxmox, 2016-11-10 09:08:45 +0000
      LV Pool metadata       data_tmeta
      LV Pool data           data_tdata
      LV Status              NOT available
      LV Size                811.39 GiB
      Current LE             207716
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-134-disk-1
      LV Name                vm-134-disk-1
      VG Name                pve
      LV UUID                VZI8m5-D7rd-ybcC-yOnC-C6Lj-yHgH-N2KH4J
      LV Write Access        read/write
      LV Creation host, time vm06, 2017-10-24 08:59:08 +0000
      LV Pool name           data
      LV Status              NOT available
      LV Size                100.00 GiB
      Current LE             25600
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-104-disk-1
      LV Name                vm-104-disk-1
      VG Name                pve
      LV UUID                wErXRU-moVR-Ya2T-qxJt-M4T8-nzGe-Tk1nT5
      LV Write Access        read/write
      LV Creation host, time vm06, 2018-06-20 13:54:47 +0000
      LV Pool name           data
      LV Status              NOT available
      LV Size                120.00 GiB
      Current LE             30720
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-103-disk-1
      LV Name                vm-103-disk-1
      VG Name                pve
      LV UUID                Ks1lyq-fgAo-8Pqj-DGrF-91hh-ApDS-lRo3uP
      LV Write Access        read/write
      LV Creation host, time vm06, 2018-06-20 14:06:14 +0000
      LV Pool name           data
      LV Status              NOT available
      LV Size                200.00 GiB
      Current LE             51200
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-113-disk-1
      LV Name                vm-113-disk-1
      VG Name                pve
      LV UUID                JrMvsW-9s0p-4hTU-LHCO-PErZ-dQ00-zIxNOL
      LV Write Access        read/write
      LV Creation host, time vm06, 2019-04-25 11:10:06 +0000
      LV Pool name           data
      LV Status              NOT available
      LV Size                200.00 GiB
      Current LE             51200
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
    root@ubuntu-server:~#
    root@ubuntu-server:~# lsblk
    NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    loop0          7:0    0 495.4M  1 loop /media/filesystem
    loop1          7:1    0 149.5M  1 loop
    loop2          7:2    0    37M  1 loop /media/region.lower
    loop3          7:3    0  21.5M  1 loop /media/rack.lower
    loop4          7:4    0  86.6M  1 loop /snap/core/4486
    loop5          7:5    0  44.6M  1 loop /snap/subiquity/346
    sda            8:0    0 931.5G  0 disk
    ├─sda1         8:1    0  1007K  0 part
    ├─sda2         8:2    0   127M  0 part
    └─sda3         8:3    0 931.4G  0 part
      ├─pve-swap 253:0    0     8G  0 lvm
      └─pve-root 253:1    0    96G  0 lvm
    sr0           11:0    1   806M  0 rom  /cdrom
    root@ubuntu-server:~#
    root@ubuntu-server:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            7.8G     0  7.8G   0% /dev
    tmpfs           1.6G  1.4M  1.6G   1% /run
    /dev/sr0        806M  806M     0 100% /cdrom
    /dev/loop0      496M  496M     0 100% /rofs
    /cow            7.9G  166M  7.7G   3% /
    tmpfs           7.9G     0  7.9G   0% /dev/shm
    tmpfs           5.0M     0  5.0M   0% /run/lock
    tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
    tmpfs           7.9G     0  7.9G   0% /tmp
    /dev/loop3       22M   22M     0 100% /media/rack.lower
    overlay          22M   22M     0 100% /media/rack
    /dev/loop2       38M   38M     0 100% /media/region.lower
    overlay          38M   38M     0 100% /media/region
    /dev/loop4       87M   87M     0 100% /snap/core/4486
    /dev/loop5       45M   45M     0 100% /snap/subiquity/346
    tmpfs           1.6G     0  1.6G   0% /run/user/999
    tmpfs           1.6G     0  1.6G   0% /run/user/0
    root@ubuntu-server:~#
    root@ubuntu-server:~# vgchange -a y pve
      /usr/sbin/thin_check: execvp failed: No such file or directory
      Check of pool pve/data failed (status:2). Manual repair required!
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      /usr/sbin/thin_check: execvp failed: No such file or directory
      2 logical volume(s) in volume group "pve" now active
    root@ubuntu-server:~#
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvchange -a y pve/data
      /usr/sbin/thin_check: execvp failed: No such file or directory
      Check of pool pve/data failed (status:2). Manual repair required!
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvconvert --repair pve/data
      WARNING: Disabling lvmetad cache for repair command.
      WARNING: Not using lvmetad because of repair.
      /usr/sbin/thin_repair: execvp failed: No such file or directory
      Repair of thin metadata volume of thin pool pve/data failed (status:2). Manual repair required!
    root@ubuntu-server:~#
    root@ubuntu-server:~# lvs -a
      WARNING: Not using lvmetad because a repair command was run.
      LV              VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      data            pve twi---tz-- 811.39g
      [data_tdata]    pve Twi------- 811.39g
      [data_tmeta]    pve ewi------- 104.00m
      [lvol0_pmspare] pve ewi------- 104.00m
      root            pve -wi-a-----  96.00g
      swap            pve -wi-a-----   8.00g
      vm-103-disk-1   pve Vwi---tz-- 200.00g data
      vm-104-disk-1   pve Vwi---tz-- 120.00g data
      vm-113-disk-1   pve Vwi---tz-- 200.00g data
      vm-134-disk-1   pve Vwi---tz-- 100.00g data
    root@ubuntu-server:~#
    
    
     
  4. lhorace

    lhorace Member

    Joined:
    Oct 17, 2015
    Messages:
    127
    Likes Received:
    14
    It might just be the metadata that is corrupt, did it shut down safely or just abrupt power cut? With that said, I pretty sure it's easy to create a new metadata, how? I have frequent power cuts without UPC and I've never personally experience this myself, but I am pretty sure the data is intact, you probably need just to re the metadata. That LVM needs!
     
    Jonpaulh likes this.
  5. Jonpaulh

    Jonpaulh New Member

    Joined:
    Nov 16, 2017
    Messages:
    12
    Likes Received:
    0
    The power just cut so it was abrupt. I have seend this a couple of times with power loss, I assume during a write operation when the power cuts it causes a corruption. The output does make me hopeful I can recover somehow, but I cannot find anything on how to proceed. I will look at metadata and how to create, any pointers or links would be greatly appreciated. Many thanks for taking the time to respond.
     
  6. lhorace

    lhorace Member

    Joined:
    Oct 17, 2015
    Messages:
    127
    Likes Received:
    14
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice