Ubuntu 20.04 VM boot disk corruption(?)

[Stark-Johan]

I've had a bunch of Ubuntu VMs running for a couple of years without any big issues until yesterday. One of my VM's running Plex Media Server started acting strange. The plexmediaserver service went down and I wasn't able to do apt update/upgrade with the following error:

dpkg: unrecoverable fatal error, aborting:
 files list file for package 'linux-generic' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

I tried a simple reboot and the VM would not recognise the boot disk anymore. "Boot failed, not a bootable disk". I checked the boot order which was still correct. I launched an ubuntu server live cd and found that the GPT is damaged. At this point a took a backup of the VM. Unfortunately the only backup that exist other than this one is a very old ZFS snapshot and an even older VM Backup.



I tried a recovery of the gpt using these instructions https://lihashgnis.blogspot.com/2016/07/recovering-from-corrupted-gpt-partition_30.html and the gpt seemed OK after that. I booted into gparted live and found that the two partitions were now visible, the second one being ext4, the first unrecognised but with a boot_grub flag. I did a "check" of the ext4 fs and is was successfully repaired. I then mounted it but found no data what so ever except "lost+found". This wasn't a fun discovery but seems very weird to me.

Using the ubuntu server live cd I manages to reinstall grub2 and got to the grub-recovery prompt. Unfortunately I can't get any further from there as the ext4 partition seems to be completely wiped. This is where I'm at right now. Any advice?

Some info on the setup and the VM:
There's only one disk on this VM and it is located on an nvme zfs pool.
Host: pve-manager/6.4-13/9f411e79 (running kernel: 5.4.140-1-pve)
VM OS: Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-86-generic x86_64)

cat /etc/pve/qemu-server/101.conf
agent: 1
boot: cdn
bootdisk: scsi0
cores: 8
hostpci0: 03:00,pcie=1
ide2: local:iso/ubuntu-20.04.3-live-server-amd64.iso,media=cdrom,size=1231808K
machine: q35
memory: 20480
name: PlexPass
net0: virtio=7A:56:98:0A:35:4D,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: nvmepool:vm-101-disk-0,size=384G
scsihw: virtio-scsi-pci
smbios1: uuid=7ea60e16-eefb-4d3f-ab3f-d1aa3d816667
sockets: 1
startup: order=2
vmgenid: 6c45f813-1265-4a6b-96dd-d3ce040791a1

 

[Stark-Johan]

I also tried the ubuntu live cd advanced option "rescue/enable=true" but that just resulted in a kernel panic as per the attached image.

 

[Stark-Johan]

The important part is to recover the ext4 file system. If I run "e2fsck" I end up with the same thing as with "check" using gparted, a mountable partition but no data except for the lost+found.

Here's some info on the fs:

root@ubuntu-server:/# dumpe2fs /dev/sda2
dumpe2fs 1.45.5 (07-Jan-2020)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          7d022b86-b4d4-4d22-bcb6-5977773abba0
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              25165824
Block count:              100662528
Reserved block count:     5033125
Free blocks:              39790167
Free inodes:              24492573
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1008
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat Feb  1 22:51:17 2020
Last mount time:          Sat Sep 25 13:36:55 2021
Last write time:          Mon Sep 27 20:08:07 2021
Mount count:              163
Maximum mount count:      -1
Last checked:             Mon Feb 24 08:57:39 2020
Check interval:           0 (<none>)
Lifetime writes:          10 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      0255b775-2b1e-4d0a-ac10-455556bda138
Journal backup:           inode blocks
FS Error count:           5
First error time:         Mon Sep 27 19:27:04 2021
First error function:     ext4_validate_block_bitmap
First error line #:       384
First error inode #:      0
First error block #:      0
Last error time:          Mon Sep 27 20:19:50 2021
Last error function:      __ext4_find_entry
Last error line #:        1541
Last error inode #:       2
Last error block #:       0
Checksum type:            crc32c
Checksum:                 0x823fdb52
dumpe2fs: Corrupt group descriptor: bad block for inode table while reading journal inode

 

[Stark-Johan]

Using info from this link I seem to have managed to locate the correct lost+found folder to access the data I needed from the ext4 partition. Not the most clean solution but I think it will do the trick this time. Recovering now, time will tell if this is enough.

https://techcult.com/how-to-restore-files-from-lostfound/

A bit alarming that the vm disk just got totally messed up like this with no apparent reason and no major, recent changes to the VM at all...