[SOLVED] Proxmox kernel panic

[borjaevo]

I have a server running proxmox 5.4
The server is a proliant dual cpu and 64gb RAM. Today we had a power problem and the SAI coundnt provide power for so long.
After restarting it can no longer boot and throws a kernel panic
The error it shows is "Initramfs unpacking failed: no cpio magic"

I have no idea how can i solve the problem

 

[Philipp Hufnagl]

First of. Your Proxmox version is End of Life. I highly encourage to upgrade to newer one for security reasons!

Next it appears like your initramfs has been corrupted. Do you have other kernels to choose from in your boot loader when you start? If not you will have to start your system from a live cd (for example Debian or Ubuntu).

 

[Philipp Hufnagl]

If you need to fix your initramfs from a live cd try

    sudo mkdir /media/RESCUE
    sudo mount /dev/pve/root /media/RESCUE/
    sudo mount -t proc proc /media/RESCUE/proc
    sudo mount -t sysfs sys /media/RESCUE/sys
    sudo mount -o bind /dev /media/RESCUE/dev
    sudo mount -o bind /run /media/RESCUE/run
    chroot /media/RESCUE
    update-initramfs -u
    exit
    reboot

 

[borjaevo]

First of. Your Proxmox version is End of Life. I highly encourage to upgrade to newer one for security reasons!

Next it appears like your initramfs has been corrupted. Do you have other kernels to choose from in your boot loader when you start? If not you will have to start your system from a live cd (for example Debian or Ubuntu).

If you need to fix your initramfs from a live cd try

    sudo mkdir /media/RESCUE
    sudo mount /dev/pve/root /media/RESCUE/
    sudo mount -t proc proc /media/RESCUE/proc
    sudo mount -t sysfs sys /media/RESCUE/sys
    sudo mount -o bind /dev /media/RESCUE/dev
    sudo mount -o bind /run /media/RESCUE/run
    chroot /media/RESCUE
    update-initramfs -u
    exit
    reboot

Thank you!

Its a little bit more complicated because i have the SO partition in RAID1 ZFS. I tried to import the pool from Ubuntu but i couldn't. I use a USB flash drive to boot. The img initramfs is there...

 

[Philipp Hufnagl]

In theory, after chroot you should be able to operate your operating system in the terminal like you have booted it normally

Are you sure you are looking at the right initramfs? Does it have *-pve in the name?

Even if it is there it might still be corrupted. You can move it to some other name and update after if you dont want to overwrite it.

 

[borjaevo]

In theory, after chroot you should be able to operate your operating system in the terminal like you have booted it normally

Are you sure you are looking at the right initramfs? Does it have *-pve in the name?

Even if it is there it might still be corrupted. You can move it to some other name and update after if you dont want to overwrite it.

This is what i have in the boot partition (picture attached). So i have the system in the ZFS RAID partition, i cannot mount the folders necesary to update initramfs. So i'm trying to import the zfs pools.

 

[Philipp Hufnagl]

Im not sure what i am looking at. It appers that you mounted your boot partition to /mnt ? Did you chroot?

Can I see the output of lsblk and mount to get to know your system better?

 

[borjaevo]

Im not sure what i am looking at. It appers that you mounted your boot partition to /mnt ? Did you chroot?

Can I see the output of lsblk and mount to get to know your system better?

I understand that you are not sure about what are you looking at, because i also dont understand my own environment

My boot partition is on a USB flash drive, and it is supposed to load rpoool that is a ZFS raid 1 pool that contains the root filesystem. I achieved to mount this root filesystem partition on an ubuntu live. But i checked this partition and i have a /boot directory there that has an initrd image 5.13 insted of 5.4 so i don't understand...
I will try to boot directly into one of the disks that contains this ZFS volume so maybe it has a MBR capable of booting the ZFS root filesystem... im not sure.
Attached lsblk output picture.

 

[borjaevo]

lsblk -f

 

[Philipp Hufnagl]

I think what you see at /boot is not your proxmox but rather your ubuntu live system.

What are we trying to do is mounting your system to /media/RESCUE and then tell your shell your file system root (/) is changed to /media/RESCUE (thats what chroot does)

to do that we first have to mount important directories of your file system to /media/RESCUE

sudo mkdir /media/RESCUE
sudo mount /dev/pve/root /media/RESCUE/
sudo mount /dev/my-usb /media/RESCUE/boot
sudo mount -t proc proc /media/RESCUE/proc
sudo mount -t sysfs sys /media/RESCUE/sys
sudo mount -o bind /dev /media/RESCUE/dev
sudo mount -o bind /run /media/RESCUE/run

Then we change the / in the terminal to /media/RESCUE

chroot /media/RESCUE

lastly we can operate our proxmox as if we would have booted. We can for example regenerate the initramfs with

update-initramfs -u

when we are done we can close our chroot environment and reboot with

exit
reboot

If there is anything unclear please ask

 

[borjaevo]

I think what you see at /boot is not your proxmox but rather your ubuntu live system.

What are we trying to do is mounting your system to /media/RESCUE and then tell your shell your file system root (/) is changed to /media/RESCUE (thats what chroot does)

to do that we first have to mount important directories of your file system to /media/RESCUE

sudo mkdir /media/RESCUE
sudo mount /dev/pve/root /media/RESCUE/
sudo mount /dev/my-usb /media/RESCUE/boot
sudo mount -t proc proc /media/RESCUE/proc
sudo mount -t sysfs sys /media/RESCUE/sys
sudo mount -o bind /dev /media/RESCUE/dev
sudo mount -o bind /run /media/RESCUE/run

Then we change the / in the terminal to /media/RESCUE

chroot /media/RESCUE

lastly we can operate our proxmox as if we would have booted. We can for example regenerate the initramfs with

update-initramfs -u

when we are done we can close our chroot environment and reboot with

exit
reboot

If there is anything unclear please ask

Hi Philipp,

Yes, i understand. But so i have the root filesystem in ZFS and the boot partition in a flash drive outside the root filesystem i tried other things first. I replaced the img file of the boot partition with the ones in boot.old folder of root filesystem and i get the initramfs to load correctly. Problem now is is rpool is detecting is was last in used in other system. Not sure how is detecting that... the same message appeared in ubuntu when trying to mount it but you only have to use the -f option. Here, im not sure...
Picture attached

 

[Philipp Hufnagl]

Is this your issue?

 

[borjaevo]

I could execute command there... so i did "zpool

Is this your issue?

Yes! I didn't know i can execute command there. I did the zpool import -f and rebooted and i have the proxmox running again!! thank you very much for the help!

 

[Philipp Hufnagl]

You are welcome. Please mark the thread as solved to make it easier for the next person to find

 

[Philipp Hufnagl]

Also you should really consider upgrading! Your Proxmox version is End of Life

 

[borjaevo]

Also you should really consider upgrading! Your Proxmox version is End of Life

Yes, i will try now. But... so my boot partition is little bit strange i think, im worried. I have to be sure of what im doing...

Thanks!

 

[truealph]

If you need to fix your initramfs from a live cd try

    sudo mkdir /media/RESCUE
    sudo mount /dev/pve/root /media/RESCUE/
    sudo mount -t proc proc /media/RESCUE/proc
    sudo mount -t sysfs sys /media/RESCUE/sys
    sudo mount -o bind /dev /media/RESCUE/dev
    sudo mount -o bind /run /media/RESCUE/run
    chroot /media/RESCUE
    update-initramfs -u
    exit
    reboot

Hi! My Proxmox host fails to boot and stays stuck on "Loading initial RAMDisk". My host has been down for some time and I really do not want to wipe it. These suggestions look close to what I need to do to fix it, but my Proxmox install is located on a ZFS RAID 1 volume that I can import while running in the shell from the Proxmox install ISO. What do I need to change within these commands to make it work like you have suggested so that I can repair the kernel? Any help would be greatly appreciated!