Can't boot in 5.3.18-2-pve image

jedo19

Active Member
Nov 16, 2018
29
3
43
49
After upgrade from 5.3.18-1-pve to -2 version, system does't boot any more and crashes with kernel panic - no working init found, screenshot attached, I can still boot in 5.3.18-1-pve kernel, any thoughts on this how to fix it?

Borut
 

Attachments

  • proxmox-init-crash.jpg
    proxmox-init-crash.jpg
    208.6 KB · Views: 26
sadly the picture you posted does not show the relevant part (the light of the camera is hiding it)
 
Sorry, was in a rush, I rebooted now again and hopefully this one is better:
 

Attachments

  • kernel-1.jpg
    kernel-1.jpg
    915.1 KB · Views: 44
hmm - my guess is that the initramfs is corrupted (hence the inability to find 'init')

boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`
 
hmm - my guess is that the initramfs is corrupted (hence the inability to find 'init')

boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`

I can boot with prior kernel, will try this option and let you know ...
Thanks
 
  • Like
Reactions: Stoiko Ivanov
I have booted to old kernel and rerun update-initramfs ... but that didn't fix the issue still same "Failed to execute /init (error -2)
Any other ideas?

Thanks
Borut
 
And now I cannot even boot in the prior kernel and now I have an unbootable system because of this last exercise, please advise...
Borut
 
boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`

How can I create a bootable live cd / usb with latest ZFS as this is zfs root boot ?? Can you please help me out with this ...
 
How can I create a bootable live cd / usb with latest ZFS as this is zfs root boot ??
You can use the PVE installer in debug mode - the second debug shell should have the zfs userspace available and the kernel module is also present

is this an uefi system? - then you would also need to sync the initramfs etc to the ESPs with pve-efiboot-tool

I hope this helps!
 
You can use the PVE installer in debug mode - the second debug shell should have the zfs userspace available and the kernel module is also present

is this an uefi system? - then you would also need to sync the initramfs etc to the ESPs with pve-efiboot-tool

I hope this helps!
So I booted in debug mode
zfs import rpool -R /mnt -f
mount -t proc /proc /mnt/proc
mount --rbind /dev /mnt/dev
mount --rbind /sys /mnt/sys
chroot /mnt /bin/bash # Chroot into Proxmox

ran pve-efiboot-tool refresh

and still same thing ?

Am I missing something
?
 
I have now tried to install an older kernel from the repository but I just can't make the system boot, it always goes to /init problem, looks like some sort of fs problem as it regenerates the image but always something is just not right ....
I am really getting frustrated with this...
 
you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.
 
  • Like
Reactions: Stoiko Ivanov
you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.
Will do that later as I haven't got access to the machine now.

It is a UEFI based boot, as I can see new options when I add kernels but when I generate new ones with update-initramfs -k all -u and after I run pve-efiboot-tool refresh it finishes without any errors. After the machine restart, it gives me options to select the kernel but no matter which kernel I choose now I get to the same point of no return
 
you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.
I did lsinitramfs and it looks ok, at least "init" is in root folder there, I actuall don't know what I should be looking at ...
 
Hi Fabian, Stoiko, I am still going in circles, I can mount from debug mode but that is about it, all that you suggested did not work, the output of update-initramfs -u is as it should be and not reporting any errors, google doesn't say much about this and usual trick ( yours and picked up from web ) do not fix the issue help would be much appreciated.
Regards
B.
 
extract the initramfs, and check what 'init' is inside? ;)
 
By properly frustrating myself with this, I installed a second proxmox instance in a vm, and copied over initrd image (initrd.img-5.3.18-2-pve), went through usual step copying image with pve-efiboot-tool refresh and system booted afterwards. When I had live system back again I extracted both working and non working intrd image and made a md5sum diff :

Code:
root@ent:~/init# diff <(find 1/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /') <(find 2/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /')
8,11c8,11
< 8ac273081969ac3e66cfc63bf4ef8dee hostid
< e4e636027ef4bf3c9f411931e7dd7ce2 hostname
< 81cf7375ac642c516023cb67b0c5fa3b initiatorname.iscsi
< 2a3551da270b60b53703a0266372291a ld.so.cache
---
> 36b6783f6889514feae6a784b99e6c7e hostid
> a4309b13fb632be8ddea6e9139ef3268 hostname
> 0ecd7fea5d953c554e635055d1d27630 initiatorname.iscsi
> 7d268169569a3257961c8c51d142c1ae ld.so.cache
29a30
> 4d9fdd2bb856a4b4ca90f571b809b535 zpool.cache
361c362
< 37e1fd29ab50948e094a00c405043741 blacklist_pve-kernel-5.3.10-1-pve.conf
---
> 37e1fd29ab50948e094a00c405043741 blacklist_pve-kernel-5.3.18-1-pve.conf

Where 1 is now is a working kernel and 2 is a non working kernel. I see only a ld.so.cache differ everything else is I guess expected
Any thoughts on this?
 
then I'd next check if your ESP contents differ from what's in /boot. you can find a list of ESP UUIDs in /etc/kernle/pve-efiboot-uuids. e.g.,
Code:
while read u; do mkdir -p "/media/esp/$u"; mount "/dev/disk/by-uuid/$u" "/media/esp/$u"; done < /etc/kernel/pve-efiboot-uuids

will mount each of them in /media/esp/$UUID, and you could then check whether they lack space, contain corrupt/truncated initrds, etc.pp. don't forget to unmount them again once you are done with investigating.
 
then I'd next check if your ESP contents differ from what's in /boot. you can find a list of ESP UUIDs in /etc/kernle/pve-efiboot-uuids. e.g.,
Code:
while read u; do mkdir -p "/media/esp/$u"; mount "/dev/disk/by-uuid/$u" "/media/esp/$u"; done < /etc/kernel/pve-efiboot-uuids

will mount each of them in /media/esp/$UUID, and you could then check whether they lack space, contain corrupt/truncated initrds, etc.pp. don't forget to unmount them again once you are done with investigating.

Thanks Fabian,
After inspection nothing seemed to be wrong in EFI partitions, after that I cleaned all the kernels out of the way with apt-get remove --purge pve-kernel*=ver*, system still boots with copied initrd image, after that I tried to install 5.3.18-1-pve from repo and it is still happening, somehow intird image gets corrupted at build ...

Any other ideas ?
Borut

PS: copied kernel initrd image from the other machine still works, I just can't generate one on mine, maybe something is corrupted in my installation ...
 
Last edited:
hm - what's the diff of ld.so.cache (diff the output of `strings 1/etc/ld.so.cache` and `strings 2/etc/ld.so.cache`)

also - maybe install and run `debsums`
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!