Can't boot in 5.3.18-2-pve image

[jedo19]

After upgrade from 5.3.18-1-pve to -2 version, system does't boot any more and crashes with kernel panic - no working init found, screenshot attached, I can still boot in 5.3.18-1-pve kernel, any thoughts on this how to fix it?

Borut

 

[Stoiko Ivanov]

sadly the picture you posted does not show the relevant part (the light of the camera is hiding it)

 

[jedo19]

Sorry, was in a rush, I rebooted now again and hopefully this one is better:

 

[Stoiko Ivanov]

hmm - my guess is that the initramfs is corrupted (hence the inability to find 'init')

boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`

 

[jedo19]

hmm - my guess is that the initramfs is corrupted (hence the inability to find 'init')

boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`

I can boot with prior kernel, will try this option and let you know ...
Thanks

 

[jedo19]

I have booted to old kernel and rerun update-initramfs ... but that didn't fix the issue still same "Failed to execute /init (error -2)
Any other ideas?

Thanks
Borut

 

[jedo19]

And now I cannot even boot in the prior kernel and now I have an unbootable system because of this last exercise, please advise...
Borut

 

[jedo19]

boot a live-cd - chroot into the installation (with all necessary bind-mounts) and run `update-initramfs -k all -u`

How can I create a bootable live cd / usb with latest ZFS as this is zfs root boot ?? Can you please help me out with this ...

 

[Stoiko Ivanov]

How can I create a bootable live cd / usb with latest ZFS as this is zfs root boot ??

You can use the PVE installer in debug mode - the second debug shell should have the zfs userspace available and the kernel module is also present

is this an uefi system? - then you would also need to sync the initramfs etc to the ESPs with pve-efiboot-tool

I hope this helps!

 

[jedo19]

You can use the PVE installer in debug mode - the second debug shell should have the zfs userspace available and the kernel module is also present

is this an uefi system? - then you would also need to sync the initramfs etc to the ESPs with pve-efiboot-tool

I hope this helps!

So I booted in debug mode
zfs import rpool -R /mnt -f
mount -t proc /proc /mnt/proc
mount --rbind /dev /mnt/dev
mount --rbind /sys /mnt/sys
chroot /mnt /bin/bash # Chroot into Proxmox

ran pve-efiboot-tool refresh

and still same thing ?

Am I missing something
?

 

[jedo19]

I have now tried to install an older kernel from the repository but I just can't make the system boot, it always goes to /init problem, looks like some sort of fs problem as it regenerates the image but always something is just not right ....
I am really getting frustrated with this...

 

[fabian]

you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.

 

[jedo19]

you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.

Will do that later as I haven't got access to the machine now.

It is a UEFI based boot, as I can see new options when I add kernels but when I generate new ones with update-initramfs -k all -u and after I run pve-efiboot-tool refresh it finishes without any errors. After the machine restart, it gives me options to select the kernel but no matter which kernel I choose now I get to the same point of no return

 

[jedo19]

you can check the content of initramfs with "lsinitramfs", and extract it with "unmkinitramfs". it might also help if you post the output of update-initramfs -u, and state whether you are booting using UEFI or legacy/BIOS boot.

I did lsinitramfs and it looks ok, at least "init" is in root folder there, I actuall don't know what I should be looking at ...

 

[jedo19]

Hi Fabian, Stoiko, I am still going in circles, I can mount from debug mode but that is about it, all that you suggested did not work, the output of update-initramfs -u is as it should be and not reporting any errors, google doesn't say much about this and usual trick ( yours and picked up from web ) do not fix the issue help would be much appreciated.
Regards
B.

 

[fabian]

extract the initramfs, and check what 'init' is inside?

 

[jedo19]

By properly frustrating myself with this, I installed a second proxmox instance in a vm, and copied over initrd image (initrd.img-5.3.18-2-pve), went through usual step copying image with pve-efiboot-tool refresh and system booted afterwards. When I had live system back again I extracted both working and non working intrd image and made a md5sum diff :

root@ent:~/init# diff <(find 1/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /') <(find 2/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /')
8,11c8,11
< 8ac273081969ac3e66cfc63bf4ef8dee hostid
< e4e636027ef4bf3c9f411931e7dd7ce2 hostname
< 81cf7375ac642c516023cb67b0c5fa3b initiatorname.iscsi
< 2a3551da270b60b53703a0266372291a ld.so.cache
---
> 36b6783f6889514feae6a784b99e6c7e hostid
> a4309b13fb632be8ddea6e9139ef3268 hostname
> 0ecd7fea5d953c554e635055d1d27630 initiatorname.iscsi
> 7d268169569a3257961c8c51d142c1ae ld.so.cache
29a30
> 4d9fdd2bb856a4b4ca90f571b809b535 zpool.cache
361c362
< 37e1fd29ab50948e094a00c405043741 blacklist_pve-kernel-5.3.10-1-pve.conf
---
> 37e1fd29ab50948e094a00c405043741 blacklist_pve-kernel-5.3.18-1-pve.conf

Where 1 is now is a working kernel and 2 is a non working kernel. I see only a ld.so.cache differ everything else is I guess expected
Any thoughts on this?

 

[fabian]

then I'd next check if your ESP contents differ from what's in /boot. you can find a list of ESP UUIDs in /etc/kernle/pve-efiboot-uuids. e.g.,

while read u; do mkdir -p "/media/esp/$u"; mount "/dev/disk/by-uuid/$u" "/media/esp/$u"; done < /etc/kernel/pve-efiboot-uuids

will mount each of them in /media/esp/$UUID, and you could then check whether they lack space, contain corrupt/truncated initrds, etc.pp. don't forget to unmount them again once you are done with investigating.

 

[jedo19]

then I'd next check if your ESP contents differ from what's in /boot. you can find a list of ESP UUIDs in /etc/kernle/pve-efiboot-uuids. e.g.,

while read u; do mkdir -p "/media/esp/$u"; mount "/dev/disk/by-uuid/$u" "/media/esp/$u"; done < /etc/kernel/pve-efiboot-uuids

will mount each of them in /media/esp/$UUID, and you could then check whether they lack space, contain corrupt/truncated initrds, etc.pp. don't forget to unmount them again once you are done with investigating.

Thanks Fabian,
After inspection nothing seemed to be wrong in EFI partitions, after that I cleaned all the kernels out of the way with apt-get remove --purge pve-kernel*=ver*, system still boots with copied initrd image, after that I tried to install 5.3.18-1-pve from repo and it is still happening, somehow intird image gets corrupted at build ...

Any other ideas ?
Borut

PS: copied kernel initrd image from the other machine still works, I just can't generate one on mine, maybe something is corrupted in my installation ...

 

[Stoiko Ivanov]

hm - what's the diff of ld.so.cache (diff the output of `strings 1/etc/ld.so.cache` and `strings 2/etc/ld.so.cache`)

also - maybe install and run `debsums`