[SOLVED] Kernel panic PVE 6.0.1 - reset kernel

[Al_mageddon]

Hi guys,

I need your help. My PVE 6.0.1 boots but I have a kernel panic which totally stop the boot. I want use chroot in order to reinstall the kernel 5.0.21 but I have an error.
Look at my shell commands.


root@debian:~# uname -r
4.19.0-6-amd64 ( I used LiveCD Debian 10 )



root@debian:~# fdisk -lu

Disk /dev/nvme0n1: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 970 EVO Plus 500GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3AC6C19B-09C1-4877-821F-21BEE6E3F8D8

Device Start End Sectors Size Type
/dev/nvme0n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme0n1p2 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p3 1050624 976773134 975722511 465.3G Linux LVM


Disk /dev/sda: 14.4 GiB, 15472047104 bytes, 30218842 sectors
Disk model: DataTraveler 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x01f04019

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 28121656 28119609 13.4G c W95 FAT32 (LBA)
/dev/sda2 28121657 30218800 2097144 1024M 83 Linux


Disk /dev/loop0: 2.1 GiB, 2277298176 bytes, 4447848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-swap: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-root: 46 GiB, 49383735296 bytes, 96452608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-vm--101--disk--0: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x90909090

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--101--disk--0-part1 * 64 20971518 20971455 10G a5 Free

Partition 1 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--100--disk--0: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x90909090

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--100--disk--0-part1 * 63 20964824 20964762 10G a5 Free

Partition 1 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--103--disk--0: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xa14d1b8f

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--103--disk--0-part1 * 2048 18874367 18872320 9G 83 Linu
/dev/mapper/pve-vm--103--disk--0-part2 18876414 20969471 2093058 1022M 5 Exte
/dev/mapper/pve-vm--103--disk--0-part5 18876416 20969471 2093056 1022M 82 Linu

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--104--disk--0: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--105--disk--0: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--106--disk--0: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--102--disk--0: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--102--disk--1: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--107--disk--0: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--110--disk--0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 248B1987-03F5-11EA-B31F-7D3559FEDCB6

Device Start End Sectors Size Type
/dev/mapper/pve-vm--110--disk--0-part1 64 1087 1024 512K FreeBSD
/dev/mapper/pve-vm--110--disk--0-part2 2048 8390655 8388608 4G FreeBSD
/dev/mapper/pve-vm--110--disk--0-part3 8390656 10487807 2097152 1G FreeBSD
/dev/mapper/pve-vm--110--disk--0-part4 10487808 31991807 21504000 10.3G FreeBSD

Partition 1 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--107--disk--1: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--107--disk--2: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes




root@debian:~# pvscan
PV /dev/nvme0n1p3 VG pve lvm2 [<465.26 GiB / 0 free]
Total: 1 [<465.26 GiB] / in use: 1 [<465.26 GiB] / in no VG: 0 [0 ]



root@debian:~# vgscan
Reading all physical volumes. This may take a while...
Found volume group "pve" using metadata type lvm2



root@debian:~# vgchange -a y
15 logical volume(s) in volume group "pve" now active



root@debian:~# lvscan
ACTIVE '/dev/pve/swap' [4.00 GiB] inherit
ACTIVE '/dev/pve/root' [45.99 GiB] inherit
ACTIVE '/dev/pve/data' [406.95 GiB] inherit
ACTIVE '/dev/pve/vm-101-disk-0' [10.00 GiB] inherit
ACTIVE '/dev/pve/vm-100-disk-0' [15.00 GiB] inherit
ACTIVE '/dev/pve/vm-103-disk-0' [10.00 GiB] inherit
ACTIVE '/dev/pve/vm-104-disk-0' [10.00 GiB] inherit
ACTIVE '/dev/pve/vm-105-disk-0' [8.00 GiB] inherit
ACTIVE '/dev/pve/vm-106-disk-0' [8.00 GiB] inherit
ACTIVE '/dev/pve/vm-102-disk-0' [15.00 GiB] inherit
ACTIVE '/dev/pve/vm-102-disk-1' [15.00 GiB] inherit
ACTIVE '/dev/pve/vm-107-disk-0' [15.00 GiB] inherit
ACTIVE '/dev/pve/vm-110-disk-0' [20.00 GiB] inherit
ACTIVE '/dev/pve/vm-107-disk-1' [15.00 GiB] inherit
ACTIVE '/dev/pve/vm-107-disk-2' [15.00 GiB] inherit



root@debian:~# mount /dev/pve/root /mnt

root@debian:~# chroot /mnt ( for test only )
chroot: failed to run command ‘/bin/bash’: No such file or directory

root@debian:~# /sbin/chroot /mnt ( for test only )
/sbin/chroot: failed to run command ‘/bin/bash’: No such file or directory



root@debian:~# mount --bind /dev /mnt/dev
root@debian:~# mount --bind /proc /mnt/proc
root@debian:~# mount --bind /sys /mnt/sys

root@debian:~# chroot /mnt ( for test only )
chroot: failed to run command ‘/bin/bash’: No such file or directory

root@debian:~# /sbin/chroot /mnt ( for test only )
/sbin/chroot: failed to run command ‘/bin/bash’: No such file or directory



root@debian:~# mount /dev/nvme0n1p2 /mnt/boot/efi

root@debian:~# chroot /mnt
chroot: failed to run command ‘/bin/bash’: No such file or directory

root@debian:~# /sbin/chroot /mnt
/sbin/chroot: failed to run command ‘/bin/bash’: No such file or directory



root@debian:~# mount /dev/nvme0n1p1 /mnt/boot
mount: /mnt/boot: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.

root@debian:~# chroot /mnt /bin/bash
chroot: failed to run command ‘/bin/bash’: No such file or directory


I don't understand how works the start of PVE, it's why I tested some command (https://gist.github.com/uglide/5f6f23903dac6b5f2567).
However, I know that chroot works because commands debootstrap --include=locales-all wheezy /mnt/TEST http://ftp.fr.debian.org/debian then chroot /mnt/TEST works very well.

What is my mistake ? Thank you for your help.

 

[Stefan_R]

First of all, what is your kernel panic? Do you have any log output?

Your chroot fails, because /bin/bash is missing from your mounted filesystem, i.e. there is no /mnt/bin/bash. Have you tried just cd'ing into /mnt after mounting /dev/pve/root and running an 'ls'? Or 'ls -la /bin' and check if there is even a bash installed?

 

[Al_mageddon]

Hi,

My startup screen is on the attached pictures. The last picture is the panic.

About the /bin/bash, here the desired commands:

user@debian:~$ su
Password:

root@debian:/home/user# cd
root@debian:~# mount /dev/pve/root /mnt
root@debian:~# cd /mnt
root@debian:/mnt# ls
bin dev home lib32 libx32 media opt root sbin sys tmp var
boot etc lib lib64 lost+found mnt proc run srv TEST usr
root@debian:/mnt# cd /bin
root@debian:/bin# ls
'[' nice
2to3-2.7 nisdomainname
7z nkf
7za nl
7zr nm
aa-enabled nm-applet
aa-exec nmcli
aconnect nm-connection-editor
acpi_listen nm-online
add-apt-repository nmtui
add-calamares-desktop-icon nmtui-connect
addpart nmtui-edit
addr2line nmtui-hostname
alsabat nohup
alsaloop nproc
alsamixer nroff
alsatplg nsenter
alsaucm nstat
amidi ntfs-3g
amixer ntfs-3g.probe
anthy-agent ntfscat
anthy-dic-tool ntfscluster
apg ntfscmp
apgbfm ntfsdecrypt
aplay ntfsfallocate
aplaymidi ntfsfix
applycal ntfsinfo
appres ntfsls
appstreamcli ntfsmove
apropos ntfsrecover
apt ntfssecaudit
apt-add-repository ntfstruncate
apt-cache ntfsusermap
apt-cdrom ntfswipe
apt-config numfmt
apt-extracttemplates obexctl
apt-ftparchive objcopy
apt-get objdump
apt-key oclock
apt-mark od
apt-sortpkgs odp2pdf
ar odp2ppt
arch ods2pdf
arch-test odt2bib
arecord odt2doc
arecordmidi odt2docbook
as odt2html
aseqdump odt2lt
aseqnet odt2pdf
aspell odt2rtf
aspell-import odt2sdw
assistant odt2sxw
atobm odt2txt
average odt2txt.unoconv
awk odt2xhtml
axfer odt2xml
b2sum oeminst
baobab ooxml2doc
base32 ooxml2odt
base64 ooxml2pdf
basename opencc
bash opencc_dict
bashbug opencc_phrase_extract
bccmd openssl
bdftopcf openvt
bdftruncate orca
bf_compact orca-dm-wrapper
bf_compact-bdb os-prober
bf_copy p11-kit
bf_copy-bdb p7zip
bf_tar pacat
bf_tar-bdb pacmd
bidiv pactl
bitmap padsp
bluemoon pager
bluetoothctl pamon
bluetooth-sendto paperconf
bmtoa paplay
bogofilter parec
bogofilter-bdb parecord
bogolexer partx
bogolexer-bdb passwd
bogotune paste
bogotune-bdb pasuspender
bogoupgrade patch
bogoupgrade-bdb pathchk
bogoutil pax11publish
bogoutil-bdb pcimodules
boltctl pdb
bootctl pdb2
broadwayd pdb2.7
bsd-from pdb3
bsd-write pdb3.7
btattach peekfd
btmgmt perf
btmon perl
btrfs perl5.28.1
btrfsck perl5.28-x86_64-linux-gnu
btrfs-find-root perlbug
btrfs-image perldoc
btrfs-map-logical perli11ndoc
btrfs-select-super perlivp
btrfstune perlthanks
buildhash pgrep
bunzip2 pic
busctl pico
busybox piconv
bwrap pidof
bzcat pinentry
bzcmp pinentry-curses
bzdiff pinentry-gnome3
bzegrep pinentry-x11
bzexe ping
bzfgrep ping4
bzgrep ping6
bzip2 pinky
bzip2recover pixeltool
bzless pkaction
bzmore pkcheck
c++ pkcon
c89 pkexec
c89-gcc pkill
c99 pkmon
c99-gcc pkttyagent
cal pl2pm
calamares pldd
calendar plog
captoinfo plymouth
caribou-preferences pmap
cat pod2html
catchsegv pod2man
catman pod2text
cb2ti3 pod2usage
cc podchecker
cctiff podselect
ccxxmake poff
cd-create-profile pon
cd-fix-profile POST
cd-iccdump ppt2odp
cd-it8 pr
cdrdao precat
c++filt preconv
chacl preparetips5
chage presage_demo
chardet3 presage_demo_text
chardetect3 presage_simulator
chartread preunzip
chattr prezip
chcon prezip-bin
cheese print
chfn printcal
chgrp printenv
chmod printerbanner
choom printf
chown printtarg
chrome-gnome-shell prlimit
chrt profcheck
chsh protocoltojson
chvt prove
ciptool proxy
ckbcomp prtstat
cksum ps
clear psfaddtable
clear_console psfgettable
clutter-scan-immodules psfstriptable
cmp psfxtable

:
:
:
( I had to shorten the post )


Bash is present in /bin and chroot is located in /sbin.

Thank you !

 

[Stefan_R]

Bash is present in /bin and chroot is located in /sbin.

In theory this should work then. 'chroot /mnt /bin/bash' or 'chroot /mnt /bin/sh' should both do the trick... Although, keep in mind that even if 'bash' is correctly installed, broken libraries can lead to the same error message. Maybe your system is broken in general? Repeated kernel panics can damage filesystems, maybe try running 'fsck' on your PVE root disk.

As for the kernel panic, sadly your screencapture doesn't include the actual error message at the top (I assume since it scrolled off), so that doesn't really help...

 

[Al_mageddon]

Hi Stefan_R,

'chroot /mnt /bin/bash' or 'chroot /mnt /bin/sh' nothing works.
'fsck' doesn't work too. But the filesystem is undamaged. Please look at my attached pictures.

Just before my kernel panic, I tested a manipulation with the intel 9560 iwlwifi wifi driver. Maybe a wrong driver or incompatibility with pve-* paquets.
From now on, I have dismantled the wifi card integrated in my motherboard.

As the chroot command doesn't work, I would like to reinstall proxmox and keep my VM. It is possible ? If so, how ?

PS: Could you explain me why none of the block devices doesn't appear in Gparted under the line /dev/nvme0n1p3 ?

Thank you !

 

[Al_mageddon]

Hi,

I succeeded in chrooting.
Inside /mnt/pve/root there are missing files. I don't know why. By mounting the missing directories ( /lib, /lib32, /lib64 and /libx32 in /mnt/lib, /mnt/lib32, /mnt/lib64 and /mnt/libx32) I was able to chroot.
We could see the error in my last boot.

Do you know how to fix this whitout losing my settings ?

Thank you !
______________________
EDIT
______________________

I solved my problem. I used a USB key with PVE 6 iso burned. In Debug mode, I mounted my original installation and I copied /lib in /media/pve/root with 'cp -avr /lib /media/pve/root'. Then, I ran 'update-initramfs -u -k all' and I had to repair pve-kernel-5.0.1-5.

But how can I check that all my packages are properly installed and configured?

Thank you very much Stefan_R for your help !