Kernel crash on Proxmox hypervisor

[Ahmet Bas]

Hello,

Last week we had a kernel crash during a kernel update. We are using UEFI with Proxmox VE 6.4-13 with zfs root partition, but we are unable to capture the kernel crash. The steps we followed can be found below.

1. Install kdump-tools:

echo "kexec-tools kexec-tools/load_kexec boolean false" | debconf-set-selections
echo "kdump-tools kdump-tools/use_kdump boolean true" | debconf-set-selections
DEBIAN_FRONTEND=noninteractive apt -y install kdump-tools

2. Since we are using ZFS we need to fix those issues. Fix issue with ZFS on / partition and initramfs generation (source):

sed -i "s|^\(sed -e 's/MODULES=\.\*/MODULES=dep/'\)|#\1|g" /etc/kernel/postinst.d/kdump-tools

3. Finish installation:

apt -f install

4. Configure bootloader: First you will need to find out which bootloader you're using:

proxmox-boot-tool status

Output
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
CEBB-ACE6 is configured with: uefi (versions: 5.4.106-1-pve, 5.4.174-2-pve)
CEBC-0075 is configured with: uefi (versions: 5.4.106-1-pve, 5.4.174-2-pve)

systemd-boot
We appended the following values "crashkernel=384M-:256M nmi_watchdog=1"

cat /etc/kernel/cmdline

Output

root=ZFS=rpool/ROOT/pve-1 boot=zfs crashkernel=384M-:768M nmi_watchdog=1

5. Apply bootloader changes:

proxmox-boot-tool refresh

6. Reboot
7. Show kernel commandline:

cat /proc/cmdline

Output

initrd=\EFI\proxmox\5.4.174-2-pve\initrd.img-5.4.174-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs crashkernel=384M-:768M nmi_watchdog=1

8.Show kdump status:

kdump-config status

Output

current state   : ready to kdump

9. When we simulate a crash with the following commands:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

A crash happens after this but we are unable to capture this crash. We see this on our console