Kernel crash issue

[fiona]

Hi,

@fiona in process of troubleshooting some pve 8.2 random freezes im currently looking into enabling kernel crashdumps. I followed the steps from this thread, but I do not get a /var/crash/<DATE> file for the crash successfully triggered with sync; echo c | tee /proc/sysrq-trigger
any idea? Do I need to have the dbsym packages? and if yes, is there a easy way to do now?

root@PMX8:/var/crash# dmesg | grep crash
[    0.000000] Command line: initrd=\EFI\proxmox\6.8.4-2-pve\initrd.img-6.8.4-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt pcie_aspm.policy=performance crashkernel=384M-:512M
[    0.004996] crashkernel reserved: 0x0000000055000000 - 0x0000000075000000 (512 MB)
[    0.401744] Kernel command line: initrd=\EFI\proxmox\6.8.4-2-pve\initrd.img-6.8.4-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt pcie_aspm.policy=performance crashkernel=384M-:512M
[    1.608775] pstore: Using crash dump compression: deflate

root@PMX8:/var/crash# ls
kdump_lock  kexec_cmd

root@PMX8:/var/crash# kdump-config show
DUMP_MODE:              kdump
USE_KDUMP:              1
KDUMP_COREDIR:          /var/crash
crashkernel addr: 0x55000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.8.4-2-pve
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.8.4-2-pve
current state:    ready to kdump


kexec command:
  /sbin/kexec -p --command-line="initrd=\EFI\proxmox.8.4-2-pve\initrd.img-6.8.4-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt pcie_aspm.policy=performance reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll usbcore.nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

EDIT: I looked at the IPMI, but kvm console shutdown completly after triggering via sync; echo c | tee /proc/sysrq-trigger
asfar as I understood, ipmi should still be able to show the kernel dump etc. but yeah its just down instantly.

unfortunately, I haven't done it recently myself. Can you check the system logs/journal if there is any information? Maybe you need more memory (512 MiB worked for my test VM with 6 GiB RAM back then)?

There is a new 6.8.4-3-pve kernel on the no-subscription repository, that should address some of the issues with 6.8.

And in 6.8 the default for the intel_iommu kernel parameter changed, if it wasn't enabled manually before, adding intel_iommu=off to the kernel commandline might be worth a try: https://bugzilla.proxmox.com/show_bug.cgi?id=5423

 

[jsterr]

We are still in need of getting kernel crash dumps up & running. Seems like the hardest part ist to get a dbg kernel. Can you explain how we can get it by using the documentation you mentioned: https://git.proxmox.com/?p=pve-kern...a483836863da4d34deb;hb=refs/heads/master#l105

Can you provide the commands to do this?

108 In order to build a -dbgsym package containing an unstripped copy of the kernel
109 image and modules, enable the 'pkg.pve-kernel.debug' build profile (e.g. by
110 exporting DEB_BUILD_PROFILES='pkg.pve-kernel.debug'). The resulting package can
111 be used together with 'crash'/'kdump-tools' to debug kernel crashes.
112
113 Note: the -dbgsym package is only valid for the pve-kernel packages produced by
114 the same build. A kernel/module from a different build will likely not match,
115 even if both builds are of the same kernel and package version.

 

[fiona]

We are still in need of getting kernel crash dumps up & running. Seems like the hardest part ist to get a dbg kernel. Can you explain how we can get it by using the documentation you mentioned: https://git.proxmox.com/?p=pve-kern...a483836863da4d34deb;hb=refs/heads/master#l105

Can you provide the commands to do this?

IIUC, executing export DEB_BUILD_PROFILES='pkg.pve-kernel.debug' in your shell before running make deb should be enough.

 

[jsterr]

Thanks for answering that quick! So this cant be done on system that have already been installed? Is this possible on a already installed system with pve 8.2? If yes, are there some commands missing (like git clone etc?)

root@PMX4:~# export DEB_BUILD_PROFILES='pkg.pve-kernel.debug'
root@PMX4:~# make deb
make: *** No rule to make target 'deb'.  Stop.
root@PMX4:~#

 

[fiona]

Thanks for answering that quick! So this cant be done on system that have already been installed? Is this possible on a already installed system with pve 8.2? If yes, are there some commands missing (like git clone etc?)

root@PMX4:~# export DEB_BUILD_PROFILES='pkg.pve-kernel.debug'
root@PMX4:~# make deb
make: *** No rule to make target 'deb'.  Stop.
root@PMX4:~#

Yes, you need to run the commands in the git repository for the kernel: https://git.proxmox.com/?p=pve-kernel.git;a=summary

See also the README file in the repository for build instructions.