Where to get -dbg kernel?

[esi_y]

Do I have to build it myself?

# apt-cache search proxmox-kernel
proxmox-kernel-6.5 - Latest Proxmox Kernel Image
proxmox-kernel-6.5.11-8-pve-signed - Proxmox Kernel Image (signed)
proxmox-kernel-helper - Function for various kernel maintenance tasks.

All I found was this [1]:

Debug kernel and modules
------------------------

In order to build a -dbgsym package containing an unstripped copy of the kernel
image and modules, enable the 'pkg.proxmox-kernel.debug' build profile (e.g. by
exporting DEB_BUILD_PROFILES='pkg.proxmox-kernel.debug'). The resulting package can
be used together with 'crash'/'kdump-tools' to debug kernel crashes.

Note: the -dbgsym package is only valid for the proxmox-kernel packages produced by
the same build. A kernel/module from a different build will likely not match,
even if both builds are of the same kernel and package version.

[1] https://github.com/proxmox/pve-kernel/blob/1b4116e1c8069bd04ce40c66c842f021f8d658a7/README#L178

 

[fabian]

yes.

 

[esi_y]

yes.

Couldn't this be ordinarily part of repo? (No one ever wanted to have kdump/kexec?)

 

[LnxBil]

No one ever wanted to have kdump/kexec?

I did some work with crashdump almost 8 years ago and was happy to have at least the last bits of information that helped me, yet a debug kernel would be also desired to further analyze the crash. Although, only a handful of people would be able to truly appreciate a debug kernel.

 

[esi_y]

I did some work with crashdump almost 8 years ago and was happy to have at least the last bits of information that helped me, yet a debug kernel would be also desired to further analyze the crash. Although, only a handful of people would be able to truly appreciate a debug kernel.

I know hot to set up the dump (but thanks for including in the thread!), I need the debug kernel grrr.

I am surprised the PVE team does not need it (to be in the repo) for the support ... I asked mostly expecting an answer like it's in subscriber-only repo, but not DIY.

 

[LnxBil]

I know hot to set up the dump (but thanks for including in the thread!), I need the debug kernel grrr.

I didn't say you don't. I just wanted to say that there are/were people that wanted and used kdump/kexec.

 

[fabian]

the problem is that the debug kernel image is huge, and very rarely used. we've had 30 kernel versions already in PVE 8.x release cycle, if we had shipped a debug kernel for each of them (IIRC they are about a GB each *after compression*), our (main PVE, but for PBS/PMG it's similar) repositories would be 80% debug kernels by size by now (and each kernel bump would make this worse!).

we do plan on splitting out all dbg/dbgsym packages into their own component(s) and reduce where those are mirrored, at that point we can likely enable them by default (this is also how Debian and Ubuntu solve this problem ).

on the rare occasion where a debug kernel helps, we just build it and make it available (and to allow regular users outside of support to do the same, it is documented how to build it - it's literally just a single option to add to the regular build invocation )

 

[esi_y]

I didn't say you don't. I just wanted to say that there are/were people that wanted and used kdump/kexec.

In that case thank you for your moral support!

the problem is that the debug kernel image is huge, and very rarely used. we've had 30 kernel versions already in PVE 8.x release cycle, if we had shipped a debug kernel for each of them (IIRC they are about a GB each *after compression*), our (main PVE, but for PBS/PMG it's similar) repositories would be 80% debug kernels by size by now (and each kernel bump would make this worse!).

But most would never be downloading it, it could be in a separate repo so that people do not do it by accident. I don't think 30G is a disaster to hold, when the traffic would be minimal (as suggested).

we do plan on splitting out all dbg/dbgsym packages into their own component(s) and reduce where those are mirrored, at that point we can likely enable them by default (this is also how Debian and Ubuntu solve this problem ).

Shall I create a bugzilla issue to track it?

on the rare occasion where a debug kernel helps, we just build it and make it available (and to allow regular users outside of support to do the same, it is documented how to build it - it's literally just a single option to add to the regular build invocation )

But the wait! And then if something goes wrong ... wait again ...

 

[fabian]

Shall I create a bugzilla issue to track it?

we don't track infrastructure things there, but like I said, it's on our TODO list and we are aware that the status quo is not ideal

But most would never be downloading it, it could be in a separate repo so that people do not do it by accident. I don't think 30G is a disaster to hold, when the traffic would be minimal (as suggested).

30G is just what would have accumulated in the past 3/4 year of 8.x. we have more than one release, and more than one node that holds that data atm, so it quickly adds up. once the split is there, keeping that one or two hundred extra GB of data in one location that has the spare capacity is easy enough.

 

[esi_y]

Thanks for the reply. For me, the frustration is basically a situation when ... a car breaks down ... and as said it might be completely useless (to diagnose that way) but just to diagnose ... I have to build the workshop first. Working on the tool when one wants the tool ... gladly not a situation when it matters or there's time critical factor, but I can imagine it happens to someone.

NB I was looking at regular Ubuntu kernel options difference to PVE. I think (not for production) that PVE should start with that just fine, right?

 

[fabian]

NB I was looking at regular Ubuntu kernel options difference to PVE. I think (not for production) that PVE should start with that just fine, right?

ZFS will be older (and thus potentially incompatible), and some other features (and a constantly changing set of fixes) that we patch in would be missing

 

[esi_y]

Although, only a handful of people would be able to truly appreciate a debug kernel.

Or someone might be asked for it and it should be easy to provide for that other person who may on their own not be "able to appreciate it".

I didn't say you don't. I just wanted to say that there are/were people that wanted and used kdump/kexec.

Thanks, it's not so lonely in the club.

 

[Ande]

The debug kernel is also important for non-kernel hackers.

I'm trying to narrow down the cause of an I/O performance bottleneck on a new PBS installation. The storage system could be part of the problem, it should be configured for the expected typical storage-request size. But I have no idea how big the individual write and read requests coming from the software are.
With systemtap (apt show systemtap) and a simple script it is possible to evaluate the size of the read and write requests arriving at the kernel over a period of time. Systemtap requires kernel headers, debug symbols and developer tools. The package includes a tool stap-prep which uses apt to install the required packages from the repository. But since the dbg-kernel package is not available, stap-prep fails on Proxmox servers.

To reproduce:

apt install systemtap
stap-prep

results in

root@pbseval:~# stap-prep
Configuring for kernel release 6.8.4-3-pve
Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS
Need to install linux-image-6.8.4-3-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-image-6.8.4-3-pve-amd64' for regex 'linux-image-6.8.4-3-pve'
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to install linux-headers-6.8.4-3-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-headers-6.8.4-3-pve-amd64' for regex 'linux-headers-6.8.4-3-pve'
Note, selecting 'proxmox-headers-6.8.4-3-pve' instead of 'linux-headers-6.8.4-3-pve-amd64'
The following NEW packages will be installed:
  proxmox-headers-6.8.4-3-pve
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.7 MB of archives.
After this operation, 97.0 MB of additional disk space will be used.
Get:1 http://download.proxmox.com/debian/pbs bookworm/pbs-no-subscription amd64 proxmox-headers-6.8.4-3-pve amd64 6.8.4-3 [13.7 MB]
Fetched 13.7 MB in 2s (7823 kB/s)
Selecting previously unselected package proxmox-headers-6.8.4-3-pve.
(Reading database ... 60441 files and directories currently installed.)
Preparing to unpack .../proxmox-headers-6.8.4-3-pve_6.8.4-3_amd64.deb ...
Unpacking proxmox-headers-6.8.4-3-pve (6.8.4-3) ...
Setting up proxmox-headers-6.8.4-3-pve (6.8.4-3) ...
Need to install linux-image-6.8.4-3-pve-dbg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-image-6.8.4-3-pve-dbg
E: Couldn't find any package by glob 'linux-image-6.8.4-3-pve-dbg'
E: Couldn't find any package by regex 'linux-image-6.8.4-3-pve-dbg'

TL;DR:
I share the desire for the dbg kernel. Although I have never knowingly used kdump or kexec in my life.

 

[fabian]

the same features are available using bpf without a debug kernel (systemtap was never really a thing on Debian-based distros, last time I tried it also didn't work with a debug build of our kernel, but that was quite a while ago ).

 

[esi_y]

the same features are available using bpf without a debug kernel (systemtap was never really a thing on Debian-based distros, last time I tried it also didn't work with a debug build of our kernel, but that was quite a while ago ).

I personally would wonder WHY something did not work with MY debug kernel instead of discrediting a tool that might be of anyone's choice, really.