Where to get -dbg kernel?

E

esi_y

Renowned Member
Nov 29, 2023
2,221
389
68
github.com
Do I have to build it myself?

Code: 
# apt-cache search proxmox-kernel
proxmox-kernel-6.5 - Latest Proxmox Kernel Image
proxmox-kernel-6.5.11-8-pve-signed - Proxmox Kernel Image (signed)
proxmox-kernel-helper - Function for various kernel maintenance tasks.

All I found was this [1]:
Code: 
Debug kernel and modules
------------------------

In order to build a -dbgsym package containing an unstripped copy of the kernel
image and modules, enable the 'pkg.proxmox-kernel.debug' build profile (e.g. by
exporting DEB_BUILD_PROFILES='pkg.proxmox-kernel.debug'). The resulting package can
be used together with 'crash'/'kdump-tools' to debug kernel crashes.

Note: the -dbgsym package is only valid for the proxmox-kernel packages produced by
the same build. A kernel/module from a different build will likely not match,
even if both builds are of the same kernel and package version.
[1] https://github.com/proxmox/pve-kernel/blob/1b4116e1c8069bd04ce40c66c842f021f8d658a7/README#L178
 
LnxBil said:
I did some work with crashdump almost 8 years ago and was happy to have at least the last bits of information that helped me, yet a debug kernel would be also desired to further analyze the crash. Although, only a handful of people would be able to truly appreciate a debug kernel.
Click to expand...

I know hot to set up the dump (but thanks for including in the thread!), I need the debug kernel grrr. :)

I am surprised the PVE team does not need it (to be in the repo) for the support ... I asked mostly expecting an answer like it's in subscriber-only repo, but not DIY.
 
the problem is that the debug kernel image is huge, and very rarely used. we've had 30 kernel versions already in PVE 8.x release cycle, if we had shipped a debug kernel for each of them (IIRC they are about a GB each *after compression*), our (main PVE, but for PBS/PMG it's similar) repositories would be 80% debug kernels by size by now (and each kernel bump would make this worse!).

we do plan on splitting out all dbg/dbgsym packages into their own component(s) and reduce where those are mirrored, at that point we can likely enable them by default (this is also how Debian and Ubuntu solve this problem ;)).

on the rare occasion where a debug kernel helps, we just build it and make it available (and to allow regular users outside of support to do the same, it is documented how to build it - it's literally just a single option to add to the regular build invocation :))
 
LnxBil said:
I didn't say you don't. I just wanted to say that there are/were people that wanted and used kdump/kexec.
Click to expand...

In that case thank you for your moral support! :D

fabian said:
the problem is that the debug kernel image is huge, and very rarely used. we've had 30 kernel versions already in PVE 8.x release cycle, if we had shipped a debug kernel for each of them (IIRC they are about a GB each *after compression*), our (main PVE, but for PBS/PMG it's similar) repositories would be 80% debug kernels by size by now (and each kernel bump would make this worse!).
Click to expand...

But most would never be downloading it, it could be in a separate repo so that people do not do it by accident. I don't think 30G is a disaster to hold, when the traffic would be minimal (as suggested).

fabian said:
we do plan on splitting out all dbg/dbgsym packages into their own component(s) and reduce where those are mirrored, at that point we can likely enable them by default (this is also how Debian and Ubuntu solve this problem ;)).
Click to expand...

Shall I create a bugzilla issue to track it? :)

fabian said:
on the rare occasion where a debug kernel helps, we just build it and make it available (and to allow regular users outside of support to do the same, it is documented how to build it - it's literally just a single option to add to the regular build invocation :))
Click to expand...

But the wait! And then if something goes wrong ... wait again ...
 
tempacc346235 said:
Shall I create a bugzilla issue to track it? :)
Click to expand...
we don't track infrastructure things there, but like I said, it's on our TODO list and we are aware that the status quo is not ideal :)
tempacc346235 said:
But most would never be downloading it, it could be in a separate repo so that people do not do it by accident. I don't think 30G is a disaster to hold, when the traffic would be minimal (as suggested).
Click to expand...
30G is just what would have accumulated in the past 3/4 year of 8.x. we have more than one release, and more than one node that holds that data atm, so it quickly adds up. once the split is there, keeping that one or two hundred extra GB of data in one location that has the spare capacity is easy enough.
 
  • Like
Reactions: esi_y
Thanks for the reply. For me, the frustration is basically a situation when ... a car breaks down ... and as said it might be completely useless (to diagnose that way) but just to diagnose ... I have to build the workshop first. Working on the tool when one wants the tool ... gladly not a situation when it matters or there's time critical factor, but I can imagine it happens to someone.

NB I was looking at regular Ubuntu kernel options difference to PVE. I think (not for production) that PVE should start with that just fine, right?
 
tempacc346235 said:
NB I was looking at regular Ubuntu kernel options difference to PVE. I think (not for production) that PVE should start with that just fine, right?
Click to expand...
ZFS will be older (and thus potentially incompatible), and some other features (and a constantly changing set of fixes) that we patch in would be missing
 
  • Like
Reactions: esi_y
LnxBil said:
Although, only a handful of people would be able to truly appreciate a debug kernel.
Click to expand...

Or someone might be asked for it and it should be easy to provide for that other person who may on their own not be "able to appreciate it".

LnxBil said:
I didn't say you don't. I just wanted to say that there are/were people that wanted and used kdump/kexec.
Click to expand...

Thanks, it's not so lonely in the club. :)
 
The debug kernel is also important for non-kernel hackers.

I'm trying to narrow down the cause of an I/O performance bottleneck on a new PBS installation. The storage system could be part of the problem, it should be configured for the expected typical storage-request size. But I have no idea how big the individual write and read requests coming from the software are.
With systemtap (apt show systemtap) and a simple script it is possible to evaluate the size of the read and write requests arriving at the kernel over a period of time. Systemtap requires kernel headers, debug symbols and developer tools. The package includes a tool stap-prep which uses apt to install the required packages from the repository. But since the dbg-kernel package is not available, stap-prep fails on Proxmox servers.

To reproduce:

apt install systemtap
stap-prep

results in
Code: 
root@pbseval:~# stap-prep
Configuring for kernel release 6.8.4-3-pve
Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS
Need to install linux-image-6.8.4-3-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-image-6.8.4-3-pve-amd64' for regex 'linux-image-6.8.4-3-pve'
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to install linux-headers-6.8.4-3-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-headers-6.8.4-3-pve-amd64' for regex 'linux-headers-6.8.4-3-pve'
Note, selecting 'proxmox-headers-6.8.4-3-pve' instead of 'linux-headers-6.8.4-3-pve-amd64'
The following NEW packages will be installed:
  proxmox-headers-6.8.4-3-pve
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.7 MB of archives.
After this operation, 97.0 MB of additional disk space will be used.
Get:1 http://download.proxmox.com/debian/pbs bookworm/pbs-no-subscription amd64 proxmox-headers-6.8.4-3-pve amd64 6.8.4-3 [13.7 MB]
Fetched 13.7 MB in 2s (7823 kB/s)
Selecting previously unselected package proxmox-headers-6.8.4-3-pve.
(Reading database ... 60441 files and directories currently installed.)
Preparing to unpack .../proxmox-headers-6.8.4-3-pve_6.8.4-3_amd64.deb ...
Unpacking proxmox-headers-6.8.4-3-pve (6.8.4-3) ...
Setting up proxmox-headers-6.8.4-3-pve (6.8.4-3) ...
Need to install linux-image-6.8.4-3-pve-dbg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-image-6.8.4-3-pve-dbg
E: Couldn't find any package by glob 'linux-image-6.8.4-3-pve-dbg'
E: Couldn't find any package by regex 'linux-image-6.8.4-3-pve-dbg'

TL;DR:
I share the desire for the dbg kernel. Although I have never knowingly used kdump or kexec in my life. :)
 
Last edited:
  • Like
Reactions: esi_y
the same features are available using bpf without a debug kernel (systemtap was never really a thing on Debian-based distros, last time I tried it also didn't work with a debug build of our kernel, but that was quite a while ago :)).
 
fabian said:
the same features are available using bpf without a debug kernel (systemtap was never really a thing on Debian-based distros, last time I tried it also didn't work with a debug build of our kernel, but that was quite a while ago :)).
Click to expand...

I personally would wonder WHY something did not work with MY debug kernel instead of discrediting a tool that might be of anyone's choice, really.
 
Hi ! Thanks for your response. Could you also provide some link to a description how to best build the debug kernel ?
The reason I ask is because if I execute make build-dir-fresh I get the following error:


make[2]: Entering directory '/root/pve-kernel/proxmox-kernel-6.14.11'
dpkg-buildapi: error: cannot read debian/control: No such file or directory
/bin/sh: 1: test: Illegal number:
dpkg-buildapi: error: cannot read debian/control: No such file or directory

Thanks in advance
 
Last edited:
1. apply the referenced patch
2. ensure git repo is clean: git status
3. make clean build-dir-fresh
4. sudo apt build-dep ./proxmox-kernel-6.14.11 (this is the only step that needs root)
5. either run DEB_BUILD_PROFILES=pkg.proxmox-kernel.debug make deb to build directly in the host environment, or run make dsc; sbuild -d trixie --profiles pkg.proxmox-kernel.debug *.dsc to build with sbuild in a clean environment
 
Hi !

I applied the patch. my git diff shows following:

Code: 
root@snuc1:~/pve-kernel# git diff
diff --git a/debian/rules b/debian/rules
index 7f6cadf..628480e 100755
--- a/debian/rules
+++ b/debian/rules
@@ -144,7 +144,7 @@ install: .install_mark .tools_install_mark .headers_install_mark .usr_headers_in
 
 binary: install
        debian/rules fwcheck abicheck
-       dh_strip -N$(PMX_HEADER_PKG) -N$(PMX_USR_HEADER_PKG)
+       dh_strip -N$(PMX_HEADER_PKG) -N$(PMX_USR_HEADER_PKG) -Xvmlinux -Xvmlinuz
        dh_makeshlibs
        dh_shlibdeps
        dh_installdeb

but when I do the make clean build-dir-fresh I get the following:

Code: 
mkdir -p proxmox-kernel-6.14.11
cp -a debian proxmox-kernel-6.14.11/debian
echo "git clone git://git.proxmox.com/git/pve-kernel.git\\ngit checkout f83854a50f89b0ca8fa8d2b8aed05e12a02fbce7" \
    >proxmox-kernel-6.14.11/debian/SOURCE
echo "KVNAME=6.14.11-1-pve" >> proxmox-kernel-6.14.11/debian/rules.d/env.mk
echo "KERNEL_MAJMIN=6.14" >> proxmox-kernel-6.14.11/debian/rules.d/env.mk
cd proxmox-kernel-6.14.11; debian/rules debian/control
make[2]: Entering directory '/root/pve-kernel/proxmox-kernel-6.14.11'
dpkg-buildapi: error: cannot read debian/control: No such file or directory
/bin/sh: 1: test: Illegal number:
dpkg-buildapi: error: cannot read debian/control: No such file or directory
/bin/sh: 1: test: Illegal number:

I execute all the commands directly on my proxmox server as root.
any idea what could be the cause for the "No such file or directory" as I can see them:
Code: 
root@snuc1:~/pve-kernel# ls -la proxmox-kernel-6.14.11/debian/
total 196
drwxr-xr-x 7 root root  4096 Sep 11 13:28 .
drwxr-xr-x 5 root root  4096 Sep 11 13:28 ..
drwxr-xr-x 2 root root  4096 Sep 11 10:57 certs
-rw-r--r-- 1 root root 81751 Sep 11 10:57 changelog
-rw-r--r-- 1 root root  4323 Sep 11 13:28 control
-rw-r--r-- 1 root root  4319 Sep 11 11:09 control.in
-rw-r--r-- 1 root root  1061 Sep 11 10:57 copyright
-rwxr-xr-x 1 root root  1661 Sep 11 13:28 proxmox-headers-6.14.11-1-pve.postinst
-rw-r--r-- 1 root root  1661 Sep 11 10:57 proxmox-headers.postinst.in
-rwxr-xr-x 1 root root   591 Sep 11 13:28 proxmox-kernel-6.14.11-1-pve.postinst
-rwxr-xr-x 1 root root  1218 Sep 11 13:28 proxmox-kernel-6.14.11-1-pve.postrm
-rwxr-xr-x 1 root root   559 Sep 11 13:28 proxmox-kernel-6.14.11-1-pve.prerm
-rwxr-xr-x 1 root root   338 Sep 11 13:28 proxmox-kernel-6.14.postinst
-rwxr-xr-x 1 root root   402 Sep 11 13:28 proxmox-kernel-6.14.postrm
-rwxr-xr-x 1 root root   348 Sep 11 10:57 proxmox-kernel-meta.postinst.in
-rwxr-xr-x 1 root root   396 Sep 11 10:57 proxmox-kernel-meta.postrm.in
-rwxr-xr-x 1 root root   588 Sep 11 10:57 proxmox-kernel.postinst.in
-rw-r--r-- 1 root root  1215 Sep 11 10:57 proxmox-kernel.postrm.in
-rwxr-xr-x 1 root root   556 Sep 11 10:57 proxmox-kernel.prerm.in
-rwxr-xr-x 1 root root 15860 Sep 11 12:05 rules
drwxr-xr-x 2 root root  4096 Sep 11 13:28 rules.d
drwxr-xr-x 2 root root  4096 Sep 11 10:57 scripts
drwxr-xr-x 3 root root  4096 Sep 11 13:28 signing-template
drwxr-xr-x 2 root root  4096 Sep 11 10:57 source
-rw-r--r-- 1 root root   105 Sep 11 13:28 SOURCE
 
You must log in or register to reply here.
Top Bottom