Custom Linux Kernel

[rtilson]

I know this question gets asked alot but all the previous found answers are either outdated or just are incorrect. I have a need to customize my kernel and its not clear on what is needed. What is the official way to build the kernel what system dependencies need to be met? My original method was to build the kernel in a ProxMox Guest VM but usually always fail during a debian script execution. Is there a simple way to do the build? Officially, how long should it take, I found it takes about 3.5hrs to build which seems a bit long.

Thanks

 

[t.lamprecht]

That thread is new enough, there were no big changes in the build-system the last few years: https://forum.proxmox.com/threads/building-the-pve-kernel-on-proxmox-ve-6-x.76137/

What is the official way to build the kernel what system dependencies need to be met?

See above link, mk-build-deps can help a lot here, if you can spare installing the devscripts package.

My original method was to build the kernel in a ProxMox Guest VM but usually always fail during a debian script execution. Is there a simple way to do the build?

PVE/Debian VM or Debian Container can work both OK.

Officially, how long should it take, I found it takes about 3.5hrs to build which seems a bit long.

There are no official numbers as it is very much dependent on the HW, can range from a few minutes to multiple hours. The more cores the faster, what can also help quite a bit is faster memory (basically, DDR3 vs. DDR4) and the underlying storage (can help to build in a tmpfs if you have slower spinning disks).

 

[rtilson]

Sorry for the late reply. That was one of the threads I did find, however, a vanilla build is still extremely difficult. Trying to customize the kernel after that I usually have to modify the make files not to do a 'rm -rf ${BUILD_DIR}/${KERNEL_SRC} $@' to the build kernel directory. Unfortunately, I don't remember how I successfully did this process about 8 months ago. I know it works but I just don't remember the voodoo involved.

I executed command:

sudo mk-build-deps --install debian/control.in

It ran to completion. I don't mind installing all those scripts on this VM since it will be dedicated to do PVE related builds. The far and few of them I actually do.

Here is the output of 'make' into freshly checked out and unmodified branch of 'remotes/origin/pve-kernel-5.4'
I did manually run:

git submodule foreach git fetch --tags
git submodule update --init

These commands ran to completion.

builder@pvebuild:~/pve-kernel$ make
test -f "submodules/ubuntu-focal/README" || git submodule update --init submodules/ubuntu-focal
test -f "submodules/zfsonlinux/Makefile" || git submodule update --init --recursive submodules/zfsonlinux
rm -rf build/modules/pkg-zfs build/modules/tmp pkg-zfs.prepared
mkdir -p build/modules/tmp
cp -a submodules/zfsonlinux/* build/modules/tmp
cd build/modules/tmp; make kernel
make[1]: Entering directory '/home/builder/pve-kernel/build/modules/tmp'
test -f "upstream/README.md" || git submodule update --init
rm -rf zfs-linux_2.0.6 zfs-linux_2.0.6.tmp
cp -a upstream zfs-linux_2.0.6.tmp
cp -a debian zfs-linux_2.0.6.tmp/debian
mv zfs-linux_2.0.6.tmp zfs-linux_2.0.6
tar czf zfs-linux_2.0.6.orig.tar.gz zfs-linux_2.0.6
cd zfs-linux_2.0.6; dpkg-buildpackage -S -uc -us -d
dpkg-buildpackage: info: source package zfs-linux
dpkg-buildpackage: info: source version 2.0.6-pve1~bpo10+1
dpkg-buildpackage: info: source distribution buster
dpkg-buildpackage: info: source changed by Proxmox Support Team <support@proxmox.com>
 dpkg-source --before-build .
 fakeroot debian/rules clean
make[2]: Entering directory '/home/builder/pve-kernel/build/modules/tmp/zfs-linux_2.0.6'
dh clean --with autoreconf,python3,sphinxdoc --parallel
   debian/rules override_dh_auto_clean
make[3]: Entering directory '/home/builder/pve-kernel/build/modules/tmp/zfs-linux_2.0.6'
find . -name .gitignore -delete
rm -rf zfs-2.0.6
dh_auto_clean
make[3]: Leaving directory '/home/builder/pve-kernel/build/modules/tmp/zfs-linux_2.0.6'
   dh_clean
make[2]: Leaving directory '/home/builder/pve-kernel/build/modules/tmp/zfs-linux_2.0.6'
 dpkg-source -b .
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building zfs-linux using existing ./zfs-linux_2.0.6.orig.tar.gz
dpkg-source: info: using patch list from debian/patches/series
can't find file to patch at input line 16
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
|From: Debian ZFS on Linux maintainers
| <pkg-zfsonlinux-devel@alioth-lists.debian.net>
|Date: Wed, 30 Jan 2019 15:12:04 +0100
|Subject: [PATCH] Check-for-META-and-DCH-consistency-in-autoconf
|
|Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
|---
| config/zfs-meta.m4 | 34 +++++++++++++++++++++++++++++-----
| 1 file changed, 29 insertions(+), 5 deletions(-)
|
|diff --git a/config/zfs-meta.m4 b/config/zfs-meta.m4
|index b3c1befaa..660d8ccb9 100644
|--- a/config/zfs-meta.m4
|+++ b/config/zfs-meta.m4
--------------------------
No file to patch.  Skipping patch.
3 out of 3 hunks ignored
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch '0001-Check-for-META-and-DCH-consistency-in-autoconf.patch' is correctly applied by quilt, use 'quilt refresh' to update it
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -E -b -B .pc/0001-Check-for-META-and-DCH-consistency-in-autoconf.patch/ --reject-file=- < zfs-linux_2.0.6.orig.KpPEBe/debian/patches/0001-Check-for-META-and-DCH-consistency-in-autoconf.patch subprocess returned exit status 1
dpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 2
make[1]: *** [Makefile:63: zfs-linux_2.0.6-pve1~bpo10+1.dsc] Error 2
make[1]: Leaving directory '/home/builder/pve-kernel/build/modules/tmp'
make: *** [Makefile:101: pkg-zfs.prepared] Error 2
builder@pvebuild:~/pve-kernel$

builder@pvebuild:~/pve-kernel$ git branch
  master
* pve54 --> remotes/origin/pve-kernel-5.4

builder@pvebuild:~/pve-kernel$ uname -ar
Linux pvebuild 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux

What am I missing here? I know the kernel build process itself is straight forward but when adding debian scripting it gets complicated.

Thank you.

 

[rtilson]

Ok. I did a new clone and a new checkout of the same branch and now it builds..... I'm confused.

Edit:
I'm going to chalk it up to the repo being in an unclean slate and when I recloned it, the vanilla build went through.

 

[rtilson]

Alrighty. When I am doing a custom build, I am greatly pairing down what is getting built, what is a module and what is built into the kernel. For instance I want vfio to take control of a usb controller before the ehci driver does. But if vfio is a module and ehci is built-in that doesn't happen. By building in vfio and making ehci a module I can ensure that it does. Also, I don't want the framebuffer to take any of the video cards I have installed. This system is completely headless and i want to be able to run two VMs with their own videocards. If I need console access to the server, I have serial setup for that purpose or SSH or the webGUI.

As a side note, the kernel doesn't take as long to build when unnecessary drivers are not being built since I'm only creating for 1 machine.

Assuming the kernel source builds correctly. I'm running into a problem with the fwlist-previous check. Right now I have worked around by commenting out the debian/rules 'diff' line and touching an empty fwlist.diff file to fake the check. (Line 259) How do I update the fwlist?

My next obsticle is the abi-generate on line 265 of the debian/rules. For some reason 'Module.symvers' is not in the expected location. When does his file get copied?

 

[rtilson]

I think I figured out how to build with minimal modification of the build environment. Assuming all goes well when doing the kernel build. You can comment out the fwcheck and the abicheck in debian/rules (line 290) just after 'binary: install'. For me the fwcheck failes because I don't know how to update the fwlist.previous file. It has something to do with pve-firmware repo but I don't know.

Next is the abicheck which fails for me because the Module.symver does not get copied to a predetermined location. I'm not sure what is happening here. After this point the debian packages get created.

To extract the deb files command

 ar x {file.deb}

can be run which will then extract some tarball files. Extract the tarballs to a different location to see verify that everything that is expected is pressent.

 

[rtilson]

The steps I took the build a custom kernel:

1. Get build OS up and running. I happened to use Debian 4.19.208-1
   1. configure the repositories with the correct repos ... I am using the pve-no-subscription repo
   2. setup "sudo" so that you can do admin actions without having to "su" to root
2. Install the devscripts package
3. Install the dwarves (version > 1.17) package from debian, though for pve-5.4 I'm not sure its needed.
   1. wget http://ftp.us.debian.org/debian/pool/main/d/dwarves-dfsg/dwarves_1.20-1_amd64.deb
   2. wget http://ftp.us.debian.org/debian/pool/main/libb/libbpf/libbpf0_0.5.0-1_amd64.deb
4. Clone pve-kernel repository
5. sudo mk-build-deps --install debian/control.in
6. make ubuntu-focal.prepared (pve-5.4 specific) - Have to look at the makefile to determine what this target actually is
7. make
   1. I beak the execution after it reaches building of the kernel. From there you can go to build/ubuntu-focal and do a "make menuconfig"
   2. After making any changes, make sure to save and make a copy of the ".config" file to somewhere outside the repository.
8. edit the Makefile under target: "${KERNEL_SRC}.prepared: ${KERNEL_SRC_SUBMODULE} | submodule"
   1. comment out ie: place a '#' at the beginning of the line where you see a "cat ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/config.common.ubuntu <....> "
   2. place another "#" on the next line below the line you just commented out.
9. Edit the "debian/rules" to not do the fwcheck or the abicheck
   1. search for "binary: install" - This should be a target
   2. comment out the line "debian/rules fwcheck abicheck"
10. make distclean
11. Repeat steps 6, 8 and 9
12. copy your saved kernel configuration file back to 'pve-kernel/build/ubuntu-focal' as .config
13. Execute make at ./pve-kernel

Its vital that the repository build directory is absolutely clean. Any failure beyond the kernel build will halt the deb pkg creation process. After you have your configuration file in place and have modified the makefile and rules file. The build should complete successfully.

edit:
update