[TUTORIAL] Proxmox VE 8.0 Mainline Kernel Builds

[1vivy]

I forked and modified the Proxmox kernel repo to build mainline releases with their modifications. The modifications I made feels like a fair amount of duct tape was used to make this work. But for now, it compiles mainline builds and I'll get around to cleaning it up + automating releases.

https://github.com/1vivy/pve-kernel is the repo.

You can grab a commit/tag for KERNEL_SHA1 and ZFS_SHA1 variables. The relevant upstream repos are in the README or .gitmodules
For KERNEL_SHA1, stick to tags with the naming scheme of cod/mainline/cod/tip/daily/2023-10-14, or cod/mainline/v6.6-rc5. Proxmox uses Ubuntu fork of the kernel (just some additional patches, configs and such; I'm sure someone can get the pure upstream to work).

Here's an overview of what you need to compile your own mainline build:

# add proxmox devel repo
## need to install proxmox base & keys plus debian bookworm container, if not on a proxmox host
echo "deb http://download.proxmox.com/debian/devel/ bookworm main" >> /etc/apt/sources.list

# some apt packages around building debian/proxmox
apt-get update
apt-get install -y build-essential git git-email debhelper pve-doc-generator devscripts python-is-python3 dh-python sphinx-common quilt libtraceevents-dev libunwind-dev libzstd-dev pkg-config equivs

# clone repo & install apt dependencies
git clone https://github.com/1vivy/pve-kernel.git
cd pve-kernel
## (ex: ZFS_SHA1=zfs-2.2.0 and KERNEL_SHA1=cod/mainline/v6.5.7)
make prep ZFS_SHA1= KERNEL_SHA1=
make build-dir-fresh ZFS_SHA1= KERNEL_SHA1=
mk-build-deps -ir proxmox-kernel-*/debian/control
mk-build-deps -ir proxmox-kernel-*/modules/pkg-zfs/debian/control

# compile kernel
make deb ZFS_SHA1= KERNEL_SHA1=

# install compiled debs
## you need this package so os-prober hook script for grub doesn't complain and
## you don't get a "can't find /scripts/zfs" on boot
apt install zfs-initramfs
apt install ./proxmox-headers-..._amd64.deb ./proxmox-kernel-..._amd64.deb

Also I don't recommend using mainline kernel builds for Proxmox; but if you're here you probably need a set of patches only upstream, new hardware support, a driver, or whatever.

Keep an eye on patches/kernel as some maybe added upstream. Also keep an eye on ZFS upstream, as kernel ABI can change and break compatibility (ex: zfs-2.2.0 only works for v6.5 pretty sure).

 

[neonmaus]

Is it normal that the kernel package is then almost 20x as big as built from the official proxmox kernel sources?

97MB vs 1,7GB

 

[AveryFreeman]

yes.

 

[silverstone]

It's a weird choice (by Ubuntu Developers and Proxmox VE Developers) to NOT use LTS Kernels (LTS defined on kernel.org Website), like 6.1 and 6.6 Instead.

Given the IOwait Issue I am currently experiencing, the ideal scenario would have been to build a custom Kernel based on 6.6 which is the latest supported LTS Version (and that 6.5 is EOL since the End of Last Year).

Or possibly use 6.1 Debian Bookworm Stock Kernel with Patches.

Any thoughts on that ?

I still see Kernel 5.15 (also LTS) having some work done on Proxmox Git Repository though. That might be too Old though.

 

[AveryFreeman]

Is it normal that the kernel package is then almost 20x as big as built from the official proxmox kernel sources?

97MB vs 1,7GB

View attachment 57188

Check your kernel compression type, it looks like it must be on the default GZIP, most distros are using ZSTD these days

in `make menuconfig` it's under `general setup` --> `kernel compression mode`. Make sure the compression option `<*>` under `cryptographic API` --> `zstd compression algorithm`

Before you do that, though, did you use a pre-existing `.config` file, or `make defconfig` without copying anything? The best way to start with a working `.config` file is to copy `/boot/config-$(uname -r)` to the root folder where you're building the kernel and run `make menuconfig`.

If you're new at this, it's good to add `-j$(nproc)` to your `make` command or you'll be waiting for ages for it to build.

If you're concerned about size, in addition to the GZIP vs. ZSTD kernel compression setting, you can also create a `.config` file with `make localmodconfig` or `localyesconfig`, and it'll only enable modules for the devices you're currently using on your machine.

In addition to that, `dracut` can be set to compile the initramfs with `hostonly` modules (I think it's `dracut-core` package in Debian-alikes?) - the product is similar to `localmodconfig`, but on the initrd side. It can be done using any kernel, not just the ones you build yourself, so you could try it with a stock kernel without having to build your own. Plug in any devices you want to use with the computer before running it so they get pulled into the initrd.

 

[AveryFreeman]

It's a weird choice (by Ubuntu Developers and Proxmox VE Developers) to NOT use LTS Kernels (LTS defined on kernel.org Website), like 6.1 and 6.6 Instead.

Given the IOwait Issue I am currently experiencing, the ideal scenario would have been to build a custom Kernel based on 6.6 which is the latest supported LTS Version (and that 6.5 is EOL since the End of Last Year).

Or possibly use 6.1 Debian Bookworm Stock Kernel with Patches.

Any thoughts on that ?

I still see Kernel 5.15 (also LTS) having some work done on Proxmox Git Repository though. That might be too Old though.

Using a mainline kernel might mean having to backport tons of drivers for newer systems, does Debian offer one? I am on Ubuntu and I don't see an LTS kernel option. They could try and use another vendor like RedHat's kernel patches, but it might be more trouble than it's worth.

I built an Oracle LTS UEK kernel (currently 5.15) on Archlinux for a Sandy Bridge Thinkpad I'm using as a router, but it obviously doesn't need (want) a lot of driver support, and it was kind of a slog.

 

[silverstone]

Using a mainline kernel might mean having to backport tons of drivers for newer systems, does Debian offer one? I am on Ubuntu and I don't see an LTS kernel option. They could try and use another vendor like RedHat's kernel patches, but it might be more trouble than it's worth.

I built an Oracle LTS UEK kernel (currently 5.15) on Archlinux for a Sandy Bridge Thinkpad I'm using as a router, but it obviously doesn't need (want) a lot of driver support, and it was kind of a slog.

Well Kernel 6.1.x is the LTS one on Kernel.org and the default one provided by Debian.

If you go the Debian Backports Route, you will get Kernel 6.9.x at the Moment. I also have linux-image-6.6.13+bpo-amd64 but that's because I installed it, it's not in the Repos anymore (and quite old ... latest is 6.6.43 for the 6.6 Branch).

IMHO, the best would have been to stick on the 6.6.x LTS Branch (from Ubuntu perspective).

 

[AveryFreeman]

Well Kernel 6.1.x is the LTS one on Kernel.org and the default one provided by Debian.

If you go the Debian Backports Route, you will get Kernel 6.9.x at the Moment. I also have linux-image-6.6.13+bpo-amd64 but that's because I installed it, it's not in the Repos anymore (and quite old ... latest is 6.6.43 for the 6.6 Branch).

IMHO, the best would have been to stick on the 6.6.x LTS Branch (from Ubuntu perspective).

Thanks for pointing that out, I didn't see 6.6 when I did the `apt search` so it's a good thing to know