building pve kernel fails with make[2]: *** [debian/rules:281: abicheck] Fehler 1

[RolandK]

i'm reproducably getting this error when trying to build pve kernel from git. i'd like to try to test a btrfs patch.

does somebody know why this happens and how to resolve?

i spent quite a wile to figure it out, but i get no clue what's the problem.

<snip>
    NEW : _RNvXs8D_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_17key__bindgen_ty_3NtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXs4M_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_10saved_msrsNtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXsD_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB5_15ratelimit_stateNtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXsw_NtNtCs8sY0e6sj61O_6kernel6device8propertyaNtB5_8Property25read_from_fwnode_property
    NEW : _RNvXsqy_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_14irq_alloc_infoNtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXsm1_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_12fprop_globalNtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXsmY_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_51bpf_jit_poke_descriptor__bindgen_ty_1__bindgen_ty_1NtNtCseXpNQPFWrFr_4core7default7Default7default
    NEW : _RNvXs4P_NtCs96IA0BLzxNi_8bindings12bindings_rawNtB6_19system_counterval_tNtNtCseXpNQPFWrFr_4core7default7Default7default
    found 2394 new symbols
WW: Found new symbols. Not recommended unless ABI was bumped
II: Checking for changes to ABI...
II: Done
make[2]: *** [debian/rules:281: abicheck] Fehler 1
make[2]: *** Es wird auf noch nicht beendete Prozesse gewartet …
mv fwlist.tmp fwlist-7.0.2-6-pve-amd64
make[2]: Verzeichnis „/build/pve-kernel/proxmox-kernel-7.0.2“ wird verlassen
make[1]: *** [debian/rules:71: binary] Fehler 2
make[1]: Verzeichnis „/build/pve-kernel/proxmox-kernel-7.0.2“ wird verlassen
dpkg-buildpackage: Fehler: Unterprozess debian/rules binary lieferte Exitstatus 2
make: *** [Makefile:75: proxmox-kernel-7.0.2-6-pve_7.0.2-6_amd64.deb] Fehler 2
roland@debian13:/build/pve-kernel$ pwd
/build/pve-kernel

i did proceed according to README:

1. installed all build dependencies

2. cd /build ; git clone git://git.proxmox.com/git/pve-kernel.git

3. cd /build/pve-kernel

4. make build-dir-fresh

5. mk-build-deps -ir proxmox-kernel-7.0.2/debian/control

6. make deb

that runs for quite a while and then finally bails out reproducably with this error.

 

[RolandK]

while running make build-dir-fresh, i see this weird error

cd proxmox-kernel-7.0.2; debian/rules debian/control
make[2]: Verzeichnis „/build/pve-kernel/proxmox-kernel-7.0.2“ wird betreten
dpkg-buildapi: Fehler: debian/control kann nicht gelesen werden: No such file or directory
/bin/sh: 1: test: Illegal number:
dpkg-buildapi: Fehler: debian/control kann nicht gelesen werden: No such file or directory
/bin/sh: 1: test: Illegal number:

dpkg-buildapi looks related

 

[daanw]

Does the folder /build/pve-kernel/proxmox-kernel-7.0.2/debian/control/ actually exist?

Looking at 3 and 5 you might now have /build/pve-kernel/pve-kernel/proxmox-kernel-7.0.2/debian/control/

 

[RolandK]

>Does the folder /build/pve-kernel/proxmox-kernel-7.0.2/debian/control/ actually exist?

yes

>Looking at 3 and 5 you might now have /build/pve-kernel/pve-kernel/proxmox-kernel-7.0.2/debian/control/

sorry, posted the wrong path. i meant mk-build-deps -ir proxmox-kernel-7.0.2/debian/control

 

[Impact]

It's extremely jarring to read different languages within the same line. Please change your system language. At least temporarily.

 

[fabian]

if you change the kernel sources in a way that changes the ABI, you need to bump the ABI ("KREL" in Makefile and changelog bump), or disable those checks (SKIPABI env variable). the other warnings are benign - they are caused by d/control being a generated file, but d/rules sourcing some things that expect it to exist already.

 

[RolandK]

yes sure, but i did not yet change the kernel sources at all.

i'm getting the same error on debian12

now with language set to en_US

    HASH : bus_register                             : 0x822ffa38 => 0x08b3d7ec
    HASH : _dev_alert                               : 0x347a8561 => 0xf799f1e1
    HASH : __devm_add_action                        : 0xcae75bdd => 0xd0fb53b6
    HASH : pm_runtime_autosuspend_expiration        : 0x23996393 => 0x720d9fea
EE: 245 symbols changed hash and weren't ignored
II: Module hash change summary...
    vmlinux                                 : 245
II: Done
make[2]: *** [debian/rules:281: abicheck] Error 1
make[2]: Leaving directory '/build/pve-kernel/proxmox-kernel-7.0.2'
make[1]: *** [debian/rules:71: binary] Error 2
make[1]: Leaving directory '/build/pve-kernel/proxmox-kernel-7.0.2'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:75: proxmox-kernel-7.0.2-6-pve_7.0.2-6_amd64.deb] Error 2

i excluded the "debian/rules fwcheck abicheck" command in rules file and now the build succeeded.

 

[fabian]

are you sure you are starting from a clean slate? what does "git status" say?

 

[fabian]

a clean checkout of current master (87f22e55de30d73b83722b86790394564036b33c) followed by `make clean deb` works fine here.

 

[RolandK]

yes, i started from clean state and git status showed nothing special .

also with exact these command, i get the same error/problem.

could you (@fabian or someone else who is able to compile successfulle) perhaps post his package list, so i can sure it's not being cause by missing package? what's your build host distro/version?

 

[fabian]

an uptodate PVE system, but it also works in a clean environment with just the build dependencies installed.. could you post the output of "git status"? do you have the "devel" repository configured (it has a newer rustc which is nowadays used for kernel builds and might cause a diff)?

 

[fabian]

here's a buildinfo for a clean build I just did, in case you want to compare.

 

[RolandK]

thank you very much! will retry and report.

 

[RolandK]

@fabian your list of packages and your pointer was very helpful.

turns out that the problem is being cause by different versions of rust or llvm toolchain.

i see that you have

libllvm21 (= 1:21.1.8-3~bpo13+2),
 libpopt0 (= 1.19+dfsg-2),
 libstd-rust-1.94 (= 1.94.1+dfsg1-1~bpo13+1),
 libstd-rust-dev (= 1.94.1+dfsg1-1~bpo13+1),
 rust-clippy (= 1.94.1+dfsg1-1~bpo13+1),
 rust-src (= 1.94.1+dfsg1-1~bpo13+1),
 rustc (= 1.94.1+dfsg1-1~bpo13+1),
 rustfmt (= 1.94.1+dfsg1-1~bpo13+1),

on your system , i.e. these are newer versions from debian trixie backports repo - and using older/default versions apparently makes generating /data/pve-kernel/proxmox-kernel-7.0.2/abi-7.0.2-6-pve-amd64 for creating the diff with abi-prev-7.0.2-6-pve-amd64 go wrong, i.e. there is massive abi change detected where in real it doesn't exist at all. (diff now is zero, whereas before it was different hunddred of lines)

i have updated these packages to the bpo versions and rebuild and the problem is resolved.

so, apparently we have subtle/sensible build tools/chain dependencies , which don't seem to be well documented.

>do you have the "devel" repository configured (it has a newer rustc which is nowadays used for kernel builds and might cause a diff)?

no , i don't but as YOU seem to have backports repo enabled and as updating the tools to backports version fixed it , it lets me assume, that the sources in git with the generated abi-prev-7.0.2-6-pve-amd64 must be from such an updated system with backport tools installed.

maybe this dependency change has been introduced recently , so nobody yet noticed - or only few people do pve kernel builds at all. or both.

i would suggest updating the developer docs/readme, as this was a real bummer. i spent several hours on digging into this issue.

fixing this helped me finish my testing with btrfs patch, reported at https://bugzilla.proxmox.com/show_bug.cgi?id=5320#c15

thank you for your help

 

[RolandK]

now i understand why you are using newer rust - at kernel.org, they even seem to use/recommend rust 1.95 for 6.12+ kernels

https://www.kernel.org/pub/tools/llvm/rust

 

[fabian]

those packages are actually from our devel repository (though they are also backports from Debian unstable/testing ).

 

[RolandK]

ok.

but how should anybody know , that we need to have specially crafted/tuned pve system for building pve kernel package?

ok, there is some mention of devel repo at https://pve.proxmox.com/wiki/Developer_Documentation

but at

https://git.proxmox.com/?p=pve-common.git;a=blob_plain;f=README.dev;hb=HEAD

and

https://git.proxmox.com/?p=pve-kernel.git;a=blob;f=README;hb=HEAD

i there is no hint that we need to have these packages as hard requirement for building pve kernel package.

is it ok to open RFE ticket for developer documentation improvement ?

 

[fabian]

e the developer documentation applies to all packages, as does the requirement for the devel repository. some packages might build fine without it by chance. it probably doesn't hurt to reference it in the kernel readme as well (since it is likely one of the packages that users have reasons to rebuild).

the abi check kind of worked as it is supposed to for what it's worth - the symbols you pasted above that changed are exported from rust code, and changing the rust toolchain version will definitely change those. for your build on Debian 12 it was probably the different gcc version, though I am not 100% sure.