Problem after updating kernel

[tlex]

So I have to admit up front.. I'm a newbie with linux...
I tried to update my kernel with apt update && apt install pve-kernel-5.15

Everything went fine and I rebooted but now when I try to update I always get that issue :

apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pve-kernel-5.13.19-3-pve (5.13.19-6) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
update-initramfs: Generating /boot/initrd.img-5.13.19-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-A6CE
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-D464
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.13.19-3-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-5.13.19-3-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.13.19-3-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.13.19-3-pve (--configure):
 installed pve-kernel-5.13.19-3-pve package post-installation script subprocess returned error exit status 2
Setting up pve-kernel-5.15.12-1-pve (5.15.12-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.12-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-A6CE
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-D464
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-5.15.12-1-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.12-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.12-1-pve (--configure):
 installed pve-kernel-5.15.12-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.13:
 pve-kernel-5.13 depends on pve-kernel-5.13.19-3-pve; however:
  Package pve-kernel-5.13.19-3-pve is not configured yet.

dpkg: error processing package pve-kernel-5.13 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.12-1-pve; however:
  Package pve-kernel-5.15.12-1-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.13.19-3-pve
 pve-kernel-5.15.12-1-pve
 pve-kernel-5.13
 pve-kernel-5.15
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried the following commands with the same results :

dpkg --configure -a

apt-get autoclean

apt --fix-broken install

Can anyone help me with that ?

 

[tlex]

These 2 packages seam to be linked to my problem but I don't know what to do

dpkg -C
The following packages have been unpacked but not yet configured.
They must be configured using dpkg --configure or the configure
menu option in dselect for them to work:
 pve-kernel-5.13      Latest Proxmox VE Kernel Image
 pve-kernel-5.15      Latest Proxmox VE Kernel Image

The following packages are only half configured, probably due to problems
configuring them the first time.  The configuration should be retried using
dpkg --configure <package> or the configure menu option in dselect:
 pve-kernel-5.13.19-3-pve The Proxmox PVE Kernel Image
 pve-kernel-5.15.12-1-pve The Proxmox PVE Kernel Image

 

[t.lamprecht]

Can you please attach /var/log/apt/history.log here /var/log/apt/term.log? So that we have a better picture of what happened on that host package-management wise.

The following command output would be good to have too:

pveversion -v
lsblk
cat /proc/cmdline

 

[fabian]

did you try doing what the error message tells you? you have kdump-tools installed, that wants to generate the kdump initrd, but can't determine the FS or / (is it on ZFS?). it suggest configuring the initrd generation to just copy most kernel modules instead of attempting auto-generation:

kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

 

[tlex]

did you try doing what the error message tells you? you have kdump-tools installed, that wants to generate the kdump initrd, but can't determine the FS or / (is it on ZFS?). it suggest configuring the initrd generation to just copy most kernel modules instead of attempting auto-generation:

kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

I wish I knew how to do that..
grep - r MODULES returns :

grep -r MODULES /var/lib/kdump/initramfs-tools
/var/lib/kdump/initramfs-tools/initramfs.conf:# MODULES: [ most | netboot | dep | list ]
/var/lib/kdump/initramfs-tools/initramfs.conf:MODULES=dep

 

[tlex]

Can you please attach /var/log/apt/history.log here /var/log/apt/term.log? So that we have a better picture of what happened on that host package-management wise.

The following command output would be good to have too:

pveversion -v
lsblk
cat /proc/cmdline

Thanks, I really appreciate your help.
As you will see in the logs I tried multiple things / times to fix this since upgrading the kernel but since I'm far from being an expert at this I hope I didn't break too many things...
Initially I tried to upgrade the kernel because I had random server crash (ie 1 every 2-3 days) and that is also why I've installed kdump.
So far, except for that error that I now have with the package manager I have to say that the server didn't crash yet.

pveversion -v:

proxmox-ve: 7.1-1 (running kernel: 5.15.12-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.15.7-1-pve: 5.15.7-1
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.6T  0 disk
├─sda1        8:1    0   1.6T  0 part
└─sda9        8:9    0     8M  0 part
sdb           8:16   0   1.6T  0 disk
├─sdb1        8:17   0   1.6T  0 part
└─sdb9        8:25   0     8M  0 part
zd0         230:0    0    32G  0 disk
├─zd0p1     230:1    0    31G  0 part
├─zd0p2     230:2    0     1K  0 part
└─zd0p5     230:5    0   975M  0 part
zd16        230:16   0    32G  0 disk
├─zd16p1    230:17   0   512M  0 part
├─zd16p2    230:18   0  30.5G  0 part
└─zd16p3    230:19   0   976M  0 part
zd32        230:32   0    60G  0 disk
├─zd32p1    230:33   0   100M  0 part
├─zd32p2    230:34   0    16M  0 part
└─zd32p3    230:35   0  59.9G  0 part
zd48        230:48   0    32G  0 disk
├─zd48p1    230:49   0   260M  0 part
└─zd48p2    230:50   0  31.7G  0 part
zd64        230:64   0    40G  0 disk
├─zd64p1    230:65   0   100M  0 part
├─zd64p2    230:66   0    16M  0 part
├─zd64p3    230:67   0  39.4G  0 part
└─zd64p4    230:68   0   507M  0 part
zd80        230:80   0     1M  0 disk
zd96        230:96   0    40G  0 disk
├─zd96p1    230:97   0   100M  0 part
├─zd96p2    230:98   0    16M  0 part
├─zd96p3    230:99   0  39.4G  0 part
└─zd96p4    230:100  0   507M  0 part
zd112       230:112  0    32G  0 disk
├─zd112p1   230:113  0    32G  0 part
├─zd112p5   230:117  0    30G  0 part
└─zd112p6   230:118  0   1.6G  0 part
zd128       230:128  0    10G  0 disk
├─zd128p1   230:129  0  1007K  0 part
├─zd128p2   230:130  0   512M  0 part
└─zd128p3   230:131  0   9.5G  0 part
nvme1n1     259:0    0 931.5G  0 disk
├─nvme1n1p1 259:1    0  1007K  0 part
├─nvme1n1p2 259:2    0   512M  0 part
└─nvme1n1p3 259:3    0 899.5G  0 part
nvme0n1     259:4    0 931.5G  0 disk
├─nvme0n1p1 259:5    0  1007K  0 part
├─nvme0n1p2 259:6    0   512M  0 part
└─nvme0n1p3 259:7    0 899.5G  0 part

cat /proc/cmdline

initrd=\EFI\proxmox\5.15.12-1-pve\initrd.img-5.15.12-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs nmi_watchdog=1 crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M

 

[tlex]

I wish I knew how to do that..
grep - r MODULES returns :

grep -r MODULES /var/lib/kdump/initramfs-tools
/var/lib/kdump/initramfs-tools/initramfs.conf:# MODULES: [ most | netboot | dep | list ]
/var/lib/kdump/initramfs-tools/initramfs.conf:MODULES=dep

ok so I tried changing MODULES to most in /var/lib/kdump/initramfs-tools/initramfs.conf
but I don't know what to do next..

 

[fiona]

IIRC, the file /var/lib/kdump/initramfs-tools/initramfs.conf is generated, so changing MODULES there does not help.

The ZFS root + kdump combination unfortunately doesn't work currently, although a patch does exists in a related bug report.

 

[fabian]

I think simply commenting out the sed .. line that overrides MODULES in the kdump initramfs config is enough (line 52 in /etc/kernel/postinst.d/kdump-tools) - the kdump initramfs will then be bigger and use the same modules as the regular one, with no chance of breaking the normal initrd generation.. or removing/puring kdump-tools if you don't use it

 

[tlex]

I think simply commenting out the sed .. line that overrides MODULES in the kdump initramfs config is enough (line 52 in /etc/kernel/postinst.d/kdump-tools) - the kdump initramfs will then be bigger and use the same modules as the regular one, with no chance of breaking the normal initrd generation.. or removing/puring kdump-tools if you don't use it

Thank you very much Fabian, all good now
Really appreciated !

 

[Okeur]

Hello,

Just to mention the solution provided by Fabian worked also in my case.
I was unable to upgrade my proxmox due to kdump, and unable to uninstall kdump either (error below).

So thanks Fabian !

root@pve:/var/lib/kdump/initramfs-tools# apt remove --purge kdump-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  kdump-tools*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up pve-kernel-5.4.174-2-pve (5.4.174-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
update-initramfs: Generating /boot/initrd.img-5.4.174-2-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.4.174-2-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for  with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.4.174-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.4.174-2-pve (--configure):
 installed pve-kernel-5.4.174-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.4:
 pve-kernel-5.4 depends on pve-kernel-5.4.174-2-pve; however:
  Package pve-kernel-5.4.174-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.4 (--configure):
 dependency problems - leaving unconfigured
Setting up pve-kernel-5.4.128-1-pve (5.4.128-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.128-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.4.128-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for  with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.4.128-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.4.128-1-pve (--configure):
 installed pve-kernel-5.4.128-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.4; however:
  Package pve-kernel-5.4 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.4.174-2-pve
 pve-kernel-5.4
 pve-kernel-5.4.128-1-pve
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[habitats-tech]

For anyone looking at this thread, trying to resolve the same problem, I provide a summary below:

The issue not being able to update the kernel is related to having installed kdump-tools while using ZFS as the filesystem.

Check this link describing the bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934736

The solution is to
either
remove kdump-tools with the command: apt remove kdump-tools
or
comment line 52 using command: nano /etc/kernel/postinst.d/kdump-tools
# sed -e 's/MODULES=.*/MODULES=dep/' /etc/initramfs-tools/initramfs.conf > "$initramfsdir/initramfs.conf" || true

 

[johnwbyrd]

Ah you clever Proxmox developers, keeping the rest of us on our toes by quietly and secretly overwriting configuration files!

If your updates are breaking with something like:

run-parts: executing /etc/kernel/postinst.d/kdump-tools 6.8.12-8-pve /boot/vmlinuz-6.8.12-8-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-6.8.12-8-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-6.8.12-8-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.8.12-8-pve-signed.postinst line 20.

Then, to fix this insanity at its core, you need to change MODULES=dep to MODULES=most in /etc/kernel/postinst.d/kdump-tools . Then your updates will work as expected.

Changing /var/lib/kdump/initramfs-tools/conf.d/zzz-kdump will NOT fix the problem! Your configuration file changes will be silently overwritten at update-initramfs time!

Insert standard warning about premature optimization being the root of all evil -- this bad idea was ostensibly to save space in the boot partition, I imagine.

 

[t.lamprecht]

Ah you clever Proxmox developers, keeping the rest of us on our toes by quietly and secretly overwriting configuration files!

If your updates are breaking with something like:

run-parts: executing /etc/kernel/postinst.d/kdump-tools 6.8.12-8-pve /boot/vmlinuz-6.8.12-8-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-6.8.12-8-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-6.8.12-8-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.8.12-8-pve-signed.postinst line 20.

Then, to fix this insanity at its core, you need to change MODULES=dep to MODULES=most in /etc/kernel/postinst.d/kdump-tools . Then your updates will work as expected.

Changing /var/lib/kdump/initramfs-tools/conf.d/zzz-kdump will NOT fix the problem! Your configuration file changes will be silently overwritten at update-initramfs time!

Insert standard warning about premature optimization being the root of all evil -- this bad idea was ostensibly to save space in the boot partition, I imagine.

System-wide configuration files reside in /etc on basically any Linux distro following the Filesystem Hierarchy Standard (FHS), /var on the other hand is for variable data and state written by various programs and almost never a place for making manual changes, well at least if one wants them to persist...
/etc has special handling to track manual changes and ask the user on package updates what to do if both, the default config from the package and the local config changed. This is not new behavior, but rather exists since >30 years.

So no, neither we nor other distros are "secretly" overwriting configuration files any program just writes our variable data like caches or state to /var.

Just edit the actual configuration in /etc/initramfs-tools/initramfs.conf or create a dedicated local.conf snippet in /etc/initramfs-tools/conf.d/