Problem after updating kernel

[tlex]

So I have to admit up front.. I'm a newbie with linux...
I tried to update my kernel with apt update && apt install pve-kernel-5.15

Everything went fine and I rebooted but now when I try to update I always get that issue :

apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pve-kernel-5.13.19-3-pve (5.13.19-6) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
update-initramfs: Generating /boot/initrd.img-5.13.19-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-A6CE
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-D464
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.13.19-3-pve /boot/vmlinuz-5.13.19-3-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.13.19-3-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-5.13.19-3-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.13.19-3-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.13.19-3-pve (--configure):
 installed pve-kernel-5.13.19-3-pve package post-installation script subprocess returned error exit status 2
Setting up pve-kernel-5.15.12-1-pve (5.15.12-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.12-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-A6CE
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/C6D5-D464
    Copying kernel and creating boot-entry for 5.13.19-3-pve
    Copying kernel and creating boot-entry for 5.15.12-1-pve
    Copying kernel and creating boot-entry for 5.15.7-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.15.12-1-pve /boot/vmlinuz-5.15.12-1-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-5.15.12-1-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.12-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.12-1-pve (--configure):
 installed pve-kernel-5.15.12-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.13:
 pve-kernel-5.13 depends on pve-kernel-5.13.19-3-pve; however:
  Package pve-kernel-5.13.19-3-pve is not configured yet.

dpkg: error processing package pve-kernel-5.13 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.12-1-pve; however:
  Package pve-kernel-5.15.12-1-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.13.19-3-pve
 pve-kernel-5.15.12-1-pve
 pve-kernel-5.13
 pve-kernel-5.15
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried the following commands with the same results :

dpkg --configure -a

apt-get autoclean

apt --fix-broken install

Can anyone help me with that ?

 

[tlex]

These 2 packages seam to be linked to my problem but I don't know what to do

dpkg -C
The following packages have been unpacked but not yet configured.
They must be configured using dpkg --configure or the configure
menu option in dselect for them to work:
 pve-kernel-5.13      Latest Proxmox VE Kernel Image
 pve-kernel-5.15      Latest Proxmox VE Kernel Image

The following packages are only half configured, probably due to problems
configuring them the first time.  The configuration should be retried using
dpkg --configure <package> or the configure menu option in dselect:
 pve-kernel-5.13.19-3-pve The Proxmox PVE Kernel Image
 pve-kernel-5.15.12-1-pve The Proxmox PVE Kernel Image

 

[t.lamprecht]

Can you please attach /var/log/apt/history.log here /var/log/apt/term.log? So that we have a better picture of what happened on that host package-management wise.

The following command output would be good to have too:

pveversion -v
lsblk
cat /proc/cmdline

 

[fabian]

did you try doing what the error message tells you? you have kdump-tools installed, that wants to generate the kdump initrd, but can't determine the FS or / (is it on ZFS?). it suggest configuring the initrd generation to just copy most kernel modules instead of attempting auto-generation:

kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

 

[tlex]

did you try doing what the error message tells you? you have kdump-tools installed, that wants to generate the kdump initrd, but can't determine the FS or / (is it on ZFS?). it suggest configuring the initrd generation to just copy most kernel modules instead of attempting auto-generation:

kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.12-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

I wish I knew how to do that..
grep - r MODULES returns :

grep -r MODULES /var/lib/kdump/initramfs-tools
/var/lib/kdump/initramfs-tools/initramfs.conf:# MODULES: [ most | netboot | dep | list ]
/var/lib/kdump/initramfs-tools/initramfs.conf:MODULES=dep

 

[tlex]

Can you please attach /var/log/apt/history.log here /var/log/apt/term.log? So that we have a better picture of what happened on that host package-management wise.

The following command output would be good to have too:

pveversion -v
lsblk
cat /proc/cmdline

Thanks, I really appreciate your help.
As you will see in the logs I tried multiple things / times to fix this since upgrading the kernel but since I'm far from being an expert at this I hope I didn't break too many things...
Initially I tried to upgrade the kernel because I had random server crash (ie 1 every 2-3 days) and that is also why I've installed kdump.
So far, except for that error that I now have with the package manager I have to say that the server didn't crash yet.

pveversion -v:

proxmox-ve: 7.1-1 (running kernel: 5.15.12-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.15.7-1-pve: 5.15.7-1
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.6T  0 disk
├─sda1        8:1    0   1.6T  0 part
└─sda9        8:9    0     8M  0 part
sdb           8:16   0   1.6T  0 disk
├─sdb1        8:17   0   1.6T  0 part
└─sdb9        8:25   0     8M  0 part
zd0         230:0    0    32G  0 disk
├─zd0p1     230:1    0    31G  0 part
├─zd0p2     230:2    0     1K  0 part
└─zd0p5     230:5    0   975M  0 part
zd16        230:16   0    32G  0 disk
├─zd16p1    230:17   0   512M  0 part
├─zd16p2    230:18   0  30.5G  0 part
└─zd16p3    230:19   0   976M  0 part
zd32        230:32   0    60G  0 disk
├─zd32p1    230:33   0   100M  0 part
├─zd32p2    230:34   0    16M  0 part
└─zd32p3    230:35   0  59.9G  0 part
zd48        230:48   0    32G  0 disk
├─zd48p1    230:49   0   260M  0 part
└─zd48p2    230:50   0  31.7G  0 part
zd64        230:64   0    40G  0 disk
├─zd64p1    230:65   0   100M  0 part
├─zd64p2    230:66   0    16M  0 part
├─zd64p3    230:67   0  39.4G  0 part
└─zd64p4    230:68   0   507M  0 part
zd80        230:80   0     1M  0 disk
zd96        230:96   0    40G  0 disk
├─zd96p1    230:97   0   100M  0 part
├─zd96p2    230:98   0    16M  0 part
├─zd96p3    230:99   0  39.4G  0 part
└─zd96p4    230:100  0   507M  0 part
zd112       230:112  0    32G  0 disk
├─zd112p1   230:113  0    32G  0 part
├─zd112p5   230:117  0    30G  0 part
└─zd112p6   230:118  0   1.6G  0 part
zd128       230:128  0    10G  0 disk
├─zd128p1   230:129  0  1007K  0 part
├─zd128p2   230:130  0   512M  0 part
└─zd128p3   230:131  0   9.5G  0 part
nvme1n1     259:0    0 931.5G  0 disk
├─nvme1n1p1 259:1    0  1007K  0 part
├─nvme1n1p2 259:2    0   512M  0 part
└─nvme1n1p3 259:3    0 899.5G  0 part
nvme0n1     259:4    0 931.5G  0 disk
├─nvme0n1p1 259:5    0  1007K  0 part
├─nvme0n1p2 259:6    0   512M  0 part
└─nvme0n1p3 259:7    0 899.5G  0 part

cat /proc/cmdline

initrd=\EFI\proxmox\5.15.12-1-pve\initrd.img-5.15.12-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs nmi_watchdog=1 crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M

 

[tlex]

I wish I knew how to do that..
grep - r MODULES returns :

grep -r MODULES /var/lib/kdump/initramfs-tools
/var/lib/kdump/initramfs-tools/initramfs.conf:# MODULES: [ most | netboot | dep | list ]
/var/lib/kdump/initramfs-tools/initramfs.conf:MODULES=dep

ok so I tried changing MODULES to most in /var/lib/kdump/initramfs-tools/initramfs.conf
but I don't know what to do next..

 

[fiona]

IIRC, the file /var/lib/kdump/initramfs-tools/initramfs.conf is generated, so changing MODULES there does not help.

The ZFS root + kdump combination unfortunately doesn't work currently, although a patch does exists in a related bug report.

 

[fabian]

I think simply commenting out the sed .. line that overrides MODULES in the kdump initramfs config is enough (line 52 in /etc/kernel/postinst.d/kdump-tools) - the kdump initramfs will then be bigger and use the same modules as the regular one, with no chance of breaking the normal initrd generation.. or removing/puring kdump-tools if you don't use it

 

[tlex]

I think simply commenting out the sed .. line that overrides MODULES in the kdump initramfs config is enough (line 52 in /etc/kernel/postinst.d/kdump-tools) - the kdump initramfs will then be bigger and use the same modules as the regular one, with no chance of breaking the normal initrd generation.. or removing/puring kdump-tools if you don't use it

Thank you very much Fabian, all good now
Really appreciated !

 

[Okeur]

Hello,

Just to mention the solution provided by Fabian worked also in my case.
I was unable to upgrade my proxmox due to kdump, and unable to uninstall kdump either (error below).

So thanks Fabian !

root@pve:/var/lib/kdump/initramfs-tools# apt remove --purge kdump-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  kdump-tools*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up pve-kernel-5.4.174-2-pve (5.4.174-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
update-initramfs: Generating /boot/initrd.img-5.4.174-2-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.4.174-2-pve /boot/vmlinuz-5.4.174-2-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.4.174-2-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for  with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.4.174-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.4.174-2-pve (--configure):
 installed pve-kernel-5.4.174-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.4:
 pve-kernel-5.4 depends on pve-kernel-5.4.174-2-pve; however:
  Package pve-kernel-5.4.174-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.4 (--configure):
 dependency problems - leaving unconfigured
Setting up pve-kernel-5.4.128-1-pve (5.4.128-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.128-1-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.4.128-1-pve /boot/vmlinuz-5.4.128-1-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.4.128-1-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for  with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.4.128-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.4.128-1-pve (--configure):
 installed pve-kernel-5.4.128-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.4; however:
  Package pve-kernel-5.4 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.4.174-2-pve
 pve-kernel-5.4
 pve-kernel-5.4.128-1-pve
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[habitats-tech]

For anyone looking at this thread, trying to resolve the same problem, I provide a summary below:

The issue not being able to update the kernel is related to having installed kdump-tools while using ZFS as the filesystem.

Check this link describing the bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934736

The solution is to
either
remove kdump-tools with the command: apt remove kdump-tools
or
comment line 52 using command: nano /etc/kernel/postinst.d/kdump-tools
# sed -e 's/MODULES=.*/MODULES=dep/' /etc/initramfs-tools/initramfs.conf > "$initramfsdir/initramfs.conf" || true