Ugh. PVE update failing, initramfs-tools bug

[MrPete]

I just attempted an update, which failed pretty miserably. I note that a LOT of old versions are still hanging around... is that an issue?

End of errors:

Errors were encountered while processing:
pve-kernel-5.13.19-4-pve
pve-kernel-5.13
pve-kernel-5.15.19-2-pve
pve-kernel-5.15
pve-kernel-5.15.17-1-pve

Below:
- pveversion -v

Attached:
- captured live errors
- mount
- cat /proc/mounts

Anything else?

pveversion -v

proxmox-ve: 7.1-1 (running kernel: 5.15.17-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.7-1-pve: 5.15.7-1
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

[See attached for error output, mount and cat /proc/mounts]

 

[Tmanok]

Yes, if your EFI space is full then you should clear old Kernels and startup components.

apt autoclean; apt autoremove

Be sure that you do not need the old versions before hitting "Y" on your keyboard. This is not a proxmox "bug" from what I can tell, simply how Linux operates and your instance was configured. Can you output the log from your update command?

To manually upgrade, only ever run:

apt update ; apt dist-upgrade -y

Cheers,


Tmanok

 

[MrPete]

I provided the update log above.
From that log, it looks like the autoupdate (initiated in the GUI) *tries* to remove old versions... but it didn't do anything?
ALSO: I find no documented way to examine nor discover the status of the /EFI partition?

Here's my proxmox-boot-tool info:

root@pve2:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.13.19-4-pve
5.15.17-1-pve
5.15.19-2-pve

 

[MrPete]

Tried apt autoclean - ok
apt autoremove - BAD

A guess: updates don't support kdump-tools? I'll try removing that next...

Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/E93D-18A0
Copying kernel and creating boot-entry for 5.13.19-4-pve
Copying kernel and creating boot-entry for 5.15.17-1-pve
Copying kernel and creating boot-entry for 5.15.19-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/E93D-A5F4
Copying kernel and creating boot-entry for 5.13.19-4-pve
Copying kernel and creating boot-entry for 5.15.17-1-pve
Copying kernel and creating boot-entry for 5.15.19-2-pve
run-parts: executing /etc/kernel/postinst.d/kdump-tools 5.15.19-2-pve /boot/vmlinuz-5.15.19-2-pve
kdump-tools: Generating /var/lib/kdump/initrd.img-5.15.19-2-pve
mkinitramfs: failed to determine device for /
mkinitramfs: workaround is MODULES=most, check:
grep -r MODULES /var/lib/kdump/initramfs-tools

Error please report bug on initramfs-tools
Include the output of 'mount' and 'cat /proc/mounts'
update-initramfs: failed for /var/lib/kdump/initrd.img-5.15.19-2-pve with 1.
run-parts: /etc/kernel/postinst.d/kdump-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.19-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.19-2-pve (--configure):
installed pve-kernel-5.15.19-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
pve-kernel-5.15 depends on pve-kernel-5.15.19-2-pve; however:
Package pve-kernel-5.15.19-2-pve is not configured yet.

 

[fabian]

no, kdump doesn't support / on ZFS you can find workarounds here on the forum though..

 

[Tmanok]

Specifically:

I think simply commenting out the sed .. line that overrides MODULES in the kdump initramfs config is enough (line 52 in /etc/kernel/postinst.d/kdump-tools) - the kdump initramfs will then be bigger and use the same modules as the regular one, with no chance of breaking the normal initrd generation.. or removing/puring kdump-tools if you don't use it

-Fabian

If I had to guess. Comes from this forum: https://forum.proxmox.com/threads/problem-after-updating-kernel.103293/ but I'm not sure why they didn't suggest uninstalling kdump-tools instead? In production I wouldn't (because it creates logs when the kernel crashes), but if you system is stable generally it is less of an issue.

Tmanok

 

[MrPete]

It's worse than that.
I documented getting it working here https://forum.proxmox.com/threads/issue-proxmox-crashing-randomly.101660/post-448261

Turns out even with the workaround, apt update fails with kdump-tools installed (and ZFS on /).

So I tried to uninstall kdump-tools. I've now documented a partial-yet-functional uninstall at the link above.

I've still got a couple hundred MB of kdump images at /var/lib/kdump, and a dozen kdump-tools scripts (some still active!) under /etc and /var.

It appears people simply assume apt remove/purge 'works'
[Updated with typos fixed]

 

[Tmanok]

It appears people simply assume apt remove/purge 'works'

As a junior Debian maintainer and senior Systems Engineer in Canadia-Land, I can confidently tell you that apt remove does indeed work.

Some things to consider:

1. Is kdmup referenced by other packages?
2. Are some packages from the greater kdump pre-installed with Debian before kdump-tools is added as a package?
3. Are the apt remove instructions explicitly told to not remove certain portions of kdump? <-- Maintainer's choice.
4. Does kdump also need the system to reboot to remove interactions with the kernel? If so, it likely also needs an autoremove after the reboot.

You can also identify what packages are responsible for what files on Debian based systems... Leave that for another day.

 

[MrPete]

Thanks, @Tmanok ... Identifying what packages are responsible for various files/folders sounds like a very helpful process. I see there's apt-file -- is that what you're thinking of? I'll try it...

kdump is not a separate package. Sorry to mis-state. kdump-tools is "everything"
The primary elements of concern are kdump-tools scripts that remain in place, and active, after removing the package. That's wierd at the least.