/boot full after upgrade, system not booting

W

woka

Member
May 23, 2021
8
1
8
Hello guys, today I apt upgraded a server I was not touching since quite a while without realizing that during the process /boot got full and the kernel update could not finish properly.

I don't have physical access to the server, it's rented from a provider which doesn't provide serial console to it.
The only thing I can do is to boot it in rescue mode and the chroot on the proxmox installation.
Apparently I'm not doing it properly, because I am not able to free up the space in /boot and to switch to one of the many old kernels in it.
Until now I was just able to mount the proxmox root partition and the boot partition before chrooting, but, other than accessing the files, I am not able to update the system properly.
Do you guys have any hint on how to restore an older kernel?
Thank you!

I attach the /var/log/apt/term.log that shows what happened during the update:
Code: 
Backing up lvm.conf before setting pve-manager specific settings..
'/etc/lvm/lvm.conf' -> '/etc/lvm/lvm.conf.bak' (backup: '/etc/lvm/lvm.conf.bak~')
Setting 'global_filter' in /etc/lvm/lvm.conf to prevent zvols and rbds from being scanned:
global_filter="r|/dev/zd.*|" => global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|"]
  LVM configuration valid.
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for ntpsec (1.2.2+dfsg1-1+deb12u1) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.2.16-12-pve
zstd: error 70 : Write error : cannot write block : No space left on device
E: mkinitramfs failure zstd -q -9 -T0 70
update-initramfs: failed for /boot/initrd.img-6.2.16-12-pve with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.36-9+deb12u4) ...
Processing triggers for rsyslog (8.2302.0-1) ...
Processing triggers for pve-ha-manager (4.0.3) ...
Errors were encountered while processing:
proxmox-kernel-6.2.16-20-pve
proxmox-kernel-6.2
proxmox-kernel-6.5.13-1-pve-signed
proxmox-kernel-6.5
proxmox-default-kernel
proxmox-ve
initramfs-tools
Log ended: 2024-03-11  12:19:19
 
the list of the installed kernels:

Code: 
# apt list --installed | grep kernel


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.


proxmox-default-kernel/stable,stable,now 1.0.1 all [installed,automatic]
proxmox-kernel-6.2.16-12-pve/stable,stable,now 6.2.16-12 amd64 [installed,auto-removable]
proxmox-kernel-6.2.16-20-pve/stable,stable,now 6.2.16-20 amd64 [installed,automatic]
proxmox-kernel-6.2/stable,stable,now 6.2.16-20 all [installed,automatic]
proxmox-kernel-6.5.13-1-pve-signed/stable,now 6.5.13-1 amd64 [installed,automatic]
proxmox-kernel-6.5/stable,now 6.5.13-1 all [installed,automatic]
proxmox-kernel-helper/stable,stable,now 8.1.0 all [installed,automatic]
pve-kernel-5.15.107-1-pve/now 5.15.107-1 amd64 [installed,local]
pve-kernel-5.15.107-2-pve/now 5.15.107-2 amd64 [installed,local]
pve-kernel-5.15.108-1-pve/now 5.15.108-1 amd64 [installed,local]
pve-kernel-5.4.189-2-pve/now 5.4.189-2 amd64 [installed,local]
pve-kernel-5.4/now 6.4-18 all [installed,local]
pve-kernel-6.2.16-3-pve/stable,stable,now 6.2.16-3 amd64 [installed,automatic]
pve-kernel-6.2/stable,stable,now 8.0.5 all [installed,automatic]
 
Kernel version 6.2 is not maintained for some time; maybe just uninstall those? apt remove pve-kernel-6.2 followed by apt autoremove? Kernel verson 5.4 is also quite old.
 
leesteken said:
Kernel version 6.2 is not maintained for some time; maybe just uninstall those? apt remove pve-kernel-6.2 followed by apt autoremove? Kernel verson 5.4 is also quite old.
Click to expand...
He wants to know how to correctly make his system bootable (maybe on an older kernel) - See OP.
 
leesteken said:
The ESP is out of space, so somebody will have to make room before the boot problem can be fixed. Kernel version 6.2 should no longer be used (for any reason).
Click to expand...
Sure I agree. But maybe give some guidance with the rest of the problem.
 
leesteken said:
Kernel version 6.2 is not maintained for some time; maybe just uninstall those? apt remove pve-kernel-6.2 followed by apt autoremove? Kernel verson 5.4 is also quite old.
Click to expand...

Thank you for your reply.
My issue is that probably I'm not chrooting properly and I can't install or remove any kernel.
Any suggestion on how to chroot on the proxmox system from the provider's rescue system so that I can fix the system?

Would freeing up some space in /boot by deleting an older kernel let me correctly install correctly the new kernel from my chrooted console?

This is what I'm doing:

Code: 
# cat /proc/partitions
major minor  #blocks  name

   7        0     603720 loop0
   8        0  976762584 sda
   8        1     523264 sda1
   8        2     524288 sda2
   8        3          1 sda3
   8        5  975712256 sda5
 253        0  975708160 dm-0



# mkdir /media/RESCUE
# mkdir /media/RESCUE/boot
# mount /dev/dm-0 /media/RESCUE
# mount /dev/sda1 /media/RESCUE/boot
# chroot /media/RESCUE/



# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Get:3 http://security.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Hit:4 http://download.proxmox.com/debian/pbs bookworm InRelease
Hit:5 http://download.proxmox.com/debian/pve bookworm InRelease
Fetched 48.0 kB in 13s (3665 B/s)                                                                   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  g++-10 libaom0 libavcodec58 libavfilter7 libavformat58 libavutil56 libboost-coroutine1.74.0
  libbpf0 libcbor0 libcodec2-0.9 libdav1d4 libdns-export1110 libevent-pthreads-2.1-7
  libfftw3-double3 libflac8 libicu67 libigdgmm11 libisc-export1105 libldap-2.4-2 libleveldb1d
  libllvm11 libmpdec3 libopts25 libperl5.32 libpostproc55 libprocps8 libprotobuf23 libpython3.9
  libpython3.9-minimal libpython3.9-stdlib libsgutils2-2 libsrt1.4-gnutls libstdc++-10-dev
  libswresample3 libswscale5 libtiff5 liburcu6 liburing1 libusb-0.1-4 libvpx6 libwebp6 libx264-160
  libx265-192 ntpsec-ntpdig perl-modules-5.32 proxmox-kernel-6.2.16-12-pve
  pve-kernel-5.15.107-1-pve pve-kernel-5.15.107-2-pve python3-ply python3.9 python3.9-minimal sntp
  telnet
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
6 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_TIME = "C.UTF-8",
    LC_MONETARY = "C.UTF-8",
    LC_ADDRESS = "C.UTF-8",
    LC_TELEPHONE = "C.UTF-8",
    LC_NAME = "C.UTF-8",
    LC_MEASUREMENT = "C.UTF-8",
    LC_IDENTIFICATION = "C.UTF-8",
    LC_NUMERIC = "C.UTF-8",
    LC_PAPER = "C.UTF-8",
    LANG = "fr_FR.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_TIME = "C.UTF-8",
    LC_MONETARY = "C.UTF-8",
    LC_ADDRESS = "C.UTF-8",
    LC_TELEPHONE = "C.UTF-8",
    LC_NAME = "C.UTF-8",
    LC_MEASUREMENT = "C.UTF-8",
    LC_IDENTIFICATION = "C.UTF-8",
    LC_NUMERIC = "C.UTF-8",
    LC_PAPER = "C.UTF-8",
    LANG = "fr_FR.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
Setting up proxmox-kernel-6.2.16-20-pve (6.2.16-20) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.2.16-20-pve /boot/vmlinuz-6.2.16-20-pve
update-initramfs: Generating /boot/initrd.img-6.2.16-20-pve
W: Couldn't identify type of root file system for fsck hook
W: initramfs-tools configuration sets RESUME=UUID=b91e0b95-7a1b-4ffc-b2d8-fc0748a451bc
W: but no matching swap device is available.
zstd: error 70 : Write error : cannot write block : No space left on device
E: mkinitramfs failure zstd -q -9 -T0 70
update-initramfs: failed for /boot/initrd.img-6.2.16-20-pve with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.2.16-20-pve.postinst
line 20.
dpkg: error processing package proxmox-kernel-6.2.16-20-pve (--configure):
 installed proxmox-kernel-6.2.16-20-pve package post-installation script subprocess returned error ex
it status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.2:
 proxmox-kernel-6.2 depends on proxmox-kernel-6.2.16-20-pve; however:
  Package proxmox-kernel-6.2.16-20-pve is not configured yet.

dpkg: error processing package proxmox-kernel-6.2 (--configure):
 dependency problems - leaving unconfigured
Setting up proxmox-kernel-6.5.13-1-pve-signed (6.5.13-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.5.13-1-pve /boot/vmlinuz-6.5.13-1-pve
update-initramfs: Generating /boot/initrd.img-6.5.13-1-pve
W: Couldn't identify type of root file system for fsck hook
W: initramfs-tools configuration sets RESUME=UUID=b91e0b95-7a1b-4ffc-b2d8-fc0748a451bc
W: but no matching swap device is available.
zstd: error 70 : Write error : cannot write block : No space left on device
E: mkinitramfs failure zstd -q -9 -T0 70
update-initramfs: failed for /boot/initrd.img-6.5.13-1-pve with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.5.13-1-pve-signed.pos
tinst line 20.
dpkg: error processing package proxmox-kernel-6.5.13-1-pve-signed (--configure):
 installed proxmox-kernel-6.5.13-1-pve-signed package post-installation script subprocess returned er
ror exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.5:
 proxmox-kernel-6.5 depends on proxmox-kernel-6.5.13-1-pve-signed | proxmox-kernel-6.5.13-1-pve; howe
ver:
  Package proxmox-kernel-6.5.13-1-pve-signed is not configured yet.
  Package proxmox-kernel-6.5.13-1-pve is not installed.
  Package proxmox-kernel-6.5.13-1-pve-signed which provides proxmox-kernel-6.5.13-1-pve is not config
ured yet.

dpkg: error processing package proxmox-kernel-6.5 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-default-kernel:
 proxmox-default-kernel depends on proxmox-kernel-6.5; however:
  Package proxmox-kernel-6.5 is not configured yet.

dpkg: error processing package proxmox-default-kernel (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on proxmox-default-kernel; however:
  Package proxmox-default-kernel is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 proxmox-kernel-6.2.16-20-pve
 proxmox-kernel-6.2
 proxmox-kernel-6.5.13-1-pve-signed
 proxmox-kernel-6.5
 proxmox-default-kernel
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
woka said:
My issue is that probably I'm not chrooting properly and I can't install or remove any kernel.
Any suggestion on how to chroot on the proxmox system from the provider's rescue system so that I can fix the system?
Click to expand...
woka said:
# mkdir /media/RESCUE
# mkdir /media/RESCUE/boot
# mount /dev/dm-0 /media/RESCUE
# mount /dev/sda1 /media/RESCUE/boot
# chroot /media/RESCUE/
Click to expand...
Looks like you are not (bind) mounting /dev, /sys and /proc, which is necessary before doing the chroot.
Here is an example of all the steps (before the chroot): https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
woka said:
Would freeing up some space in /boot by deleting an older kernel let me correctly install correctly the new kernel from my chrooted console?
Click to expand...
I expect so, but I do think it's necessary anyway. Which specific old kernel do you want to use?

PS:
gfngfn256 said:
Sure I agree. But maybe give some guidance with the rest of the problem.
Click to expand...
Feel free to chip in, instead of only complaining about me.
 
  • Like
Reactions: Kingneutron
leesteken said:
Looks like you are not (bind) mounting /dev, /sys and /proc, which is necessary before doing the chroot.
Here is an example of all the steps (before the chroot): https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
Click to expand...
Thank you so much!
This is what I was missing.

I was able to clean up some space and to boot the system up again.

Now the installed kernels are the following:

Code: 
apt list --installed | grep kernel

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

proxmox-default-kernel/stable,stable,now 1.0.1 all [installed,automatic]
proxmox-kernel-6.2.16-20-pve/stable,stable,now 6.2.16-20 amd64 [installed,automatic]
proxmox-kernel-6.2/stable,stable,now 6.2.16-20 all [installed,automatic]
proxmox-kernel-6.5.13-1-pve-signed/stable,now 6.5.13-1 amd64 [installed,automatic]
proxmox-kernel-6.5/stable,now 6.5.13-1 all [installed,automatic]
proxmox-kernel-helper/stable,stable,now 8.1.0 all [installed,automatic]
pve-kernel-5.4.189-2-pve/now 5.4.189-2 amd64 [installed,local]

Does that look alright or are there any other kernel that you suggest to remove?
Thank you very much again
 
woka said:
proxmox-default-kernel/stable,stable,now 1.0.1 all [installed,automatic]
proxmox-kernel-6.2.16-20-pve/stable,stable,now 6.2.16-20 amd64 [installed,automatic]
proxmox-kernel-6.2/stable,stable,now 6.2.16-20 all [installed,automatic]
proxmox-kernel-6.5.13-1-pve-signed/stable,now 6.5.13-1 amd64 [installed,automatic]
proxmox-kernel-6.5/stable,now 6.5.13-1 all [installed,automatic]
proxmox-kernel-helper/stable,stable,now 8.1.0 all [installed,automatic]
pve-kernel-5.4.189-2-pve/now 5.4.189-2 amd64 [installed,local][/CODE]

Does that look alright or are there any other kernel that you suggest to remove?
Click to expand...
I would remove kernel 6.2 completely because it is not maintained (no security fixes!), as I wrote before.
 
leesteken said:
Feel free to chip in, instead of only complaining about me.
Click to expand...
Sorry. I didn't mean to complain. I had to go out just then, and didn't have time to chip in. You've given him excellent advice, as you always do to so many on this forum. I've enjoyed your advice & insight many a time, and also learned a lot from you, thanks.
 
  • Like
Reactions: Kingneutron
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom