Error found when starting Update from 7 to 8

[james9911]

No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.83-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.83-1-pve (--configure):
 installed pve-kernel-5.15.83-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.143-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 pve-kernel-5.15.131-1-pve
 pve-kernel-5.15.116-1-pve
 pve-kernel-5.15.143-1-pve
 pve-kernel-5.15
 pve-kernel-5.15.131-2-pve
 pve-kernel-5.15.83-1-pve
 proxmox-ve
 initramfs-tools

havnt noticed this on normal updates but now i cant autoremove old packages.
From what i can see the message is saying out of disk space but

root@pve-hosted:~# df -h
Filesystem        Size  Used Avail Use% Mounted on
udev               14G     0   14G   0% /dev
tmpfs             2.8G  1.1M  2.8G   1% /run
rpool/ROOT/pve-1  129G   12G  118G   9% /
tmpfs              14G   49M   14G   1% /dev/shm
tmpfs             5.0M     0  5.0M   0% /run/lock
zfs-raid1         126G  128K  126G   1% /zfs-raid1
rpool             118G  128K  118G   1% /rpool
rpool/ROOT        118G  128K  118G   1% /rpool/ROOT
rpool/data        118G  128K  118G   1% /rpool/data
/dev/fuse         128M   16K  128M   1% /etc/pve
tmpfs             2.8G  4.0K  2.8G   1% /run/user/0

makes me think its something else.
I dont want to attempt the upgrade as the machine isnt easily accessable and is running currently on

Kernel Version
 
Linux 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200)
PVE Manager Version
 
pve-manager/7.4-17/513c62be

 

[sherminator]

What exactly did you do that lead to this output?
Did you follow these instructions?
what PVE repo do you use?

please post the output of
ls -al /etc/apt/sources.list.d/
cat /etc/apt/sources.list.d/*

 

[james9911]

What exactly did you do that lead to this output?

apt autoremove to start with as the apt dist-upgrade had this error

Did you follow these instructions?

didnt get as far as that. I dont want to attempt the upgrade from 7 to 8 until this is resolved

what PVE repo do you use?

repo is no-subscription

ls -al /etc/apt/sources.list.d/

root@pve-hosted:~# ls -al /etc/apt/sources.list.d/
total 18
drwxr-xr-x 2 root root  4 Dec  7 18:36 .
drwxr-xr-x 8 root root 10 Aug  4  2022 ..
-rw-r--r-- 1 root root 72 Aug  4  2022 pve-enterprise.list
-rw-r--r-- 1 root root 70 Jul  5  2022 pve-enterprise.list.dpkg-dist
root@pve-hosted:~#

cat /etc/apt/sources.list.d/*

root@pve-hosted:~# cat /etc/apt/sources.list.d/*
# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

However those commands dont show what is listed on the web gui

 

[james9911]

Thankyou for your reply i forgot to add!

 

[james9911]

root@pve-hosted:~# apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  pve-kernel-5.15.108-1-pve pve-kernel-5.15.74-1-pve
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
10 not fully installed or removed.
After this operation, 800 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 123602 files and directories currently installed.)
Removing pve-kernel-5.15.108-1-pve (5.15.108-2) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.15.108-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.15.108-1-pve (--remove):
 installed pve-kernel-5.15.108-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 pve-kernel-5.15.108-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@pve-hosted:~#

whole shell copy with command entered

 

[sherminator]

didnt get as far as that.

ok, so the errors occur while you tried to update to the current PVE 7. That's fine, but then your topic is a little bit misleading.

repo is no-subscription

so far I can't see any no-subscription repo. To be sure, please also post the output of cat /etc/apt/sources.list.

However those commands dont show what is listed on the web gui

can you also post a screenshot of the webGUI "Updates" -> "Repositories"?

 

[james9911]

ok, so the errors occur while you tried to update to the current PVE 7. That's fine, but then your topic is a little bit misleading.

Sorry Yes your quite right but i only noticed while beginning to attepmt the update.

cat /etc/apt/sources.list

root@pve-hosted:~# cat /etc/apt/sources.list
deb http://ftp.uk.debian.org/debian bullseye main contrib


# security updates
deb http://security.debian.org bullseye-security main contrib

deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

root@pve-hosted:~#

can you also post a screenshot of the webGUI "Updates" -> "Repositories"?

Thanks again for all your help!

James

 

[sherminator]

Ok, here it is, the no-subscription repo. I think you can remove /etc/apt/sources.list.d/*, because it's not needed. Does not solve your problem, but make things clearer. Does apt update show something unusual?

 

[james9911]

root@pve-hosted:~# apt update
Hit:1 http://security.debian.org bullseye-security InRelease
Hit:2 http://ftp.uk.debian.org/debian bullseye InRelease
Hit:3 http://download.proxmox.com/debian/pve bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
root@pve-hosted:~#

then upgrade

root@pve-hosted:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
10 not fully installed or removed.
Need to get 154 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 pve-kernel-5.15.108-1-pve amd64 5.15.108-2 [76.8 MB]
Get:2 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 pve-kernel-5.15.74-1-pve amd64 5.15.74-1 [76.7 MB]
Fetched 154 MB in 2s (81.7 MB/s)                   
Setting up initramfs-tools (0.140) ...
update-initramfs: deferring update (trigger activated)
Setting up pve-kernel-5.15.131-1-pve (5.15.131-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.131-1-pve /boot/vmlinuz-5.15.131-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.131-1-pve /boot/vmlinuz-5.15.131-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.131-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.131-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.131-1-pve (--configure):
 installed pve-kernel-5.15.131-1-pve package post-installation script subprocess returned error exit status 2
Setting up pve-kernel-5.15.116-1-pve (5.15.116-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.116-1-pve /boot/vmlinuz-5.15.116-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.116-1-pve /boot/vmlinuz-5.15.116-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.116-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.116-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.116-1-pve (--configure):
 installed pve-kernel-5.15.116-1-pve package post-installation script subprocess returned error exit status 2
Setting up pve-kernel-5.15.143-1-pve (5.15.143-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.143-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.143-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.143-1-pve (--configure):
 installed pve-kernel-5.15.143-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.143-1-pve; however:
  Package pve-kernel-5.15.143-1-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
Setting up pve-kernel-5.15.131-2-pve (5.15.131-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
update-initramfs: Generating /boot/initrd.img-5.15.131-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.131-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.131-2-pve (--configure):
 installed pve-kernel-5.15.131-2-pve package post-installation script subprocess returned error exit status 2
Setting up pve-kernel-5.15.83-1-pve (5.15.83-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.83-1-pve /boot/vmlinuz-5.15.83-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.83-1-pve /boot/vmlinuz-5.15.83-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.83-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.83-1-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.83-1-pve (--configure):
 installed pve-kernel-5.15.83-1-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.143-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/3A1B-D616
        Copying kernel 5.15.131-2-pve
cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 pve-kernel-5.15.131-1-pve
 pve-kernel-5.15.116-1-pve
 pve-kernel-5.15.143-1-pve
 pve-kernel-5.15
 pve-kernel-5.15.131-2-pve
 pve-kernel-5.15.83-1-pve
 proxmox-ve
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@pve-hosted:~#

 

[leesteken]

cp: error writing '/var/tmp/espmounts/3A1B-D616/vmlinuz-5.15.131-2-pve': No space left on device

Have you tried removing some unused old kernels to make some space on the ESP partitions?

 

[sherminator]

Have you tried removing some unused old kernels to make some space on the ESP partitions?

maybe this can help:
-> Proxmox VE Tools -> Proxmox VE Kernel Clean

Can somebody confirm that it is safe to run this script?

 

[tteckster]

maybe this can help:
-> Proxmox VE Tools -> Proxmox VE Kernel Clean

Can somebody confirm that it is safe to run this script?

A straightforward and simple script. However, it's always advisable for individuals to verify it themselves.

https://raw.githubusercontent.com/tteck/Proxmox/main/misc/kernel-clean.sh

 

[james9911]

there is very little in the var tmp directory and the drive has loads of space. I dotn want to just delete the stuff thats in there but its only directorys no files

 

[leesteken]

there is very little in the var tmp directory and the drive has loads of space. I dotn want to just delete the stuff thats in there but its only directorys no files

It's the 512MB ESP partitions (that are usually not mounted) that are full. Can you uninstall (apt remove pve-kernel-... or apt autoremove) some older kernels to make room?

 

[gfngfn256]

Can somebody confirm that it is safe to run this script?

Used often. Never ran into problems. Gives me the opportunity to thank the author I believe, of many useful scripts. Thanks @tteckster , its you right?

 

[james9911]

how do i know which kernels i can remove?

 

[fabian]

@james9911 you can use "proxmox-boot-tool kernel list" to list those that PVE would like to put onto the ESP - everything else can normally be removed.

 

[gfngfn256]

how do i know which kernels i can remove?

The above mentioned script by tteck makes life really easy, it will make available to you, in a selectable-list format, only those kernels that are currently unused. Just choose some of the oldest ones, and you should be able to clean up a lot of ESP real estate.