[SOLVED] Proxmox VE 4.4 Kernel Update Failed

DeepCoreDev · May 20, 2017

Hello all, wondering if anyone can point me in the right direction today. I updated my Proxmox VE 4.4 server yesterday about 26 hours ago and there was a kernel update been trying to get it back since. I remember that it only failed to update the kernel with an error code of (1). Here's the console output for devices and boot files. Roughly, How would I go about updating the Proxmox kernel with SystemRescueCD? My machine will not boot, I can not see console screen. I only have SSH access through SystemRescueCD, right now. I have been having trouble finding the boot logs for Proxmox, any constructive criticism, advice or comment will be greatly appreciated.
Edit: I did the system update through the Proxmox control panel.

Code:

root@sysresccd /root % fsarchiver probe simple
[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN]
[sda             ] [TOSHIBA DT01ACA1               ] [   931.51 GB] [  8] [  0]

[=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN]
[loop0           ] [squashfs   ] [<unknown>        ] [   275.14 MB] [  7] [  0]
[sda1            ] [ext2       ] [<unknown>        ] [   243.00 MB] [  8] [  1]
[sda5            ] [LVM2_member] [<unknown>        ] [   931.27 GB] [  8] [  5]
[dm-0            ] [ext4       ] [<unknown>        ] [   899.85 GB] [253] [  0]
[dm-1            ] [swap       ] [<unknown>        ] [    31.42 GB] [253] [  1]
root@sysresccd /root % mkdir /mnt/linux
root@sysresccd /root % mount /dev/dm-0 /mnt/linux
root@sysresccd /root % mount -o bind /proc /mnt/linux/proc
root@sysresccd /root % mount -o bind /dev /mnt/linux/dev
root@sysresccd /root % mount -o bind /sys /mnt/linux/sys
root@sysresccd /root % chroot /mnt/linux /bin/bash
root@sysresccd:/# ls -l /boot/
total 0
root@sysresccd:/# mount /dev/sda1 /boot/
root@sysresccd:/# ls -l /boot/
total 214411
-rw-r--r-- 1 root root  2681172 Mar  7 18:59 System.map-3.16.0-4-amd64
-rw-r--r-- 1 root root  2117983 Oct 19  2016 System.map-3.2.0-4-amd64
-rw-r--r-- 1 root root  3906508 Dec 22 09:04 System.map-4.4.35-1-pve
-rw-r--r-- 1 root root  3906508 Jan 20 08:03 System.map-4.4.35-2-pve
-rw-r--r-- 1 root root  3906557 Feb 23 09:53 System.map-4.4.40-1-pve
-rw-r--r-- 1 root root  3914955 Mar  9 06:25 System.map-4.4.44-1-pve
-rw-r--r-- 1 root root  3886548 May 31  2016 System.map-4.4.8-1-pve
-rw-r--r-- 1 root root   157815 Mar  7 18:59 config-3.16.0-4-amd64
-rw-r--r-- 1 root root   129281 Oct 19  2016 config-3.2.0-4-amd64
-rw-r--r-- 1 root root   189740 Dec 22 09:04 config-4.4.35-1-pve
-rw-r--r-- 1 root root   189693 Jan 20 08:03 config-4.4.35-2-pve
-rw-r--r-- 1 root root   189693 Feb 23 09:53 config-4.4.40-1-pve
-rw-r--r-- 1 root root   189893 Mar  9 06:25 config-4.4.44-1-pve
-rw-r--r-- 1 root root   189190 May 31  2016 config-4.4.8-1-pve
drwxr-xr-x 5 root root     5120 May 19 10:24 grub
-rw-r--r-- 1 root root 11789040 Dec 22 03:49 initrd.img-2.6.32-20-pve
-rw-r--r-- 1 root root 15964497 Dec 22 03:55 initrd.img-2.6.32-26-pve
-rw-r--r-- 1 root root 20864183 Feb 28 20:40 initrd.img-3.16.0-4-amd64
-rw-r--r-- 1 root root 14000048 Dec 22 03:55 initrd.img-3.2.0-4-amd64
-rw-r--r-- 1 root root 23942427 Dec 30 18:53 initrd.img-4.4.35-1-pve
-rw-r--r-- 1 root root 23941346 Feb  1 22:11 initrd.img-4.4.35-2-pve
-rw-r--r-- 1 root root 23942769 Feb 28 20:41 initrd.img-4.4.40-1-pve
-rw-r--r-- 1 root root 16860673 Dec 22 05:39 initrd.img-4.4.8-1-pve
drwxr-xr-x 2 root root    12288 Dec 22 03:36 lost+found
-rw-r--r-- 1 root root  3128784 Mar  7 18:58 vmlinuz-3.16.0-4-amd64
-rw-r--r-- 1 root root  2849984 Oct 19  2016 vmlinuz-3.2.0-4-amd64
-rw-r--r-- 1 root root  7165760 Dec 22 09:04 vmlinuz-4.4.35-1-pve
-rw-r--r-- 1 root root  7165696 Jan 20 08:03 vmlinuz-4.4.35-2-pve
-rw-r--r-- 1 root root  7166224 Feb 23 09:53 vmlinuz-4.4.40-1-pve
-rw-r--r-- 1 root root  7183952 Mar  9 06:25 vmlinuz-4.4.44-1-pve
-rw-r--r-- 1 root root  7114192 May 31  2016 vmlinuz-4.4.8-1-pve

Sorry for any lack of detail as I am not familiar with things that well.

DeepCoreDev · May 20, 2017

Code:

menuentry 'Proxmox Virtual Environment GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentr$
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci$
        else
          search --no-floppy --fs-uuid --set=root d70179a9-8d5e-41e6-85dc-94e08aa5f783
        fi
        echo    'Loading Linux 4.4.44-1-pve ...'
        linux   /vmlinuz-4.4.44-1-pve root=/dev/mapper/c31018h54224-root ro  quiet
}

Here's the entry from the grub.cfg. Looks like the update was to version: vmlinuz-4.4.8-1-pve.

Code:

root@sysresccd:/boot/grub# ls
device.map  fonts  grub.cfg  grubenv  i386-pc  locale  unicode.pf2
root@sysresccd:/boot/grub# ls -l
total 2384
-rw-r--r-- 1 root root      55 Dec 22 03:47 device.map
drwxr-xr-x 2 root root    1024 Dec 22 05:36 fonts
-r--r--r-- 1 root root   16696 May 19 10:24 grub.cfg
-rw-r--r-- 1 root root    1024 Dec 22 03:47 grubenv
drwxr-xr-x 2 root root    6144 May 19 23:35 i386-pc
drwxr-xr-x 2 root root    1024 May 19 23:35 locale
-rw-r--r-- 1 root root 2400500 Dec 22 05:36 unicode.pf2

DeepCoreDev · May 20, 2017

Well in grub.cfg I changed the initial 'menuentry' to have:

Code:

echo    'Loading Linux 4.4.40-1-pve ...'
linux   /vmlinuz-4.4.40-1-pve root=/dev/mapper/c31018h54224-root ro  quiet
echo    'Loading initial ramdisk ...'
initrd  /initrd.img-4.4.40-1-pve

Instead of:

Code:

echo    'Loading Linux 4.4.44-1-pve ...'
linux   /vmlinuz-4.4.44-1-pve root=/dev/mapper/c31018h54224-root ro  quiet

Which was the previous entry before update I am assuming..
Everything booted up for me, But this still didn't solve my problem of trying to update.

Code:

root@proxmox:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following package was automatically installed and is no longer required:
  netcat6
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  pve-kernel-4.4.59-1-pve
The following NEW packages will be installed:
  pve-kernel-4.4.59-1-pve
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
11 not fully installed or removed.
Need to get 0 B/44.3 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 70101 files and directories currently installed.)
Preparing to unpack .../pve-kernel-4.4.59-1-pve_4.4.59-87_amd64.deb ...
Unpacking pve-kernel-4.4.59-1-pve (4.4.59-87) ...
dpkg: error processing archive /var/cache/apt/archives/pve-kernel-4.4.59-1-pve_4                                     .4.59-87_amd64.deb (--unpack):
 cannot copy extracted data for './boot/System.map-4.4.59-1-pve' to '/boot/Syste                                     m.map-4.4.59-1-pve.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.4.59-1-pve /boot/vml                                     inuz-4.4.59-1-pve
update-initramfs: Deleting /boot/initrd.img-4.4.59-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.4.59-1-pve /boot/vmli                                     nuz-4.4.59-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.40-1-pve
Found initrd image: /boot/initrd.img-4.4.40-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
Found linux image: /boot/vmlinuz-3.16.0-4-amd64
Found initrd image: /boot/initrd.img-3.16.0-4-amd64
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
done
Errors were encountered while processing:
 /var/cache/apt/archives/pve-kernel-4.4.59-1-pve_4.4.59-87_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Now I have to edit the grub.cfg again if I want to be able to boot. Last things I did on the proxmox node was clone a machine and upload an ISO from the proxmox webgui which failed 3 times all the way to 100% and still showed no file uploaded.
Edit: So my update was not able to complete due to lack of space on boot.

udo · May 20, 2017

Hi,
why you don't purge old kernels?

Esp. the old debian kernel 3.2 + 3.16?

I remove after updates the oldest kernel (keep the new and the before running one).

Udo

DeepCoreDev · May 20, 2017

I tried to purge with apt-get purge and dpkg -r and it couldn't, it said that they are not found even though the names are spot on. I only way I can think of deleting them is from the systemrescuecd by mounting sda1 to /boot/ and to just run rem on the old versions. Which is what I am probably going to do later tonight when the server is less active.

DeepCoreDev · May 22, 2017

Code:

root@proxmox:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following package was automatically installed and is no longer required:
  netcat6
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  pve-kernel-4.4.59-1-pve
The following NEW packages will be installed:
  pve-kernel-4.4.59-1-pve
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
3 not fully installed or removed.
Need to get 0 B/44.3 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 70101 files and directories currently installed.)
Preparing to unpack .../pve-kernel-4.4.59-1-pve_4.4.59-87_amd64.deb ...
Unpacking pve-kernel-4.4.59-1-pve (4.4.59-87) ...
Setting up linux-image-3.16.0-4-amd64 (3.16.39-1+deb8u2) ...
Internal Error: Could not find image (/boot/vmlinuz-3.16.0-4-amd64)
dpkg: error processing package linux-image-3.16.0-4-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 2
Setting up pve-kernel-4.4.59-1-pve (4.4.59-87) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
update-initramfs: Generating /boot/initrd.img-4.4.59-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.59-1-pve
Found initrd image: /boot/initrd.img-4.4.59-1-pve
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.40-1-pve
Found initrd image: /boot/initrd.img-4.4.40-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
done
Setting up proxmox-ve (4.4-87) ...
Setting up pve-kernel-4.4.44-1-pve (4.4.44-84) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
update-initramfs: Generating /boot/initrd.img-4.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.59-1-pve
Found initrd image: /boot/initrd.img-4.4.59-1-pve
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found initrd image: /boot/initrd.img-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.40-1-pve
Found initrd image: /boot/initrd.img-4.4.40-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
done
Errors were encountered while processing:
 linux-image-3.16.0-4-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I removed the all the 3.X versions in boot, tried and run apt-get -f install and it's saying it can't find one of the old Linux files? "/boot/vmlinuz-3.16.0-4-amd64" Why Is it trying to use that old file.
I am running 4.4.40-1-pve.
Kernel Version
Linux 4.4.40-1-pve #1 SMP PVE 4.4.40-82 (Thu, 23 Feb 2017 15:14:06 +0100)

udo · May 22, 2017

DeepCoreDev said:

Code:

root@proxmox:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following package was automatically installed and is no longer required:
  netcat6
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  pve-kernel-4.4.59-1-pve
The following NEW packages will be installed:
  pve-kernel-4.4.59-1-pve
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
3 not fully installed or removed.
Need to get 0 B/44.3 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 70101 files and directories currently installed.)
Preparing to unpack .../pve-kernel-4.4.59-1-pve_4.4.59-87_amd64.deb ...
Unpacking pve-kernel-4.4.59-1-pve (4.4.59-87) ...
Setting up linux-image-3.16.0-4-amd64 (3.16.39-1+deb8u2) ...
Internal Error: Could not find image (/boot/vmlinuz-3.16.0-4-amd64)
dpkg: error processing package linux-image-3.16.0-4-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 2
Setting up pve-kernel-4.4.59-1-pve (4.4.59-87) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
update-initramfs: Generating /boot/initrd.img-4.4.59-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.4.59-1-pve /boot/vmlinuz-4.4.59-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.59-1-pve
Found initrd image: /boot/initrd.img-4.4.59-1-pve
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.40-1-pve
Found initrd image: /boot/initrd.img-4.4.40-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
done
Setting up proxmox-ve (4.4-87) ...
Setting up pve-kernel-4.4.44-1-pve (4.4.44-84) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
update-initramfs: Generating /boot/initrd.img-4.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.4.44-1-pve /boot/vmlinuz-4.4.44-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.59-1-pve
Found initrd image: /boot/initrd.img-4.4.59-1-pve
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found initrd image: /boot/initrd.img-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.40-1-pve
Found initrd image: /boot/initrd.img-4.4.40-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
done
Errors were encountered while processing:
 linux-image-3.16.0-4-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I removed the all the 3.X versions in boot, tried and run apt-get -f install and it's saying it can't find one of the old Linux files? "/boot/vmlinuz-3.16.0-4-amd64" Why Is it trying to use that old file.
I am running 4.4.40-1-pve.
Kernel Version
Linux 4.4.40-1-pve #1 SMP PVE 4.4.40-82 (Thu, 23 Feb 2017 15:14:06 +0100)

Hi,
look with dpkg:

Code:

dpkg -l | grep linux-image

and purge the old 3.16 kernel with "apt purge linux-image-3.16.0-4-amd64".

Udo

DeepCoreDev · May 22, 2017

Thank you that worked perfectly, I ran apt-get dist-upgrade and it finished for the most part. Said there was not enough space on device, so I deleted the oldest pve kernel and freed up a little more and ran the upgrade again. Now it's fully updated thank you very much for the assistance.

Code:

root@proxmox:~# dpkg -l | grep linux-image
ii  linux-image-3.2.0-4-amd64      3.2.82-1                         amd64        Linux 3.2 for 64-bit PCs

Search

Search

[SOLVED] Proxmox VE 4.4 Kernel Update Failed

DeepCoreDev

Member

DeepCoreDev

Member

DeepCoreDev

Member

udo

Distinguished Member

DeepCoreDev

Member

DeepCoreDev

Member

udo

Distinguished Member

DeepCoreDev

Member