Proxmox 6-2-6 stuck during update / help request or any suggestions?

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
Hi, I have a small weird problem, wondering if anyone has seen this. Proxmox 6.x host running on a stock template install on OVH environment (SW raid 2x2tb config which OVH provide and I've used many times before without issue).

Normally when I do updates on such an environment, I do
Code:
apt-get update

(wait / proceed)

apt-get upgrade

(wait / proceed)

then

apt-get dist-upgrade

(wait/proceed)
then all is good and happy days.
(I find some other forum posts that suggest now one should not do apt-get upgrade at all? but only dist-upgrade?)

However, this time - it seems to be getting stuck, right now here:

Code:
... lots of output from updates...
then...
Setting up pve-kernel-5.4.41-1-pve (5.4.41-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.41-1-pve /boot/vmlinuz-5.4.41-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.41-1-pve /boot/vmlinuz-5.4.41-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.41-1-pve

and then, nothing. I've left it overnight in a 'screen' session and it just won't push past this. I am not sure what it is waiting for.

I googled and found a similar discussion in the forum, and it had suggested a few things to validate grub and md / disk grub setup, approx hints below are shown:

Code:
root@proxmox6:~#  grub-probe --target=device /
/dev/md2

root@proxmox6:~#  grub-probe --target=device /boot
/dev/md2

root@proxmox6:~#  grub-probe --device /dev/md2 --target=fs_uuid
6fb5fb9e-188e-457d-8310-f9985659e25d

root@proxmox6:~#  cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

# openstack-debian-images disables OS prober to avoid
# loopback detection which breaks booting
GRUB_DISABLE_OS_PROBER=true
GRUB_DEVICE=UUID=6fb5fb9e-188e-457d-8310-f9985659e25d
root@proxmox6:~#



SO, HERE WE ARE:

root@proxmox6:~# apt-get dist-upgrade
E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

(other update did not exit cleanly so locks were still in place - remove them manually).

root@proxmox6:~# rm /var/lib/dpkg/lock-frontend
root@proxmox6:~# rm /var/lib/dpkg/lock

Then try to kick it along again:

root@proxmox6:~# apt-get dist-upgrade

E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

OK, fine, we do it your way:
root@proxmox6:~# dpkg --configure -a
Setting up pve-kernel-5.4.41-1-pve (5.4.41-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.41-1-pve /boot/vmlinuz-5.4.41-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.41-1-pve /boot/vmlinuz-5.4.41-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.41-1-pve

(wait forever, have a nice day)

So. Right now I am pretty sure if I reboot this box, it will not come back up, which would be a drama/hassle I prefer to avoid.

I'm not sure how to force it to fix itself up. I've tried to remove and reinstall the kernel package but that didn't go so well.

Clearly it is grumpy trying to generate the new initrd boot image file but having some problem.

Sigh.

Any hints are greatly appreciated.

Otherwise plan B, which arguably is not impossible, just feels stupid.
-- make backup of VMs on the box. Make backup of config (really not all that much ...)
-- blow it away using template OVH proxmox install, re-do from scratch onto the bare metal setup.
-- update to latest proxmox, before doing anything further.
-- restore config and VM, and happy days. Just an hour or two of outage for some fun admin work. Whee.
-- but I am happier if I don't have to go that route since ?it is more work, and more gross? and ideally this sort of problem is one that I would prefer to know how to actually fix instead of work-around.

Thanks,

Tim
 

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
Brief ping followup, I am guessing based on the feedback (ie, none) that I will proceed with the plan-B solution to this (ie, backup, nuke, restore). Woot.
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,620
959
163

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
Hi, yes, I realize mdraid is not supported by the proxmox team in any way. I guess I was more posting to the forum, in case some other non-proxmox-team-person who also uses it, has seen this before. or alternately, at a more general scope (nothing to do with MDraid) - if anyone had ever seen this kind of situation before, where proxmox updates are broken / and fail to install updates at the step I have described.

I've used mdraid proxmox for quite a few years (and mdraid on vanilla linux, nothing to do with proxmox) / on quite a lot of servers and never been in this situation before with any such setup. So, a learning opportunity.

Hence the post to forum.

thanks,

Tim
 

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
Little followup note, in case this might help anyone else out. I think I might have figured it out.

Google search for keywords took me to https://unix.stackexchange.com/questions/428001/update-initramfs-hangs-on-debian-stretch
where it seemed like a good idea to try this:

Code:
Building the new initramfs file can take minutes.
Find the PID of the cpio process under update-initramfs in the output of ps af,
and run strace -p <PID> to see if it's chugging along.

Also check your kern.log for disk errors. – Ferenc Wágner Mar 4 '18 at 9:29

So based on this, I opened up 2 x ssh sessions into my server with issues. In the first one I kicked it along so it was hanging on the update. Specficically I manually started the command as root, "update-initramfs -k all -c" to force an update of initrd boot images.

Then in the second ssh session,

Code:
root@proxmox6:~# ps af
  PID TTY      STAT   TIME COMMAND
12068 ?        Ss+    0:00 /sbin/agetty --noclear --keep-baud tty1 115200,38400,9600 linux
12067 pts/1    Ss+    0:00 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 linux
12066 pts/1    Ss+    0:00 /sbin/agetty --noclear --keep-baud tty2 115200,38400,9600 linux
8308 pts/15   Ss     0:00 -bash
8352 pts/15   R+     0:00  \_ ps af
5297 pts/14   Ss     0:00 -bash
5645 pts/14   S+     0:00  \_ /bin/sh /usr/sbin/update-initramfs -k all -c
5651 pts/14   S+     0:00      \_ /bin/sh /usr/sbin/update-initramfs -c -k 4.19.0-5-cloud-amd64 -b /boot
5653 pts/14   S+     0:00          \_ /bin/sh /usr/sbin/mkinitramfs -o /boot/initrd.img-4.19.0-5-cloud-amd64.new 4.19.0-5-cloud-amd64
8119 pts/14   S+     0:00              \_ /bin/sh /usr/share/initramfs-tools/hooks/mdadm
8232 pts/14   S+     0:00                  \_ /bin/sh /usr/share/mdadm/mkconf
8238 pts/14   D+     0:00                      \_ /sbin/mdadm --examine --scan --config=partitions
1075 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root@proxmox6:~#

which seems to hint it is indeed having trouble figuring out the current mdadmin config maybe

which led me to take a look in /etc/mdadm

where I noted my /etc/mdadm/mdadm.conf was present but not quite complete IMHO.

and also that there was another /etc/mdadm.conf file, which contained only,

Code:
root@proxmox6:/# cat /etc/mdadm.conf
ARRAY /dev/md2 UUID=40feccf8:866195dd:a4d2adc2:26fd5302
ARRAY /dev/md4 UUID=691852e1:f72e73bc:a4d2adc2:26fd5302
So, my test for fun,

Code:
cat /etc/mdadm.conf >> /etc/mdadm/mdadm.conf
so that now I've got,

Code:
root@proxmox6:~# cd /etc/mdadm/
root@proxmox6:/etc/mdadm# ls -la
total 12
drwxr-xr-x   2 root root 4096 Oct 14  2019 .
drwxr-xr-x 101 root root 4096 Jun 21 10:40 ..
-rw-r--r--   1 root root  763 Jun 21 11:10 mdadm.conf
root@proxmox6:/etc/mdadm# cat mdadm.conf
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# This configuration was auto-generated on Mon, 14 Oct 2019 15:11:02 +0000 by mkconf
ARRAY /dev/md2 UUID=40feccf8:866195dd:a4d2adc2:26fd5302
ARRAY /dev/md4 UUID=691852e1:f72e73bc:a4d2adc2:26fd5302


and note as a reference point that md2 and md4 do indeed 'seem to be sane' based on what I have in /proc/mdstat, ie,

root@proxmox6:/etc/mdadm# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md4 : active raid1 sdb4[1] sda4[0]
      1931489216 blocks [2/2] [UU]
      bitmap: 2/15 pages [8KB], 65536KB chunk

md2 : active raid1 sda2[0] sdb2[1]
      20970432 blocks [2/2] [UU]

unused devices: <none>



and now, miracle, the thing no longer stalls when I kick along the apt-get update ; apt-get dist-upgrade

instead we get more-happy-looking output, thus,

Code:
root@proxmox6:~# apt-get update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster-updates InRelease
Hit:3 http://security.debian.org buster/updates InRelease
Hit:4 http://deb.debian.org/debian buster-backports InRelease
Hit:5 http://download.proxmox.com/debian/pve buster InRelease
Reading package lists... Done
root@proxmox6:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libgnutls-dane0 libunbound8
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  pve-kernel-5.4.44-1-pve
The following packages will be upgraded:
  pve-edk2-firmware pve-kernel-5.4 pve-kernel-helper
3 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 62.0 MB of archives.
After this operation, 287 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://download.proxmox.com/debian/pve buster/pve-no-subscription amd64 pve-edk2-firmware all 2.20200531-1 [1,710 kB]
Get:2 http://download.proxmox.com/debian/pve buster/pve-no-subscription amd64 pve-kernel-5.4.44-1-pve amd64 5.4.44-1 [60.3 MB]
Get:3 http://download.proxmox.com/debian/pve buster/pve-no-subscription amd64 pve-kernel-5.4 all 6.2-3 [3,452 B]
Get:4 http://download.proxmox.com/debian/pve buster/pve-no-subscription amd64 pve-kernel-helper all 6.2-3 [9,368 B]
Fetched 62.0 MB in 8s (8,230 kB/s)
(Reading database ... 64281 files and directories currently installed.)
Preparing to unpack .../pve-edk2-firmware_2.20200531-1_all.deb ...
Unpacking pve-edk2-firmware (2.20200531-1) over (2.20200229-1) ...
Selecting previously unselected package pve-kernel-5.4.44-1-pve.
Preparing to unpack .../pve-kernel-5.4.44-1-pve_5.4.44-1_amd64.deb ...
Unpacking pve-kernel-5.4.44-1-pve (5.4.44-1) ...
Preparing to unpack .../pve-kernel-5.4_6.2-3_all.deb ...
Unpacking pve-kernel-5.4 (6.2-3) over (6.2-2) ...
Preparing to unpack .../pve-kernel-helper_6.2-3_all.deb ...
Unpacking pve-kernel-helper (6.2-3) over (6.2-2) ...
Setting up pve-kernel-helper (6.2-3) ...
Setting up pve-kernel-5.4.44-1-pve (5.4.44-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/pve-auto-removal 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-pve-efiboot 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.44-1-pve
Found initrd image: /boot/initrd.img-5.4.44-1-pve
Found linux image: /boot/vmlinuz-5.4.41-1-pve
Found initrd image: /boot/initrd.img-5.4.41-1-pve
Found linux image: /boot/vmlinuz-5.0.21-5-pve
Found initrd image: /boot/initrd.img-5.0.21-5-pve
Found linux image: /boot/vmlinuz-4.19.0-5-cloud-amd64
Found initrd image: /boot/initrd.img-4.19.0-5-cloud-amd64
done
Setting up pve-edk2-firmware (2.20200531-1) ...
Setting up pve-kernel-5.4 (6.2-3) ...


check again are we good?

root@proxmox6:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libgnutls-dane0 libunbound8
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@proxmox6:~#

So. I have not rebooted the box yet. But I will do so shortly / next while and I will followup to thread after that.

woot.



Tim
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!