Update problem - hang at update-initramfs

[Rob Freeman]

# pveversion
pve-manager/5.3-5/97ae681d (running kernel: 4.15.18-7-pve)

I logged in to update my proxmox today and noticed that

apt upgrade

gave an error:

# apt upgrade
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

So I dutifully ran it and observed it hung at the end of this output:

sh-4.4# dpkg --configure -a
Setting up initramfs-tools (0.130) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Processing triggers for systemd (232-25+deb9u6) ...
Setting up pve-kernel-4.15.18-9-pve (4.15.18-30) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.15.18-9-pve /boot/vmlinuz-4.15.18-9-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.15.18-9-pve /boot/vmlinuz-4.15.18-9-pve
update-initramfs: Generating /boot/initrd.img-4.15.18-9-pve
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file

<<<<<< HANG >>>>>>

Thinking this may be a disk-space problem, I checked - it isn't:

# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                      32G     0   32G   0% /dev
tmpfs                    6.3G  666M  5.7G  11% /run
/dev/md2                  20G  3.6G   15G  20% /
tmpfs                     32G   31M   32G   1% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                     32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                510M  152K  510M   1% /boot/efi
/dev/mapper/pve-data     217G  161G   47G  78% /var/lib/vz
/dev/fuse                 30M   56K   30M   1% /etc/pve

I let it run and ps shows this after waiting ~30 minutes:

# ps aux | grep dpkg
root       414  0.0  0.0  19208  4936 pts/7    S+   16:30   0:00 dpkg --configure -a
root       591  0.0  0.0  17504  4336 pts/7    S+   16:30   0:00 /usr/bin/perl -w /var/lib/dpkg/info/pve-kernel-4.15.18-9-pve.postinst configure

I kill the processes (gracefully):

kill -15 591
kill -15 414

After killing the configure process, scrollback shows:

cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
dpkg: error processing package pve-kernel-4.15.18-9-pve (--configure):
 subprocess installed post-installation script was killed by signal (Terminated)
Setting up libvotequorum8:amd64 (2.4.4-pve1) ...
Setting up libcpg4:amd64 (2.4.4-pve1) ...
Setting up libx11-6:amd64 (2:1.6.4-3+deb9u1) ...
Setting up libgd3:amd64 (2.2.4-2+deb9u3) ...
Setting up libcmap4:amd64 (2.4.4-pve1) ...
Setting up libquorum5:amd64 (2.4.4-pve1) ...
Setting up libtotem-pg5:amd64 (2.4.4-pve1) ...
dpkg: dependency problems prevent configuration of pve-kernel-4.15:
 pve-kernel-4.15 depends on pve-kernel-4.15.18-9-pve; however:
  Package pve-kernel-4.15.18-9-pve is not configured yet.

dpkg: error processing package pve-kernel-4.15 (--configure):
 dependency problems - leaving unconfigured
Setting up ceph-common (10.2.11-2) ...
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-4.15; however:
  Package pve-kernel-4.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Setting up libnginx-mod-http-image-filter (1.10.3-1+deb9u2) ...
Setting up nginx-full (1.10.3-1+deb9u2) ...
Upgrading binary: nginx.
Setting up corosync (2.4.4-pve1) ...
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
dpkg: error processing package pve-kernel-4.15.18-9-pve (--configure):
 subprocess installed post-installation script was killed by signal (Terminated)
Setting up libvotequorum8:amd64 (2.4.4-pve1) ...
Setting up libcpg4:amd64 (2.4.4-pve1) ...
Setting up libx11-6:amd64 (2:1.6.4-3+deb9u1) ...
Setting up libgd3:amd64 (2.2.4-2+deb9u3) ...
Setting up libcmap4:amd64 (2.4.4-pve1) ...
Setting up libquorum5:amd64 (2.4.4-pve1) ...
Setting up libtotem-pg5:amd64 (2.4.4-pve1) ...
dpkg: dependency problems prevent configuration of pve-kernel-4.15:
 pve-kernel-4.15 depends on pve-kernel-4.15.18-9-pve; however:
  Package pve-kernel-4.15.18-9-pve is not configured yet.

dpkg: error processing package pve-kernel-4.15 (--configure):
 dependency problems - leaving unconfigured
Setting up ceph-common (10.2.11-2) ...
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-4.15; however:
  Package pve-kernel-4.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Setting up libnginx-mod-http-image-filter (1.10.3-1+deb9u2) ...
Setting up nginx-full (1.10.3-1+deb9u2) ...
Upgrading binary: nginx.
Setting up corosync (2.4.4-pve1) ...
Removing obsolete conffile /etc/init/corosync.conf ...
Setting up pve-cluster (5.0-31) ...
Setting up libpve-storage-perl (5.0-33) ...
Setting up libpve-access-control (5.1-3) ...
Setting up nginx (1.10.3-1+deb9u2) ...
Setting up pve-container (2.0-31) ...
Setting up pve-firewall (3.0-16) ...
Setting up qemu-server (5.0-43) ...
Created symlink /etc/systemd/system/multi-user.target.wants/qmeventd.service → /lib/systemd/system/qmeventd.service.
Setting up pve-manager (5.3-5) ...
Processing triggers for pve-ha-manager (2.0-5) ...
Processing triggers for initramfs-tools (0.130) ...
update-initramfs: Generating /boot/initrd.img-4.15.18-9-pve
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file

<<<<<< HANG >>>>>>

Rinse and repeat.

Any tips please?

 

[wolfgang]

Hi,

as far I see it comes from the network interfaces.
You have custom network udev rules.
The problem is you lose network connectivity when the udev rule is applied.

Try to reconfigure the packages on the server directly.
Do not use ssh or any network connection, this will fail.

 

[Rob Freeman]

Thanks for responding - this is interesting since I haven't made any manual adjustments to any udev configuration files, literally none.

However, I took a closer look at the udev rules file that it's hanging on and this is the content (mac address obfuscated):

# cat /etc/udev/rules.d/70-persistent-net.rules
# rule added automagically by OVH installer - keep it unless you know what you are doing
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="0c:c4:xx:xx:xx:xx", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eno1"

What ought that line be?

I haven't lost network connectivity while trying thus far and I've been running this inside a screen session (just in case):

# dpkg --configure -a

I've searched the forum and found a bunch of links (shown below) however these don't help fix this issue.

https://forum.proxmox.com/threads/proxmox-5-ethernet-issues.35556/
https://forum.proxmox.com/threads/pve5-fix-those-predictable-network-interface-names.37210/
https://forum.proxmox.com/threads/no-70-persistent-net-rules-on-proxmox-5.38501/

 

[wolfgang]

What ought that line be?

This is just a renaming of the nic.
I guess OVH has setup scripts what assume a nic called eth* and not the predictable name as Debian uses now.

Generall this rule is not needed but when you remove it you will lose network because of the name of the nic change.

I do not know this behavior.
try to remove the kernel 4.15.18.9 and install the 4.15.18-10

 

[Rob Freeman]

try to remove the kernel 4.15.18.9 and install the 4.15.18-10

I'm trying again today and can see 4.15.18-11-pve is trying to be installed:

Found linux image: /boot/vmlinuz-4.15.18-11-pve
Found initrd image: /boot/initrd.img-4.15.18-11-pve
Found linux image: /boot/vmlinuz-4.15.18-9-pve
Found initrd image: /boot/initrd.img-4.15.18-9-pve
Found linux image: /boot/vmlinuz-4.15.18-7-pve
Found initrd image: /boot/initrd.img-4.15.18-7-pve
Adding boot menu entry for EFI firmware configuration
done
Setting up pve-kernel-4.15.18-11-pve (4.15.18-33) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.15.18-11-pve /boot/vmlinuz-4.15.18-11-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.15.18-11-pve /boot/vmlinuz-4.15.18-11-pve
update-initramfs: Generating /boot/initrd.img-4.15.18-11-pve
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file

Again it hangs - cancelling it shows this:

^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-4.15.18-11-pve.postinst line 19.

I have a feeling something in the pve updates has hosed this system...

 

[fabian]

if it is update-initramfs that is hanging, try running it manually with '-v'.

update-initramfs does a (global!) sync at the end of generating each initrd, which could hang if you have a mount point that is blocking (e.g., an NFS share that is unresponsive?). this behaviour will finally be fixed in Buster, where it only syncs the file system that contains the initrd.

 

[Rob Freeman]

Thanks for the hint - I checked my filesystems in /etc/fstab and found that there was indeed an entry for an old LV which had been removed. I deleted the entry in fstab and verified it wasn't listed under the current `mount`. All filesystems are now correct and corresponding to /etc/fstab.

So, I tried again, after making modifications to these files to enable the -v arg against update-initramfs:
/etc/kernel/postinst.d/initramfs-tools
/var/lib/dpkg/info/initramfs-tools.postinst

This time, as would be expected the output was much more detailed. I've pasted the full output here and here's the tail'd snippet:

Calling hook dmsetup
Calling hook klibc-utils
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
Calling hook netdevnames
cp: '/etc/udev/rules.d/70-persistent-net.rules' and '/etc/udev/rules.d/70-persistent-net.rules' are the same file
Calling hook reiserfsprogs
/usr/share/initramfs-tools/scripts/local-block/ORDER ignored: not executable
/usr/share/initramfs-tools/scripts/local-top/ORDER ignored: not executable
/usr/share/initramfs-tools/scripts/local-bottom/ORDER ignored: not executable
/usr/share/initramfs-tools/scripts/local-premount/ORDER ignored: not executable
/usr/share/initramfs-tools/scripts/init-top/ORDER ignored: not executable
/usr/share/initramfs-tools/scripts/init-bottom/ORDER ignored: not executable
Building cpio /boot/initrd.img-4.15.18-11-pve.new initramfs

Progress: [  0%] [...................

In short - it's still hanging on that cpio command. I checked the directory where it's trying to write and sure enough, there's a file there alright:

# ls -hal /boot/initrd.*
-rw-r--r-- 1 root root 35M Feb 11 13:08 /boot/initrd.img-4.15.18-11-pve
-rw-r--r-- 1 root root 35M Feb 11 13:05 /boot/initrd.img-4.15.18-11-pve.dpkg-bak
-rw-r--r-- 1 root root 34M Oct 22 13:21 /boot/initrd.img-4.15.18-7-pve
-rw-r--r-- 1 root root 35M Feb  7 10:45 /boot/initrd.img-4.15.18-9-pve

Back to scratching my head...

 

[fabian]

please verify that it is really cpio that is hanging - the next steps after generating the archive are

• moving the .new archive to the final location
• calculating a checksum
• calling 'sync'

since there is no .new file (which happens directly after calling cpio), and the initrd in question is not truncated (which happens directly before calling cpio), my money is on hanging in 'sync' (you can easily check this by looking at the process tree below the dpkg call).