upgrade interrupted, dpkg half-configured server not booting

P

pixelplumber

Renowned Member
Proxmox Subscriber
Oct 21, 2009
46
1
73
Australia
This is the proverbial face-palm moment.

The most recent batch of updates from the pve-enterprise repository to my server were interrupted before completion. The facepalm from myself is that I ended up rebooting anyway.

Proxmox is now hanging during the boot process. It hangs around the point where dlm_controld starts, after this point it hangs indefinitely.

Restarting and changing kernels to an earlier version does not help. I can boot to the recovery kernel/maintenance mode however, so I assume it is a problem with one of the installed/updated deamons.

Doing some investigating it appears that a lot of the packages in /var/log/dpkg.log are in a half-configured state.

I tried apt-get install --reinstall (and dpkg --configure -a) for each of the packages in half-configured state below, and restarted with no luck.

Code: 
2013-10-01 16:39:20 startup archives unpack
2013-10-01 16:39:20 upgrade apt-transport-https:amd64 0.9.7.9 0.9.7.10
2013-10-01 16:39:20 status half-configured apt-transport-https:amd64 0.9.7.9
2013-10-01 16:39:20 status unpacked apt-transport-https:amd64 0.9.7.9
2013-10-01 16:39:20 status half-installed apt-transport-https:amd64 0.9.7.9
2013-10-01 16:39:20 status half-installed apt-transport-https:amd64 0.9.7.9
2013-10-01 16:39:20 status unpacked apt-transport-https:amd64 0.9.7.10
2013-10-01 16:39:20 status unpacked apt-transport-https:amd64 0.9.7.10
2013-10-01 16:39:20 upgrade ceph-common:amd64 0.67.2-1~bpo70+1 0.67.3-1~bpo70+1
2013-10-01 16:39:20 status half-configured ceph-common:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:20 status unpacked ceph-common:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:20 status half-installed ceph-common:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:20 status triggers-pending man-db:amd64 2.6.2-1
2013-10-01 16:39:20 status half-installed ceph-common:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed ceph-common:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status unpacked ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 upgrade librbd1:amd64 0.67.2-1~bpo70+1 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status half-configured librbd1:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked librbd1:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed librbd1:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed librbd1:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed librbd1:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked librbd1:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status unpacked librbd1:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 upgrade librados2:amd64 0.67.2-1~bpo70+1 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status half-configured librados2:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked librados2:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed librados2:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed librados2:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked librados2:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status unpacked librados2:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 upgrade python-ceph:amd64 0.67.2-1~bpo70+1 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status half-configured python-ceph:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked python-ceph:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed python-ceph:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status half-installed python-ceph:amd64 0.67.2-1~bpo70+1
2013-10-01 16:39:21 status unpacked python-ceph:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 status unpacked python-ceph:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:21 install libcrypt-ssleay-perl:amd64 <none> 0.58-1
2013-10-01 16:39:21 status half-installed libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:21 status half-installed libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:21 status unpacked libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:21 status unpacked libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:21 upgrade libpve-storage-perl:all 3.0-10 3.0-13
2013-10-01 16:39:21 status half-configured libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status unpacked libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status half-installed libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status triggers-pending pve-manager:amd64 3.1-4
2013-10-01 16:39:21 status half-installed libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status half-installed libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status half-installed libpve-storage-perl:all 3.0-10
2013-10-01 16:39:21 status unpacked libpve-storage-perl:all 3.0-13
2013-10-01 16:39:22 status unpacked libpve-storage-perl:all 3.0-13
2013-10-01 16:39:22 install pve-kernel-2.6.32-24-pve:amd64 <none> 2.6.32-111
2013-10-01 16:39:22 status half-installed pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:23 status unpacked pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:23 status unpacked pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:23 upgrade qemu-server:amd64 3.1-1 3.1-4
2013-10-01 16:39:23 status half-configured qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status unpacked qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status half-installed qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status half-installed qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status half-installed qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status half-installed qemu-server:amd64 3.1-1
2013-10-01 16:39:23 status unpacked qemu-server:amd64 3.1-4
2013-10-01 16:39:23 status unpacked qemu-server:amd64 3.1-4
2013-10-01 16:39:23 upgrade pve-manager:amd64 3.1-4 3.1-14
2013-10-01 16:39:23 status half-configured pve-manager:amd64 3.1-4
2013-10-01 16:39:23 status unpacked pve-manager:amd64 3.1-4
2013-10-01 16:39:23 status half-installed pve-manager:amd64 3.1-4
2013-10-01 16:39:23 status half-installed pve-manager:amd64 3.1-4
2013-10-01 16:39:24 status half-installed pve-manager:amd64 3.1-4
2013-10-01 16:39:24 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:24 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:24 upgrade proxmox-ve-2.6.32:all 3.1-109 3.1-111
2013-10-01 16:39:24 status half-configured proxmox-ve-2.6.32:all 3.1-109
2013-10-01 16:39:24 status unpacked proxmox-ve-2.6.32:all 3.1-109
2013-10-01 16:39:24 status half-installed proxmox-ve-2.6.32:all 3.1-109
2013-10-01 16:39:24 status half-installed proxmox-ve-2.6.32:all 3.1-109
2013-10-01 16:39:24 status unpacked proxmox-ve-2.6.32:all 3.1-111
2013-10-01 16:39:24 status unpacked proxmox-ve-2.6.32:all 3.1-111
2013-10-01 16:39:24 upgrade python-openssl:amd64 0.13-2 0.13-2+deb7u1
2013-10-01 16:39:24 status half-configured python-openssl:amd64 0.13-2
2013-10-01 16:39:24 status triggers-pending python-support:all 1.0.15
2013-10-01 16:39:24 status unpacked python-openssl:amd64 0.13-2
2013-10-01 16:39:24 status half-installed python-openssl:amd64 0.13-2
2013-10-01 16:39:24 status half-installed python-openssl:amd64 0.13-2
2013-10-01 16:39:24 status unpacked python-openssl:amd64 0.13-2+deb7u1
2013-10-01 16:39:24 status unpacked python-openssl:amd64 0.13-2+deb7u1
2013-10-01 16:39:24 upgrade pve-libspice-server1:amd64 0.12.4-1 0.12.4-2
2013-10-01 16:39:24 status half-configured pve-libspice-server1:amd64 0.12.4-1
2013-10-01 16:39:24 status unpacked pve-libspice-server1:amd64 0.12.4-1
2013-10-01 16:39:24 status half-installed pve-libspice-server1:amd64 0.12.4-1
2013-10-01 16:39:24 status half-installed pve-libspice-server1:amd64 0.12.4-1
2013-10-01 16:39:24 status unpacked pve-libspice-server1:amd64 0.12.4-2
2013-10-01 16:39:24 status unpacked pve-libspice-server1:amd64 0.12.4-2
2013-10-01 16:39:24 trigproc man-db:amd64 2.6.2-1 2.6.2-1
2013-10-01 16:39:24 status half-configured man-db:amd64 2.6.2-1
2013-10-01 16:39:25 status installed man-db:amd64 2.6.2-1
2013-10-01 16:39:25 trigproc python-support:all 1.0.15 1.0.15
2013-10-01 16:39:25 status half-configured python-support:all 1.0.15
2013-10-01 16:39:26 status installed python-support:all 1.0.15
2013-10-01 16:39:27 startup packages configure
2013-10-01 16:39:27 configure apt-transport-https:amd64 0.9.7.10 <none>
2013-10-01 16:39:27 status unpacked apt-transport-https:amd64 0.9.7.10
2013-10-01 16:39:27 status half-configured apt-transport-https:amd64 0.9.7.10
2013-10-01 16:39:27 status installed apt-transport-https:amd64 0.9.7.10
2013-10-01 16:39:27 configure librados2:amd64 0.67.3-1~bpo70+1 <none>
2013-10-01 16:39:27 status unpacked librados2:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status half-configured librados2:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status installed librados2:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 configure librbd1:amd64 0.67.3-1~bpo70+1 <none>
2013-10-01 16:39:27 status unpacked librbd1:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status half-configured librbd1:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status installed librbd1:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 configure python-ceph:amd64 0.67.3-1~bpo70+1 <none>
2013-10-01 16:39:27 status unpacked python-ceph:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status half-configured python-ceph:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status installed python-ceph:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 configure ceph-common:amd64 0.67.3-1~bpo70+1 <none>
2013-10-01 16:39:27 status unpacked ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status unpacked ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status unpacked ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status half-configured ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 status installed ceph-common:amd64 0.67.3-1~bpo70+1
2013-10-01 16:39:27 configure libcrypt-ssleay-perl:amd64 0.58-1 <none>
2013-10-01 16:39:27 status unpacked libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:27 status half-configured libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:27 status installed libcrypt-ssleay-perl:amd64 0.58-1
2013-10-01 16:39:27 configure libpve-storage-perl:all 3.0-13 <none>
2013-10-01 16:39:27 status unpacked libpve-storage-perl:all 3.0-13
2013-10-01 16:39:27 status half-configured libpve-storage-perl:all 3.0-13
2013-10-01 16:39:27 status installed libpve-storage-perl:all 3.0-13
2013-10-01 16:39:27 configure pve-kernel-2.6.32-24-pve:amd64 2.6.32-111 <none>
2013-10-01 16:39:27 status unpacked pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:27 status half-configured pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:41 status installed pve-kernel-2.6.32-24-pve:amd64 2.6.32-111
2013-10-01 16:39:41 configure qemu-server:amd64 3.1-4 <none>
2013-10-01 16:39:41 status unpacked qemu-server:amd64 3.1-4
2013-10-01 16:39:41 status unpacked qemu-server:amd64 3.1-4
2013-10-01 16:39:41 status half-configured qemu-server:amd64 3.1-4
2013-10-01 16:39:41 status installed qemu-server:amd64 3.1-4
2013-10-01 16:39:41 configure pve-manager:amd64 3.1-14 <none>
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status unpacked pve-manager:amd64 3.1-14
2013-10-01 16:39:41 status half-configured pve-manager:amd64 3.1-14
2013-10-01 16:39:53 status installed pve-manager:amd64 3.1-14
2013-10-01 16:39:53 configure proxmox-ve-2.6.32:all 3.1-111 <none>
2013-10-01 16:39:53 status unpacked proxmox-ve-2.6.32:all 3.1-111
2013-10-01 16:39:53 status half-configured proxmox-ve-2.6.32:all 3.1-111
2013-10-01 16:39:53 status installed proxmox-ve-2.6.32:all 3.1-111
2013-10-01 16:39:53 configure python-openssl:amd64 0.13-2+deb7u1 <none>
2013-10-01 16:39:53 status unpacked python-openssl:amd64 0.13-2+deb7u1
2013-10-01 16:39:53 status half-configured python-openssl:amd64 0.13-2+deb7u1
2013-10-01 16:39:53 status installed python-openssl:amd64 0.13-2+deb7u1
2013-10-01 16:39:53 status triggers-pending python-support:all 1.0.15
2013-10-01 16:39:53 configure pve-libspice-server1:amd64 0.12.4-2 <none>
2013-10-01 16:39:53 status unpacked pve-libspice-server1:amd64 0.12.4-2
2013-10-01 16:39:53 status half-configured pve-libspice-server1:amd64 0.12.4-2
2013-10-01 16:39:53 status installed pve-libspice-server1:amd64 0.12.4-2
2013-10-01 16:39:53 trigproc python-support:all 1.0.15 <none>
2013-10-01 16:39:53 status half-configured python-support:all 1.0.15
2013-10-01 16:39:53 status installed python-support:all 1.0.15

Any advice for recovery? Any way to reinstall packages in place when they're half-configured like this? Or do you think a reinstall of proxmox is the way to go?

How will this affect the subscription key that was on this server? is there a way to back that up prior to reinstall?
 
If I boot to the maintenance mode and try starting proxmox manually I get the hang again on dlm_controld, the console displays a kernel message:

crash.jpg

At this stage I'm assuming the best thing to do is reinstall. I can't migrate machines as proxmox won't start, but I will copy vzdump files off manually and follow the instructions in the re-install cluster node howto on the wiki.
 
When booting into maintenance mode try the following:
1) dpkg --configure -a
2) apt-get -f install
3) apt-get dist-upgrade
4) reboot the server
 
Thanks for the suggestion.

When I do that, for the commands 2) & 3) I only get
---------------
Reading package lists... Done
Building dependaency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded
-----------

So I suppose it's not reinstalling the already installed packages. can you run a force-reinstall on the whole dist-upgrade without wiping out existing configuration?
 
A very very long shot, cannot think of any better way..... How about you pick the half-installed files one by one, remove them by apt-get remove then try to install them one by one from Maintenance console? Dont know how mission critical your current system is but may be you can at least get through the broken os.
 
symmcom said:
A very very long shot, cannot think of any better way..... How about you pick the half-installed files one by one, remove them by apt-get remove then try to install them one by one from Maintenance console? Dont know how mission critical your current system is but may be you can at least get through the broken os.
Click to expand...

Thanks for the suggestion.

My understanding is that apt-get --reinstall will purge and reinstall from scratch the package without requiring dependencies. So I would guess that action may have covered your suggestion (both actions in one command)? It doesn't look like there's an equivalent way to do that for dist-upgrade, I guess that's what a reinstall is for?

Unfortunately, this node has not had nightly backups due to space (only weekly). So I'm copying off current data before trying a reinstall. Fortunately though, the KVM disks and VZ data were manually moved to another volume group so a reinstall from the ISO should not clobber the existing disk images. Still copying them off just in case though.

Thanks for the help.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back