grub2 update problems for 3.4 (GUI only), leading to not correctly installed packages

[tom]

Doing a package upgrade via GUI is not working in some situations, due to the latest grub2 updates. the problem is already found and also fixed, but the fix is in the new packages and installing them via GUI will move you into this issue.

Best way to workaround the issue
do NOT use the GUI for upgrading to latest packages from (pvetest and pve-no-subscription repo), just update/upgrade via CLI (apt-get update && apt-get dist-upgrade)

If you already upgraded via GUI and the window disappeared and you have unconfigured packages
check your package status via 'pveversion -v' - if you have issues like (not correctly installed packages) do the following:

> killall dpkg
> dpkg --configure -a

This will kill the background dpkg process and continue configuration of all packages, including the new grub2.

In order to install grub2, you need to know the device where your Proxmox VE is installed. in most cases this is /dev/sda

you can check your device (harddisks) list this by analysing the output of:

> parted -l

if you want to manually install the grub2 into your MBR (in most cases to /dev/sda)

> grub-install /dev/sda

Finally, reboot your host to activate the new kernel.

 

[vcp_ai]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

Just updated (10 minutes ago) from cli:

aptitude update && aptitude dist-upgrade

ang got to

Wellcome to GRUB!
 
error: file not found.
Entering rescue mode ....
grub rescue >

I received a queation about keeping grub config, and selected default: Keep unchanged (or similar)

Any hint/help ?

 

[drott]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

Just updated (10 minutes ago) from cli:

aptitude update && aptitude dist-upgrade

ang got to

Wellcome to GRUB!
 
error: file not found.
Entering rescue mode ....
grub rescue >

I received a queation about keeping grub config, and selected default: Keep unchanged (or similar)

Any hint/help ?

Hi! There are a number of posts on this one but there is a grub repair tool that should help. supergrub. I was a bit cautious it just did not make sense with a reinstall of grub it did not pickup thr oiginal config. in my case this box started life as a 2.x.

The proxmox team was great and provided an answer sunday evening their time.

A grub-install /dev/sdx worked for me since i had yet to reboot. The only odd thing is I cannot select an older kernel. The submenu not there but it boots. yeh.

 

[Alex1]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

I'm using the enterprise repository and just updated using the web GUI, and this issue affects me too.

Log from apt-get dist-upgrade:

Start-Date: 2015-02-19  13:53:07Commandline: apt-get dist-upgrade
Install: zfs-doc:amd64 (0.6.3-2~wheezy, automatic), libnvpair1:amd64 (0.6.3-2~wheezy, automatic), numactl:amd64 (2.0.8~rc4-1, automatic), libparted2:amd64 (3.2-6~bpo70+1, automatic), libzpool2:amd64 (0.6.3-2~wheezy, automatic), libuutil1:amd64 (0.6.3-2~wheezy, automatic), libzfs2:amd64 (0.6.3-2~wheezy, automatic), pve-kernel-2.6.32-37-pve:amd64 (2.6.32-147, automatic), libnuma1:amd64 (2.0.8~rc4-1, automatic), libev-perl:amd64 (4.11-2, automatic)
Upgrade: libpve-storage-perl:amd64 (3.0-25, 3.0-31), pve-qemu-kvm:amd64 (2.1-10, 2.1-12), bind9-host:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), librbd1:amd64 (0.80.6-1~bpo70+1, 0.80.8-1~bpo70+1), ceph-common:amd64 (0.80.6-1~bpo70+1, 0.80.8-1~bpo70+1), libpve-common-perl:amd64 (3.0-19, 3.0-24), dnsutils:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), proxmox-ve-2.6.32:amd64 (3.3-139, 3.3-147), qemu-server:amd64 (3.3-3, 3.3-20), libdns88:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), librados2:amd64 (0.80.6-1~bpo70+1, 0.80.8-1~bpo70+1), parted:amd64 (2.3-12, 3.2-6~bpo70+1), grub-pc:amd64 (1.99-27+deb7u2, 2.02~bpo70+3), libisccc80:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), pve-manager:amd64 (3.3-5, 3.4-1), liblwres80:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), libleveldb1:amd64 (1.9.0-1~bpo70+1, 1.12.0-1~bpo70+1.ceph), fence-agents-pve:amd64 (4.0.10-1, 4.0.10-2), python-ceph:amd64 (0.80.6-1~bpo70+1, 0.80.8-1~bpo70+1), pve-cluster:amd64 (3.0-15, 3.0-16), libbind9-80:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), libpve-access-control:amd64 (3.0-15, 3.0-16), grub-pc-bin:amd64 (1.99-27+deb7u2, 2.02~bpo70+3), libisccfg82:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4), pve-firewall:amd64 (1.0-9, 1.0-18), grub-common:amd64 (1.99-27+deb7u2, 2.02~bpo70+3), grub2-common:amd64 (1.99-27+deb7u2, 2.02~bpo70+3), libisc84:amd64 (9.8.4.dfsg.P1-6+nmu2+deb7u3, 9.8.4.dfsg.P1-6+nmu2+deb7u4)

Output from pveversion -v:

proxmox-ve-2.6.32: not correctly installed (running kernel: 2.6.32-34-pve)pve-manager: not correctly installed (running version: 3.4-1/3f2d890e)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-34-pve: 2.6.32-140
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: not correctly installed
pve-cluster: not correctly installed
qemu-server: not correctly installed
pve-firmware: 1.1-3
libpve-common-perl: not correctly installed
libpve-access-control: not correctly installed
libpve-storage-perl: not correctly installed
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: not correctly installed
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

I have no issues if I update using pve-no-subscription on a other server.


Output from parted -l:

Model: LSI MR9240-8i (scsi)Disk /dev/sda: 600GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 


Number  Start   End     Size    File system  Name     Flags
 1      1049kB  2097kB  1049kB               primary  bios_grub
 2      2097kB  537MB   535MB   ext3         primary  boot, esp
 3      537MB   600GB   599GB                primary  lvm




Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-swap: 16.1GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Disk Flags: 


Number  Start  End     Size    File system     Flags
 1      0.00B  16.1GB  16.1GB  linux-swap(v1)




Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data: 463GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Disk Flags: 


Number  Start  End    Size   File system  Flags
 1      0.00B  463GB  463GB  ext3




Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-root: 103GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Disk Flags: 


Number  Start  End    Size   File system  Flags
 1      0.00B  103GB  103GB  ext3

The machines currently affected are those using the enterprise repo. The no-subscription repo works.

The problem is that our production servers now have this problem, and all servers are located at a remote location, so I need to be sure that the fix will work. I'm sure a lot of other people are wondering about the same thing.

Are you guys confident that dpkg --configure -a will fix this?

 

[wolfgang]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

]
The machines currently affected are those using the enterprise repo. The no-subscription repo works.

You should use only enterprise or no-subscription on the same machine.

]Are you guys confident that dpkg --configure -a will fix this?

yes dpkg --configure -a will fix this!

 

[Bedlore]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

May I humbly suggest that proxmox do not send out new release announcement emails without including in the email a warning not to upgrade via the GUI. Perhaps even consider including a warning features in the upgrade GUI for times such as this.

 

[jimmyjoe]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

You should use only enterprise or no-subscription on the same machine.


yes dpkg --configure -a will fix this!

Um..Are you sure? I just ran that and getting an error about initramfs-tools being broken. dpkg-reconfigure did ask about installing grub to /dev/sda (which is my correct boot drive) so I ticked it and hit OK.

# dpkg-reconfigure -a
gpg: key B98321F9: "Squeeze Stable Release Key <debian-release@lists.debian.org>" not changed
gpg: key 473041FA: "Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>" not changed
gpg: key 65FFB764: "Wheezy Stable Release Key <debian-release@lists.debian.org>" not changed
gpg: key 46925553: "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>" not changed
gpg: key 518E17E1: "Jessie Stable Release Key <debian-release@lists.debian.org>" not changed
gpg: key 2B90D010: "Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>" not changed
gpg: key C857C906: "Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>" not changed
gpg: Total number processed: 7
gpg:              unchanged: 7
Replacing config file /etc/apt/listchanges.conf with new version
Stopping deferred execution scheduler: atd.
Starting deferred execution scheduler: atd.
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz (builtins.7.gz) in auto mode
Looking for keymap to install:
NONE
unicode_stop skipped on not a tty
Looking for keymap to install:
NONE
Setting console screen modes and fonts.
setterm: cannot (un)set powersave mode: Invalid argument
Stopping periodic command scheduler: cron.
Starting periodic command scheduler: cron.
update-initramfs: deferring update (trigger activated)
Regenerating fonts cache... done.
update-initramfs: deferring update (trigger activated)
Replacing config file /etc/default/grub with new version
Installing for i386-pc platform.
Installation finished. No error reported.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-2.6.32-37-pve
Found initrd image: /boot/initrd.img-2.6.32-37-pve
Found linux image: /boot/vmlinuz-2.6.32-34-pve
Found initrd image: /boot/initrd.img-2.6.32-34-pve
Found linux image: /boot/vmlinuz-2.6.32-32-pve
Found initrd image: /boot/initrd.img-2.6.32-32-pve
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
/usr/sbin/dpkg-reconfigure: initramfs-tools is broken or not fully installed

[08:48 02/19/15]
[root@piston ~]
# pveversion 
pve-manager/3.4-1/3f2d890e (running kernel: 2.6.32-34-pve)

 

[tom]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

May I humbly suggest that proxmox do not send out new release announcement emails without including in the email a warning not to upgrade via the GUI. Perhaps even consider including a warning features in the upgrade GUI for times such as this.

you just need to follow the upgrade howto on:

http://pve.proxmox.com/wiki/Downloads#Update_a_running_Proxmox_Virtual_Environment_3.x_to_latest_3.4

we cannot write every bug in each email or announcement separately but yes, I will pass your suggestions to our devs, telling them to implement a warning about unknown bug in future releases - just tell me which bug we should warn ...

 

[JBB]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

Hi - just a suggestion: when you say "do NOT use the GUI for upgrading" it would be clearer to say "Upgrade over SSH, do NOT use your web browser for upgrading."

This is because I read the above warning, and thought I could therefore do the upgrade from the CLUI using a NoVNC terminal session from the browser-based GUI. But I ended up breaking the upgrade.

 

[Alex1]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

You should use only enterprise or no-subscription on the same machine.

I'm not using both, I tried to recreate it on a dev machine using the no-subscription repo, but this error did only occur using the enterprise repo on production servers.

dpkg --configure -a and selecting where to install GRUB did fix it.

I do agree with others that a warning should be displayed when users are using the web GUI for updating if there is a higher risk of failure when doing it that way. I was not sure if I should use the GUI or SSH, but I choose the GUI believing that would be the better way and maybe do something in addition to a simple apt-get dist-upgrade.

Now I know that the best way is to update using SSH, but a general warning would be nice for those in doubt.

 

[Bedlore]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

you just need to follow the upgrade howto on:

http://pve.proxmox.com/wiki/Downloads#Update_a_running_Proxmox_Virtual_Environment_3.x_to_latest_3.4

we cannot write every bug in each email or announcement separately but yes, I will pass your suggestions to our devs, telling them to implement a warning about unknown bug in future releases - just tell me which bug we should warn ...

I'm not been flippant and I don't consider myself an alarmist, however it struck me that including a warning about a known bug that could cause your server to be inoperable if using the provided upgraded GUI could be something worth mentioning in the email I received alerting me to the available upgrade. Otherwise you are assuming that people will check the forums before performing an upgrade. If you the devs wish to create a separate email list for admins to subscribe to for this purpose that's probably worth considering and could be beneficial in lots of other ways too.

 

[m.ardito]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

perhaps, in future, it could be possible to make gui update system aware of issues like this one (possibly because pve upgrade packages have special flags, or the like)
so that the gui knows if an update is dangerous done through itself, and does not allow it, and inform user to proceed by ssh, console, whatever.

just an idea
Marco

 

[tom]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

I'm not been flippant and I don't consider myself an alarmist, however it struck me that including a warning about a known bug that could cause your server to be inoperable if using the provided upgraded GUI could be something worth mentioning in the email I received alerting me to the available upgrade. Otherwise you are assuming that people will check the forums before performing an upgrade. If you the devs wish to create a separate email list for admins to subscribe to for this purpose that's probably worth considering and could be beneficial in lots of other ways too.

I just went through all our emails and I see that in all emails we pointed or linked to this issue. So I wonder which "email" you refer to?

 

[tom]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

perhaps, in future, it could be possible to make gui update system aware of issues like this one (possibly because pve upgrade packages have special flags, or the like)
so that the gui knows if an update is dangerous done through itself, and does not allow it, and inform user to proceed by ssh, console, whatever.

just an idea
Marco

The issue is already fixed in the current 3.4 - see top post in this thread. So there is no need for further actions.

 

[m.ardito]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

I just went through all our emails and I see that in all emails we pointed or linked to this issue. So I wonder which "email" you refer to?

I don't know what mail is Bedlore referring to, but I received one with subject: "Proxmox Newsletter, February 2015: Proxmox VE 3.4 released"
where it is not explicitly said, because it points you to download the iso (so there is not this problem), not to update your system

ie: it doesn't explicitly say: "if you are going to update a previous version, instead, don't do it through gui!"

Marco

 

[Bedlore]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

I just went through all our emails and I see that in all emails we pointed or linked to this issue. So I wonder which "email" you refer to?

Date: Thu, 19 Feb 2015 14:51:27 +0100
From: Proxmox Server Solutions GmbH <news@proxmox.com>
Subject: Proxmox Newsletter, February 2015: Proxmox VE 3.4 released

Also while not directly related it could also happen via the systems own upgrades advice email too, eg.

SUBJECT: New software packages available (myserver.com)
The following updates are available:
ceph-common: 0.80.6-1~bpo70+1 =3D=3D> 0.80.8-1~bpo70+1
etc...

 

[Bedlore]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

The issue is already fixed in the current 3.4 - see top post in this thread. So there is no need for further actions.

Nothing similar could ever recur? Its not a bad suggestion m.ardito made I thought, even adding the ability for devs to include notes or recommendations about specific upgrades.

Anyhow, I'm loving proxmox very much. Great job to all you devs.

 

[Bedlore]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

Murphy's law it seems. I tried the upgrade, naturally via ssh and things haven't gone well.

I checked all my apt sources and updated and then dist-upgraded. It said grub wasn't installed and asked where I wanted to install grub to; it offered 3 choices, from memory I think they were;
/dev/sda
/dev/sda1
/dev/dm-0

I wasn't sure so I selected all 3, however "Writing GRUB to boot device failed"

root@s1:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
6 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up pve-cluster (3.0-16) ...
Restarting pve cluster filesystem: pve-cluster[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
(warning).
invoke-rc.d: initscript pve-cluster, action "restart" failed.
dpkg: error processing pve-cluster (--configure):
subprocess installed post-installation script returned error exit status 255
dpkg: dependency problems prevent configuration of libpve-access-control:
libpve-access-control depends on pve-cluster; however:
Package pve-cluster is not configured yet.

dpkg: error processing libpve-access-control (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-firewall:
pve-firewall depends on pve-cluster; however:
Package pve-cluster is not configured yet.
pve-firewall depends on libpve-access-control; however:
Package libpve-access-control is not configured yet.

dpkg: error processing pve-firewall (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
qemu-server depends on libpve-access-control; however:
Package libpve-access-control is not configured yet.
qemu-server depends on pve-cluster; however:
Package pve-cluster is not configured yet.
qemu-server depends on pve-firewall; however:
Package pve-firewall is not configured yet.

dpkg: error processing qemu-server (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
pve-manager depends on qemu-server (>= 1.1-1); however:
Package qemu-server is not configured yet.
pve-manager depends on pve-cluster (>= 1.0-29); however:
Package pve-cluster is not configured yet.
pve-manager depends on libpve-access-control (>= 3.0-2); however:
Package libpve-access-control is not configured yet.
pve-manager depends on pve-firewall; however:
Package pve-firewall is not configured yet.

dpkg: error processing pve-manager (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve-2.6.32:
proxmox-ve-2.6.32 depends on pve-manager; however:
Package pve-manager is not configured yet.
proxmox-ve-2.6.32 depends on qemu-server; however:
Package qemu-server is not configured yet.

dpkg: error processing proxmox-ve-2.6.32 (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
pve-cluster
libpve-access-control
pve-firewall
qemu-server
pve-manager
proxmox-ve-2.6.32
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried a dpkg-reconfigure with no joy.

root@s1:~# dpkg --configure -a
Setting up pve-cluster (3.0-16) ...
Restarting pve cluster filesystem: pve-cluster[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
(warning).
invoke-rc.d: initscript pve-cluster, action "restart" failed.
dpkg: error processing pve-cluster (--configure):
subprocess installed post-installation script returned error exit status 255
dpkg: dependency problems prevent configuration of pve-manager:
pve-manager depends on pve-cluster (>= 1.0-29); however:
Package pve-cluster is not configured yet.

dpkg: error processing pve-manager (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-firewall:
pve-firewall depends on pve-cluster; however:
Package pve-cluster is not configured yet.

dpkg: error processing pve-firewall (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve-2.6.32:
proxmox-ve-2.6.32 depends on pve-manager; however:
Package pve-manager is not configured yet.

dpkg: error processing proxmox-ve-2.6.32 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-access-control:
libpve-access-control depends on pve-cluster; however:
Package pve-cluster is not configured yet.

dpkg: error processing libpve-access-control (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
qemu-server depends on libpve-access-control; however:
Package libpve-access-control is not configured yet.
qemu-server depends on pve-cluster; however:
Package pve-cluster is not configured yet.
qemu-server depends on pve-firewall; however:
Package pve-firewall is not configured yet.

dpkg: error processing qemu-server (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
pve-cluster
pve-manager
pve-firewall
proxmox-ve-2.6.32
libpve-access-control
qemu-server
root@s1:~#

This machine is stand alone and I can't do a reboot without been sure it will come up again, its remote and I have client's VPS running.

Perhaps I should of waited longer for these bugs to be found and sorted, in any case I really need help now please.

## update ##
I found I had a running pmxcfs process even though its a stand alone machine. The dist-upgrade completed, however I still don't want to reboot until I can verify the grub install. What's the best way to do that?

Thanks

 

[dietmar]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

## update ##
I found I had a running pmxcfs process even though its a stand alone machine. The dist-upgrade completed, however I still don't want to reboot until I can verify the grub install. What's the best way to do that?

You always need pmxcfs, even on non-clustered systems. Also, you need to install grub on /dev/sda

# grub-install /dev/sda

 

[franca]

Re: grub2 update problems (GUI only), leading to not correctly installed packages

I updated thorugh the CLI but stil had the Grub not completing the installation issue.

I could boot the Server with this recovery disk: http://www.supergrubdisk.org/super-grub2-disk/

Burn it, Boot with it and you will be able to see your original Grub menu, select ProxMox and wait for the server to go up.

Once logged to the server:

grub-install /dev/pathtoyourharddiskboot
(in my case /dev/cciss/c0d0 <-- HP Proliant Server)
update-grub
update-grub2
(just in case)