HUGE problem - Update that failed - NOTHING works

[TinusBlaauw]

Hi!

Thank you for taking time to read this.

I have commenced an upgrade on the Proxmox 3.4 yesterday evening, to be met by a lot of errors.

I rebooted the host, and now nothing works. I can get to the CLI via SSH; but no webgui.

No vzdump is found (in order to backup my vm's); the script for the subscription key is also not working.

How do I backup my VM's?

Is there any way to repair this WITHOUT loosing my VM's? I did do a manual backup, but only two went through.

On Host:

NAME                  USED  AVAIL  REFER  MOUNTPOINT
rpool                 273G   184G    96K  /rpool
rpool/ROOT           19.5G   184G    96K  /rpool/ROOT
rpool/ROOT/pve-1     19.5G   184G  19.5G  /
rpool/swap           15.9G   200G    88K  -
rpool/vm-100-disk-1  10.3G   193G  1.68G  -
rpool/vm-101-disk-1  82.5G   258G  8.57G  -
rpool/vm-102-disk-1  82.5G   215G  51.6G  -
rpool/vm-104-disk-1  20.6G   203G  1.39G  -
rpool/vm-106-disk-1  20.6G   202G  2.32G  -
rpool/vm-107-disk-1  20.6G   204G   619M  -

On Backup destination:

-rw-r--r--  1 root  wheel        3760 Jun 24 15:14 vzdump-qemu-100-2015_06_24-15_13_27.log
-rw-r--r--  1 root  wheel  1047505373 Jun 24 15:14 vzdump-qemu-100-2015_06_24-15_13_27.vma.lzo
-rw-r--r--  1 root  wheel       10085 Jun 24 15:45 vzdump-qemu-101-2015_06_24-15_16_05.log
-rw-r--r--  1 root  wheel  5009008213 Jun 24 15:45 vzdump-qemu-101-2015_06_24-15_16_05.vma.gz

Any/ALL help appreciated!

Tinus

 

[tom]

please post the output of:

> pveversion -v

if you miss pacakges, try a re-installation by running:

> apt-get update
> apt-get install proxmox-ve-2.6.32

if you have any error, please post it here.

 

[TinusBlaauw]

please post the output of:

> pveversion -v

if you miss pacakges, try a re-installation by running:

> apt-get update
> apt-get install proxmox-ve-2.6.32

if you have any error, please post it here.

Hi!

pveversion:

 pveversion -v
-bash: pveversion: command not found

apt-get update:

root@vm20:/etc/pve/openvz# apt-get update
Hit http://security.debian.org wheezy/updates Release.gpg
Hit http://ftp.debian.org wheezy Release.gpg
Hit http://http.us.debian.org testing Release.gpg
Hit http://security.debian.org wheezy/updates Release
Hit http://ftp.debian.org wheezy Release
Hit http://http.debian.net wheezy Release.gpg
Hit http://http.us.debian.org testing Release
Hit http://http.debian.net wheezy Release
Hit http://ftp.debian.org wheezy/main amd64 Packages
Hit http://http.us.debian.org testing/main amd64 Packages/DiffIndex
Hit http://ftp.debian.org wheezy/contrib amd64 Packages
Hit http://ftp.debian.org wheezy/contrib Translation-en
Hit http://ftp.debian.org wheezy/main Translation-en
Hit http://security.debian.org wheezy/updates/main amd64 Packages
Ign https://enterprise.proxmox.com wheezy Release.gpg
Hit http://http.debian.net wheezy/main amd64 Packages
Hit http://security.debian.org wheezy/updates/contrib amd64 Packages
Hit http://http.debian.net wheezy/contrib amd64 Packages
Hit http://http.us.debian.org testing/contrib amd64 Packages/DiffIndex
Hit http://security.debian.org wheezy/updates/contrib Translation-en
Hit http://http.debian.net wheezy/non-free amd64 Packages
Hit http://security.debian.org wheezy/updates/main Translation-en
Hit http://http.debian.net wheezy/contrib Translation-en
Ign https://enterprise.proxmox.com wheezy Release
Hit http://http.debian.net wheezy/main Translation-en
Hit http://http.debian.net wheezy/non-free Translation-en
Hit http://http.us.debian.org testing/non-free amd64 Packages/DiffIndex
Hit http://http.us.debian.org testing/contrib Translation-en/DiffIndex
Hit http://http.us.debian.org testing/main Translation-en/DiffIndex
Hit http://http.us.debian.org testing/non-free Translation-en/DiffIndex
Err https://enterprise.proxmox.com wheezy/pve-enterprise amd64 Packages
  The requested URL returned error: 401
Ign https://enterprise.proxmox.com wheezy/pve-enterprise Translation-en_US
Ign https://enterprise.proxmox.com wheezy/pve-enterprise Translation-en
W: Failed to fetch https://enterprise.proxmox.com/debian/dists/wheezy/pve-enterprise/binary-amd64/Packages  The requested URL returned error: 401


E: Some index files failed to download. They have been ignored, or old ones used instead.

apt-get install proxmox-ve-2.6.32:

root@vm20:/etc/pve/openvz# apt-get install proxmox-ve-2.6.32
Reading package lists... Done
Building dependency tree
Reading state information... Done
W: Ignoring Provides line with DepCompareOp for package php-psr-log-implementation
W: Ignoring Provides line with DepCompareOp for package php-math-biginteger
W: You may want to run apt-get update to correct these problems
E: Unable to locate package proxmox-ve-2.6.32
E: Couldn't find any package by regex 'proxmox-ve-2.6.32'

 

[tom]

pls add temporary the following repo to your /etc/apt/sources.list:

deb [URL]http://download.proxmox.com/debian[/URL] wheezy pve-no-subscription

and run again:

> apt-get update
> apt-get install proxmox-ve-2.6.32

 

[TinusBlaauw]

Hi Tom;

apt-get update

root@vm20:/etc/pve/openvz# nano /etc/apt/sources.list
root@vm20:/etc/pve/openvz# apt-get update
Hit http://ftp.debian.org wheezy Release.gpg
Hit http://security.debian.org wheezy/updates Release.gpg
Get:1 http://download.proxmox.com wheezy Release.gpg [198 B]
Get:2 http://http.us.debian.org testing Release.gpg [1,554 B]
Hit http://ftp.debian.org wheezy Release
Hit http://security.debian.org wheezy/updates Release
Get:3 http://download.proxmox.com wheezy Release [4,047 B]
Get:4 http://http.us.debian.org testing Release [115 kB]
Hit http://ftp.debian.org wheezy/main amd64 Packages
Hit http://security.debian.org wheezy/updates/main amd64 Packages
Get:5 http://download.proxmox.com wheezy/pve-no-subscription amd64 Packages [122 kB]
Ign https://enterprise.proxmox.com wheezy Release.gpg
Hit http://http.debian.net wheezy Release.gpg
Hit http://ftp.debian.org wheezy/contrib amd64 Packages
Hit http://security.debian.org wheezy/updates/contrib amd64 Packages
Hit http://ftp.debian.org wheezy/contrib Translation-en
Hit http://security.debian.org wheezy/updates/contrib Translation-en
Hit http://http.debian.net wheezy Release
Hit http://ftp.debian.org wheezy/main Translation-en
Hit http://security.debian.org wheezy/updates/main Translation-en
Hit http://http.debian.net wheezy/main amd64 Packages
Ign https://enterprise.proxmox.com wheezy Release
Hit http://http.debian.net wheezy/contrib amd64 Packages
Hit http://http.debian.net wheezy/non-free amd64 Packages
Hit http://http.us.debian.org testing/main amd64 Packages/DiffIndex
Hit http://http.debian.net wheezy/contrib Translation-en
Hit http://http.debian.net wheezy/main Translation-en
Hit http://http.debian.net wheezy/non-free Translation-en
Hit http://http.us.debian.org testing/contrib amd64 Packages/DiffIndex
Hit http://http.us.debian.org testing/non-free amd64 Packages/DiffIndex
Hit http://http.us.debian.org testing/contrib Translation-en/DiffIndex
Hit http://http.us.debian.org testing/main Translation-en/DiffIndex
Hit http://http.us.debian.org testing/non-free Translation-en/DiffIndex
Ign http://download.proxmox.com wheezy/pve-no-subscription Translation-en_US
Ign http://download.proxmox.com wheezy/pve-no-subscription Translation-en
Err https://enterprise.proxmox.com wheezy/pve-enterprise amd64 Packages
  The requested URL returned error: 401
Ign https://enterprise.proxmox.com wheezy/pve-enterprise Translation-en_US
Ign https://enterprise.proxmox.com wheezy/pve-enterprise Translation-en
Fetched 243 kB in 16s (15.0 kB/s)
W: Failed to fetch https://enterprise.proxmox.com/debian/dists/wheezy/pve-enterprise/binary-amd64/Packages  The requested URL returned error: 401


E: Some index files failed to download. They have been ignored, or old ones used instead.

apt-get install proxmox-ve-2.6.32

root@vm20:/etc/pve/openvz# apt-get install proxmox-ve-2.6.32
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 clvm : Depends: libcman3 (>= 3.1.8) but it is not going to be installed
        Depends: libconfdb4 (>= 1.4.6) but it is not going to be installed
        Depends: libcpg4 (>= 1.4.6) but it is not going to be installed
        Depends: libdlm3 (>= 3.1.8) but it is not going to be installed
        Depends: libquorum4 (>= 1.4.6) but it is not going to be installed
        Depends: libsalck3 (>= 1.1.4) but it is not going to be installed
 libapt-pkg-perl : Depends: perlapi-5.14.2
 libauthen-pam-perl : Depends: perlapi-5.14.2
 libcrypt-openssl-bignum-perl : Depends: perlapi-5.14.2
 libcrypt-openssl-random-perl : Depends: perlapi-5.14.2
 libcrypt-openssl-rsa-perl : Depends: perlapi-5.14.2
 libcrypt-ssleay-perl : Depends: perlapi-5.14.2
 libev-perl : Depends: perlapi-5.14.2
 libfile-sync-perl : Depends: perlapi-5.14.2
 libfilesys-df-perl : Depends: perlapi-5.14.2
 libhtml-parser-perl : Depends: perlapi-5.14.2
 libjson-xs-perl : Depends: perlapi-5.14.2
 liblinux-inotify2-perl : Depends: perlapi-5.14.2
 liblocale-gettext-perl : PreDepends: perlapi-5.14.2
 libnet-dns-perl : Depends: perlapi-5.14.2
 libnet-ssleay-perl : Depends: perlapi-5.14.2
 librrds-perl : Depends: perlapi-5.14.2
 libterm-readline-gnu-perl : Depends: perlapi-5.14.2
 libtext-charwidth-perl : Depends: perlapi-5.14.2
 libtext-iconv-perl : Depends: perlapi-5.14.2
 libuuid-perl : Depends: perlapi-5.14.2
 libxml-parser-perl : Depends: perlapi-5.14.2
 lsof : Depends: libperl4-corelibs-perl or
                 perl (< 5.12.3-7) but 5.20.2-6 is to be installed
 proxmox-ve-2.6.32 : Depends: pve-firmware but it is not going to be installed
                     Depends: pve-manager but it is not going to be installed
                     Depends: qemu-server but it is not going to be installed
 pve-cluster : Depends: perlapi-5.14.2
 redhat-cluster-pve : Depends: perlapi-5.14.2
W: Ignoring Provides line with DepCompareOp for package php-psr-log-implementation
W: Ignoring Provides line with DepCompareOp for package php-math-biginteger
W: You may want to run apt-get update to correct these problems
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
root@vm20:/etc/pve/openvz#

 

[TinusBlaauw]

If all else fails - is there perhaps a way I can access the KVM's and scp/ftp/sftp the files to a seperate storage in order to wipe the machine and restore? I have a ZFS-mirrored setup....

 

[tom]

seems you have Debian Testing in your sources.list, this break of course your whole system! you can only add Debian Wheezy.

> http://http.us.debian.org testing

adding this was a bad idea. check your /etc/apt/sources.list
but downgrading is now very hard, I suggest a new installation.

copy your virtual disks - e.g. to another server, also all your VMID.conf files.
if you have no access to /etc/pve/, check http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)#Recovery

 

[TinusBlaauw]

Hi

OK the first part is done - I can see the .conf files. However, where do I find the virtual disks? I can 'see' the files using the ZFS commands, but I can't physically get access to them to copy...

Tinus

 

[tom]

Hi

OK the first part is done - I can see the .conf files. However, where do I find the virtual disks? I can 'see' the files using the ZFS commands, but I can't physically get access to them to copy...

Tinus

where are your virtual disk stored? as you configured it, you should know it.

if you store them using the ZFS plugin (introduced in 3.4), you will see on the CLI by using the standard ZFS tools. and for example, you can also copy them to another ZFS server by using ZFS send.