can not start vz containers after upgrade

[cschnee]

Hello,
I have 10 nodes in 2 clusters, the larger group of them has been running "aptitude update; aptitude upgrade" yesterday and I received a few pve related updates on all those machines.

Since then I can't start or reboot any openvz containers, the only hint I get is that when starting them from cmd-line with --verbose I get:

vzctl --verbose start 107
Starting container ...
Container is unmounted
Container is mounted
Adding IP address(es): xxx.xxx.xxx.9
vps-net_add WARNING: Device list is empty
vps-net_add WARNING: Device list is empty
Running container script: /etc/vz/dists/scripts/debian-add_ip.sh
Setting CPU units: 1000
Setting CPUs: 1
Configuring meminfo: 262144
Set hostname: nagios.tisnet.ch
Running container script: /etc/vz/dists/scripts/debian-set_hostname.sh
Running container script: /etc/vz/dists/scripts/set_dns.sh
File resolv.conf was modified
Setting quota ugidlimit: 0
Running container script: /etc/vz/dists/scripts/set_ugid_quota.sh
Container start in progress...

The "Device list empty" above triggered my interest and I found that the containers are reachable from the node itself but not from network. So I added manually the hardware address of the node with arp -s on another node and then I could ping the container from the node where I added the mac manually.

I verified that on the cluster that hasn't updated yet everything (create, restart, migrate) works perfect.

I did try with 2.6.18-2, 2.16.18-3 and 2.6.24-10 : no difference.

Does anybody else experience similar problems ?


EDIT: If I use /usr/lib/vzctl/scripts/vps-functions from the not-yet-upgraded install I get a few errors when starting the container but things start working again, this file seems borked in the update that is currently available.

 

[tom]

I assume you working with the test repository (pvetest)?

if yes, downgrade the vzctl package.

wget ftp://download.proxmox.com/debian/dists/lenny/pve/binary-amd64/vzctl_3.0.23-1pve8_amd64.deb

and

dpkg -i vzctl_3.0.23-1pve8_amd64.deb

 

[cschnee]

tom,

Thank you! It seems that did the trick, after downgrading vzctl I was able to restart containers. Didn't realize I was using the test repo, but you are completely right.

I am unsure about why I added the test repo to all nodes, but I guess now I can't just remove the line in sources.lst and hope things will be fine then, right ?
Is there any chance for me to revert to stable without reinstalling the nodes ?

Cheers,
Christoph

 

[tom]

change pvetest to pve in the sources list.

and post the output of 'pveversion -v' so we can see which packages needs to be downgraded (like vzctl).

 

[cschnee]

I have commented the pvetest repo in sources.lst already. After downgrading vzctl I've got the following pveversion -v output:

pve-manager: 1.5-9 (pve-manager/1.5/4728)
running kernel: 2.6.24-10-pve
proxmox-ve-2.6.18: 1.5-6
pve-kernel-2.6.18-3-pve: 2.6.18-6
pve-kernel-2.6.24-10-pve: 2.6.24-21
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-14
pve-firmware: 1.0-4
libpve-storage-perl: 1.0-11
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-6

Cheers,
Christoph

 

[tom]

there is still one major issue.

you are running a 2.6.24 kernel (running kernel: 2.6.24-10-pve) but you installed the pve-qemu-kvm package from the 2.6.18 branch.#
see also http://pve.proxmox.com/wiki/Proxmox_VE_Kernel

if you want to run the 2.6.24 kernel branch, run:

apt-get install proxmox-ve-2.6.24

 

[cschnee]

Thanks, this is only from my testing while in trouble with vzctl and I had no chance to reboot the node since then.
While I was unable to start new containers I sorta panicked and installed the 2.6.24 kernel to see if it helps - at the same time I found the problem with vzctl. Actually this node should run 2.6.18-x because it has only openvz containers.
Therefor I guess the route for me should be to reboot back into 2.6.18 and then I would "aptitude purge pve-kernel-2.6.24-10-pve" ?

I was in a brave mood this morning and tried to "aptitude update; aptitude upgrade" on the mentioned node to see if any packages would get downgraded - however none was proposed to downgrade. Am I right supposing that once the stable repo gets updates that have a higher release than the testing I have installed right now I will be switched back to stable more or less automatically ?

I am still wondering why I decided to add the pvetest repo to this productions nodes - but well... ;-P
Thanks for helping me out of this unfortunate situation.

 

[tom]

if you want the stable 2.6.18 branch you need to downgrade the following packages (the same procedure as with vzctl):

ftp://download.proxmox.com/debian/d...64/pve-kernel-2.6.18-2-pve_2.6.18-5_amd64.deb

and

ftp://download.proxmox.com/debian/dists/lenny/pve/binary-amd64/pve-qemu-kvm-2.6.18_0.9.1-5_amd64.deb

 

[cschnee]

Thanks again, I was able to downgrade all 2.6.18 nodes so far, I was also able to remove 2.6.18-3 on all nodes that received it (the pure vz container nodes). On the 2.6.24 nodes replacing vzctl was enough as it seems to get back on track (stable branch).

I highly appreciated your kind and quick help!



Cheers,
Christoph