Node locks up after upgrade

[r4pt0x]

I updated one of our nodes today (subscription repository). The node was running kernel version 2.6.32-23-pve before, update ended without any errors or warnings.

At first reboot the system locked up after starting the first container.
After disabling any Container or VM via single-user mode and editing the config files, the system hangs after the message:

"startpar: service(s) returned failure: apache2 ... failed!

With apache2 disabled the host starts up, i can get to management console and ssh, but as soon as i start up any VM or container, the system locks up again...

The system then sits there for a while, then starts to drop random tasks with "blocked for more than 120seconds" messages.
I can still type but the system doesn't respond - ping goes through, but sshd is not running, so i can't get into the system.

no response to any keyboard input to reboot it, so i have to colt-reset the machine...


no errors (except the "blocked for more...") in syslog or kern.log


I'm pretty down with ideas where to start - disks are clean, RAM is OK (according to fsck and memtest)...

 

[r4pt0x]

After reverting back these packages the system is working again:

pve-kernel-2.6.32-23-pve:amd64 2.6.32-109
pve-manager:amd64 3.1-3
qemu-server:amd64 3.1-1
proxmox-ve-2.6.32:all 3.1-109
pve-libspice-server1:amd64 0.12.4-1

I've take a snapshot of the system and will try to figure out which package caused the problems and why. It seems the system wasn't able to get up some virtual network interfaces before locking up.

current package versions of the system:

# pveversion -v
proxmox-ve-2.6.32: 3.1-109 (running kernel: 2.6.32-23-pve)
pve-manager: 3.1-3 (running version: 3.1-3/dc0e9b0e)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-25-pve: 2.6.32-113
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-17-pve: 2.6.32-83
pve-kernel-2.6.32-14-pve: 2.6.32-74
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-18-pve: 2.6.32-88
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-7
qemu-server: 3.1-1
pve-firmware: 1.0-23
libpve-common-perl: 3.0-6
libpve-access-control: 3.0-6
libpve-storage-perl: 3.0-13
pve-libspice-server1: 0.12.4-1
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.0-2