Kernel crash on boot with 2.6.32-10-pve host kernel

[bert64]

After upgrading to the 2.6.32-10 kernel, certain guest images now crash with the following errors:

---
init-early.sh[845]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003961c08f5b8 error 5
grsec: Segmentation fault occurred at ffffffffff600400 in /lib64/rc/sh/init-early.sh[init-early.sh:845] uid/euid:0/0 gid/egid:0/0, parent /sbin/rc[rc:844] uid/euid:0/0 gid/egid:0/0
grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /lib64/rc/sh/init-early.sh[init-early.sh:845] uid/euid:0/0 gid/egid:0/0, parent /sbin/rc[rc:844] uid/euid:0/0 gid/egid:0/0


OpenRC 0.9.4 is starting up Gentoo Linux (x86_64)


Press I to enter interactive boot mode


init.sh[846]: segfault at ffffffffff600400 ip ffffffffff600400 sp 00000386669ad038 error 5
grsec: Segmentation fault occurred at ffffffffff600400 in /lib64/rc/sh/init.sh[init.sh:846] uid/euid:0/0 gid/egid:0/0, parent /sbin/rc[rc:844] uid/euid:0/0 gid/egid:0/0
grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /lib64/rc/sh/init.sh[init.sh:846] uid/euid:0/0 gid/egid:0/0, parent /sbin/rc[rc:844] uid/euid:0/0 gid/egid:0/0
* Caching service dependencies ...
sh[847]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003910172c588 error 5
grsec: Segmentation fault occurred at ffffffffff600400 in /bin/bash[sh:847] uid/euid:0/0 gid/egid:0/0, parent /sbin/rc[rc:844] uid/euid:0/0 gid/egid:0/0
grsec: more alerts, logging disabled for 10 seconds
fopen `/lib64/rc/init.d/deptree': Read-only file system
* Failed to update the dependency tree
[ !! ]
rc[844]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003fd84704798 error 5
init[1]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003d4d569d9a8 error 5
init[1]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003d4d569cf98 error 5
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 1 seconds..
---

Downgrading to the 2.6.32-6 kernel i was using with 1.9 works correctly with no problems.
The kernel version in question is:
ii pve-kernel-2.6.32-10-pve 2.6.32-63 The Proxmox PVE Kernel Image

Hardware is an HP DL140 G3, using the following cpu:
Intel(R) Xeon(R) CPU E5310 @ 1.60GHz

I assume its some kind of compatibility issue with the hardware, as these same kernel versions work fine on other hardware...

 

[tom]

just to make sure - you talk about KVM guest kernel issues?

if yes, give more details about your guests, including OS and VMID conf files so that we can try to test this here.

 

[bert64]

Yes, the guest OS crashes on boot if the host is running 2.6.32-10-pve, but the exact same guest boots fine if the host is running 2.6.32-6-pve

Example conf file:
---
name: ns1
net0: virtio=00:80:10A:98:A9,bridge=vmbr1
ostype: l26
memory: 256
onboot: 1
sockets: 1
boot: c
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
bootdisk: virtio0
virtio0: ISCSI-LVM:vm-114-disk-1,cache=writethrough
virtio1: ISCSI-LVM:vm-114-disk-2,cache=unsafe
args: -serial unix:/var/run/qemu-server/114.serial,server,nowait -cpu host
---

Guest OS is gentoo linux using the hardened kernel and hardened userland profile.

The kernel image running on the guests is available from:
http://www.ev6.net/kernel/bzImage

This kernel causes a crash on the HP DL140 G3 as soon as it tries booting the userland (even init causes a segfault)

 

[ccube]

try using default qemu64 cpu. gentoo doesnt like host cpu passthrough!

instead of this you could help me investigating which kernel setting will boot the gentoos well

 

[bert64]

Default cpu has the same problem...

-cpu host also works just fine when running 2.6.32-6-pve kernel on the host system

 

[ccube]

send me a mail and youll get my gentoo kernel config. this forum does not allow me to send it to you here

 

[bert64]

My kernel config used in the guests is available from http://www.ev6.net/kernel/config, this applies to the gentoo hardened-sources 3.3.0

On my DL140 G3, this works perfectly under kvm when the host is running 2.6.32-6-pve, but crashes when the host runs 2.6.32-10-pve, and i'm at a loss as to why...

 

[bert64]

Even specifying init=/bin/bash crashes:

VFS: Mounted root (ext4 filesystem) readonly on device 254:1.
Freeing unused kernel memory: 412k freed
bash[1]: segfault at ffffffffff600400 ip ffffffffff600400 sp 000003b7a4812958 error 5
grsec: Segmentation fault occurred at ffffffffff600400 in /bin/bash[bash:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper/0:0] uid/euid:0/0 gid/egid:0/0
grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /bin/bash[bash:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper/0:0] uid/euid:0/0 gid/egid:0/0
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 1 seconds..

only logs displayed on the host are:

kvm: 2001: cpu0 unhandled rdmsr: 0x345
kvm: 2001: cpu0 unhandled rdmsr: 0x38d
kvm: 1956: cpu0 unhandled rdmsr: 0x345
kvm: 1956: cpu0 unhandled rdmsr: 0x38d


Has anyone else been able to reproduce this?

All i've been able to find, were a few posts on lkml:

http://lkml.indiana.edu/hypermail/linux/kernel/1202.2/00302.html

 

[ccube]

any news on this?

my gentoo boxes are still crashing. (gentoo-sources-3.5)

fully upgraded pve2