veird problem and kernel panic after uping to 1.9

RRJ

Member
Apr 14, 2010
245
0
16
Estonia, Tallinn
weird problem and kernel panic after uping to 1.9

Hello dear proxmox team!

Usually i get answers for my questions here on the free community support forum and i greatly appreciate it. Thank you for your project and work.

I run 2 proxmox servers in production. one of them was updated to 1.9 without any problems. i run there openvz containers and kvm linux guests. (was small issue
on this server with hpacucli and new hpsa driver but got resolved very fast)
on the other server there is something very weird going on.

1. after cold reboot i run into continuously repeating grub error: Loading stage 1.5, after i press ctrl+alt+delete it boots fine and runs fsck.
What could be reason of this? i did run grubinstall /dev/sda1 (thats the /boot mount)
to be completely honest i should say, that i think, that this error comes out of the second error (it explains why i have to make a cold restart to the server via intel remote management console)
2. after upgrading to 1.9 and 2.6.32-6-pve kernel, this server started to crash with kernel panic (@ random time after reboot), so i had to boot with 2.6.32-4-pve kernel and it seems it runs well atm.
in the kernel panic messages i remember only something about net_if send blablabla... (i know i had take a pic of this error, but as it is production server, i had not enough time to think about this)
on this server there are openvz containers and (i suspect this one to be the root of the problem) one kvm guest with ms windows server 2000.
If You guys need those kernel panic rows i will reproduce this crash tonight (can't do this @ worktime) and attach it here.

versions:
Code:
virt:~# pveversion  -v
pve-manager: 1.9-24 (pve-manager/1.9/6542)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.9-43
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-6-pve: 2.6.32-43
qemu-server: 1.1-32
pve-firmware: 1.0-13
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.28-1pve5
vzdump: 1.2-15
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6
Code:
virt:~# uname -a
Linux virt 2.6.32-4-pve #1 SMP Mon May 9 12:59:57 CEST 2011 x86_64 GNU/Linux
 
Last edited:
any details about your hardware?
 
Re: weird problem and kernel panic after uping to 1.9

sure.
Code:
Network: igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) / Intel 82575EB Ethernet
Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
CPU:  2x Version: Intel(R) Xeon(R) CPU           E5620 (total of 16 cores)
3ware 9000 Storage Controller
RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
PCI bridge: Intel Corporation 5520/5500/X58
USB Controller: Intel Corporation 82801JI (ICH10 Family)
PCI bridge: Intel Corporation 82801JI (ICH10 Family)
ISA bridge: Intel Corporation 82801JIR (ICH10R)
VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)
Code:
virt:~# free
             total       used       free     shared    buffers     cached
Mem:      12221864    1875568   10346296          0     246868     462648
-/+ buffers/cache:    1166052   11055812
Swap:     11534328          0   11534328
Code:
virt:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root  136G  1.3G  128G   1% /
tmpfs                 5.9G     0  5.9G   0% /lib/init/rw
udev                   10M  620K  9.4M   7% /dev
tmpfs                 5.9G     0  5.9G   0% /dev/shm
/dev/sda1             504M   49M  431M  11% /boot
/dev/sdb1             1.8T   49G  1.7T   3% /var/lib/vz

Code:
virt:~# /usr/sbin/tw_cli /c0 show


Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       149.001   RiW    ON
u1    RAID-1    OK             -       -       -       1862.63   RiW    ON


VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   149.05 GB SATA  0   -            ST3160813AS
p1    OK             u0   149.05 GB SATA  1   -            ST3160813AS
p2    OK             u1   1.82 TB   SATA  2   -            WDC WD2001FASS-00W2
p3    OK             u1   1.82 TB   SATA  3   -            WDC WD2001FASS-00W2


Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx
 
Last edited:
Re: weird problem and kernel panic after uping to 1.9

mainboard? do you have ASPM enabled in your bios? disable it.
 
Re: weird problem and kernel panic after uping to 1.9

Code:
Base Board Information
        Manufacturer: Intel Corporation
        Product Name: S5520UR
        Version: E22554-751
        Serial Number: BZUB04203592

not sure about ASPM, will check it later. thanx for an advice.
 
Re: weird problem and kernel panic after uping to 1.9

forgot to mention, that win2k server uses virtio drivers for network adpater and Cirrus Logic 5446 Compatible Grahphic adapter for VGA
 
Re: weird problem and kernel panic after uping to 1.9

w2k with virtio net? are you sure?
 
Re: weird problem and kernel panic after uping to 1.9

I confirm Win2k can run virtio net without any problem. I have a production server up smoothly since 2 years :-D
 
Re: weird problem and kernel panic after uping to 1.9

yes, i'm :) there are problems with virtio for hdd, but network drivers for XP suit well. mby there is some change in kvm in 1.9 that runs in conflict with them.
 
Re: weird problem and kernel panic after uping to 1.9

so should i reproduce the kernel panic or it will be enough of turning ASPM off in bios?
 
Re: weird problem and kernel panic after uping to 1.9

I confirm Win2k can run virtio net without any problem. I have a production server up smoothly since 2 years :-D

what about proxmox 1.9 ? runs well?
 
Re: weird problem and kernel panic after uping to 1.9

no idea if ASPM is enabled in your bios, so I suggest you find it out.
 
Re: weird problem and kernel panic after uping to 1.9

no idea if ASPM is enabled in your bios, so I suggest you find it out.

just restarted the server and couldnt find anything like ASPM in bios. so i rebooted in the lates kernel and i'll make a pic of kernel panic error.
 
Re: weird problem and kernel panic after uping to 1.9

well, now it seems to be running ok, but i've lost network on this win2k kvm virtual client. cant get it back not with virtio nor other emulated nic. i will fight it a bit more. seems like there are some changes in kvm in new kernel.
 
Re: weird problem and kernel panic after uping to 1.9

After the upgrade to 1.9, it took me a number of hours to resolve the lack of networking on two win2k servers and one winxp SP3. The resolution was updating the computer driver via the Windows device manager from "Standard PC" to "ACPI Uniprocessor PC". WinXP required a special procedure to update the driver though because the ACPI drivers are hidden. The networking came back. I'm using virtio Ethernet devices as well. Backup VM's before doing anything.
 
Re: weird problem and kernel panic after uping to 1.9

well.
after reinstalling win2k server kvm client it seems to be running ok. so i thing it was networki drivers + acpi uniprocessor pc problem.
i dont even want to imagine, what will do those guys whos running some windows services in kvm :) (domains, exchange etc). seems like kvm team did some sweet changes.
 
Re: weird problem and kernel panic after uping to 1.9

Yes, fresh w2k and winXP installations automatically pick up the ACPI CPU PC driver. I purposely switched our w2k production systems to the standard PC driver a while ago to resolve high CPU usage at the time of the pve 1.7 if I'm not mistaken. It looks like the pve 1.9 does not support the standard PC driver. Anyway, we now know what the solution is.
 
Re: weird problem and kernel panic after uping to 1.9

well, third proxmox upgrade to 1.9 (i forgot about that) and the same kernel panic. now i've got a pic of the error via ilo. this machine runs only openvz containers (4 of them)
i think its a kernel/module bug.
any ideas? this is third semi-production server...

panic.PNG
 
Re: weird problem and kernel panic after uping to 1.9

did you use latest kernel from pvetest repository? if not, try the new one.
(pve-kernel-2.6.32-6-pve: 2.6.32-46)
 
Re: weird problem and kernel panic after uping to 1.9

and it done it again after about 30 minutes after my post :S truly sad. running old kernel now.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!