Kernel Panic when disconnecting USB hardware

[hydrosIII]

Hi. i was working with proxmox VE. when i got a sudden kernel panic, i had to hard reboot the server and attached a USB keyboard and a scrren to it to look for the logs, the logs had no trace of the first kernel panic . One hour later everyhting was working when i decided to disconnect the keyboard, so i got another kernel panic, i repeated this process just to confirm, disconnecting USB keyboard gave panic again. Couldnt catch the panic in syslog. Maybe it is a hardware issue. pversion -v : proxmox-ve-2.6.32: 3.2-124 (running kernel: 2.6.32-28-pve) pve-manager: 3.2-3 (running version: 3.2-3/38bfef76) pve-kernel-2.6.32-27-pve: 2.6.32-121 pve-kernel-2.6.32-28-pve: 2.6.32-124 pve-kernel-2.6.32-26-pve: 2.6.32-114 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.5-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.5-1 pve-cluster: 3.0-12 qemu-server: 3.1-16 pve-firmware: 1.1-2 libpve-common-perl: 3.0-14 libpve-access-control: 3.0-11 libpve-storage-perl: 3.0-19 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-6 vzctl: 4.0-1pve5 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 1.7-7 ksm-control-daemon: 1.1-1 glusterfs-client: 3.4.2-1

 

[snowman66]

Re: Kernel Panic when disconnecting keyboard

What happens if you connect keyboard to external USB HUB with its own power supply?
Is there any UPS software installed on pve host?
Can you tell us which motherboard is in use?

look for kernel messages in /var/log/syslog

something like this:
Apr 19 13:14:16 pve1 kernel: usb 1-1.4.3: usbfs: USBDEVFS_CONTROL failed cmd java rqt 33 rq 9 len 8 ret -110

http://en.it-usenet.org/thread/15303/1297/

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

I don't have a external USB hub with own power supply to try on, sorry.I have the apcupsd controlling the UPS. These panics began to occur after a long powerfailure made the server shutdown automatically.

The motherboard in use is FOXCONN model A6VMX. I did a firmware upgrade before the production use of the server started. So i think it cannot be a firmware bug.

I cannot do to much tests in the server since this weekend is in full production, and it has to be on forfecully during the day. Im having a hard time with these panics. At night i can do some tests.
There are some kernel messages related to USB in the syslog, but it is just detecting them, no errors visible.

I couldnt catch the panics in the logs but it tooka picture of the monitor when the oops:


Thank you.

 

[udo]

Re: Kernel Panic when disconnecting keyboard

Hi,
I can confirm this bug on my old test-laptop nx6325 - It's enough to disconnect an USB-mouse to get an kernel panic.

Udo

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

Hii, i hace just made a couple of test on the server. Today i couldnt reproduce the problem, i tried disconnecting and connecting USB's devices but nothing happened. Maybe it only happens when the server has certain load or maybe is just a hardware failure. I'm puzzled.

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

udo, we're ou doing something especial, was your laptop under a particular high load??? can you repeat the bug always??

 

[diaolin]

Re: Kernel Panic when disconnecting keyboard

For me the same with a UPS via USB...

when i disconnect the plug the system freezes and dies

Diaolin

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

Seeems to be the problem. The strange thing it that the server was working fine before ( a week ago ). Today i got another kernel panic, again i couldn't catch anything in the logs but a little before the kernel paniqued i got:

Apr 22 13:29:12 servidor apcupsd[2831]: Power failure.
Apr 22 13:29:14 servidor apcupsd[2831]: Power is back. UPS running on mains.

Then five minutes later i got the paniq, maybe this is a coincidence, could test further since the server is in production hours, just rebooted and got everything back to normal. But since others are replying to the thread, seems to be a bug in the kernel. I will check this out today at night and post further results.

 

[udo]

Re: Kernel Panic when disconnecting keyboard

udo, we're ou doing something especial, was your laptop under a particular high load??? can you repeat the bug always??

Hi,
no the laptop was calm and it's allways happens if I disconnect an USB-device (mouse, wifi).
But my laptop installation are not "normal" - with lxde-desktop an some other helpfull stuff installed.

I don't fill an bugreport, because it's no server-hardware but if the same happens on an server it's looks different.

@Dietmar: if it's helpfull to test the laptop with an standard pve-installation I can do that.

Udo

 

[diaolin]

Re: Kernel Panic when disconnecting keyboard

In my situation the hang happens on a server...

:-(

Diaolin

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

In my situation the hang happens on a server...

:-(

Diaolin

My server is not made of server specialized hardware is just and old computer that is acting as a server. However this happens to me also, but i cannot replicate it always, just sometimes, im puzzled of what triggers the event.

I think there is a minimum load in the server necesary for this to happen. Im using around 50% of the total memory and around 3% of total CPU when this happens, so not particular high load but it has certain load. I cannot succesfully corelate the load with the kernel panic it is just an assumption since this doesn't happens always.

udo maybe you can report if this happens on you lapptop with no DE running and thus using less
resources.

I've got the panics both with kernel 2.6.32.28 and 2.6.32.27, so if this is a kernel bug is in different kernel versions. It would be good is somebody can test it with previous kernel versions, or the default debian kernel 3.2 or the next proxmox kernel 3.10 to see is the bug is there.

 

[wahmed]

Re: Kernel Panic when disconnecting keyboard

I too have seen this issue. For me it is not consistent, but whenever the panic happened it was because of unplugging USB keyboard. I have 3 identical nodes with identical components and i only have seen this panic on one of them. It never caused the whole server crash though. Just panic.

 

[hydrosIII]

Re:Kernel Panic when disconnecting keyboard

Ok i did some tests today, and i have come conclusions to share maybe it will help other people. I tried to recreate the panics at night disconnecting USB hardware from the server, when i did this i was running programs like memtester and cpuburn to see if the load has something to do. I couldn't recreate the panics. However i got several panics earlier that day, around 3 o'clock in my country, thinking about these panics i figures out that they ocurred while using a laser printer connected to the server, this printer is a muntifunctional printer that is also used to scan and copy. When i used the copy function the server paniqued, this printer is also connected to the same power supply as the server. The clue is that when i use the printer i got a small difference in the light inside the room, it seems to shimmer for an instant. Also disconnecting USB hardware like the UPS or the printer itself (by turning it off ) made the server paniq but just during the busy hours of the day (around 1 to 4 of clock in the afternoon). So i think my issue is related to voltage. There seems to be an hour of the day when the voltage is low, affecting the performance of the server, and thus making it vulnerable to voltage changes, maybe disconnecting/connecting USB triggers a small voltage change enough to cause a hardware failure. This seems to happen when in the office we got all computers on, and there is low voltage.When i did the tests at night i didn't got the same amount of computers on, also when using the printer there was no significant change in the room electric light, so i assume that the panics had something to do with voltage, not enough voltage for the server. Tomorrow i will try confirm the cause of the issue and if possible solve it. Maybe my problem isn't correlated to other people's problem, i don't know. However it would be good if other people suffering from this problem verify that their servers are getting enough power.

 

[snowman66]

Re: Kernel Panic when disconnecting keyboard

Strange thing is if i stop UPS monitor service (using Tecnoware UPS) i cannot reproduce kernel panics and only this messages shows up each time the keyboard is disconnected:

pve1 login: drivers/hid/usbhid/hid-core.c: can't reset device, 0000:00:1a.0-1.4.4.4/input0, status -71 drivers/hid/usbhid/hid-core.c: can't reset device, 0000:00:1a.0-1.4.4.4/input0, status -71

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

Hi, I just did further experiments with the server, and here are the results. This is definitively a software bug. I cannot replicate the failure with kernel 3.2 or 3.10, i also tried with an old openvz kernel present in debian squeeze 2.6.32-5-amd64-openvz, this is also not happening.It is happening with 2.6.32-26, 2.6.32-27 and 2.6.32-28 (these are the ones i tested maybe older kernels are also affected) also with official openvz kernel for debian downloaded from: http://download.openvz.org/debian. It doesn't happens everytime it seems to be a condition for the panics to occur maybe related to voltage or to hardware, however as i said it doesn't happens with the other kernels. Also stopping the ups daemon doesn't solve anything. This issue is very important to me as i'm running only openvz virtualization, and now my server doesn´t work. I have just ended to migrate the services from plain systems to openvz containers and now i have to do the oppossite as i cannot run openvz containers !!!. I will post a bug report for this in proxmox bugzilla. If anyone knows any solutions please tell me.

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

i just created a bug report for this issue, please see: https://bugzilla.proxmox.com/show_bug.cgi?id=518, i marked the bug as critical, hope i get some feedback soon.

 

[mmenaz]

Re: Kernel Panic when disconnecting keyboard

Just for curiosity, but in the meantime waiting the bug to be solved, why not just keep the keyboard connected or don't use keyboard at all (I always log in my servers with ssh)?

 

[udo]

Re: Kernel Panic when disconnecting keyboard

...
udo maybe you can report if this happens on you lapptop with no DE running and thus using less
resources.

Hi,
I have installed yesterday an fresh pve3.2 on the laptop and the kernel panic don't happens... strange, before I was able to reproduce this on this laptop every time I disconnect an USB-device.
The problem is, that the old installation is gone (this laptop is used by my daughter now).

Udo

 

[hydrosIII]

Re: Kernel Panic when disconnecting keyboard

Hi,
I have installed yesterday an fresh pve3.2 on the laptop and the kernel panic don't happens... strange, before I was able to reproduce this on this laptop every time I disconnect an USB-device.
The problem is, that the old installation is gone (this laptop is used by my daughter now).

Udo

I have found this bug to be not consistent everytime.I think it has something to do with low voltage. It doesnt happen everytime. If the laptops have the same type of connector try the old laptop connector in the new laptop.

 

[wahmed]

Re: Kernel Panic when disconnecting keyboard

Just for curiosity, but in the meantime waiting the bug to be solved, why not just keep the keyboard connected or don't use keyboard at all (I always log in my servers with ssh)?

I have seen this issue when used a KVM Switch with 8 ports. When i swtiched nodes, it disconnected keyboard from one node to enable it for another node and the panic occured on the first node.