Error kvm : cpu0 unhandled wrmsr & unhandled rdmsr

Yes, we for one had this problem all the way from the beginning of the 2.6.3x series of kernels. Further back in the past we used other means of virtualization.
Usually in the Debian boxes a "/etc/init.d/network" restart is enough to fix it.

What's new though is that one machines which produce a lot of traffic on the network interface fail quicker or even like this in harsh circumstances:
http://kth5.archlinuxppc.org/misc/bugs/20110427-crash-bad-page-state.png

This one happened today with the following versions (I already tried disabling the KSM daemon):
# pveversion -v
pve-manager: 1.8-15 (pve-manager/1.8/5754)
running kernel: 2.6.35-1-pve
proxmox-ve-2.6.35: 1.8-10
pve-kernel-2.6.32-4-pve: 2.6.32-32
pve-kernel-2.6.35-1-pve: 2.6.35-10
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-3
ksm-control-daemon: 1.0-5
Click to expand...
[edit]
Forgot to mention that the crash occured after an attempt to restart the network and pve-kernel-2.6.35-1-pve is running.
 
Last edited:
do also see this on 2.6.18? any hardware issue? faulty ram?
 
Ram is OK, checked after an upgrade a week ago or so. I can't downgrade to 2.6.18 right now as this would block a bunch of devs from doing their work. ;) I'll see what I can get done on the weekend perhaps.

Funny thing is, it always only affects the same two VMs on that host this way. The others are always fine with a network restart (haven't had another case yet), only the two that produce loads of traffic crash like this it seems.
 
>Funny thing is,
There is nothing funny to this, sorry. I have the same error messages on the console of a (kvm) production machine:

Code: 
blade3:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.24-8-pve: 2.6.24-16
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6
blade3:~#

Has this been solved?

j.
 
I got this issue too.

I have joined the dmesg output of my 3 servers.
virt1 = Dell PowerEdge 2950 / Uptime:112 days
virt2 = Dell PowerEdge 2950 / Uptime:112 days
virt3 = Dell PowerVault NF500 / Uptime:43 days

All vms :
Disks: RAW - VirtIO/IDE on NFS/LOCAL.
Network : Debian:VirtIO - Windows:e1000


pveversion -v for all hosts:
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6
 

Attachments

  • virt1.txt.zip
    9.2 KB · Views: 3
  • virt2.txt.zip
    9.3 KB · Views: 0
  • virt3.txt.zip
    9.5 KB · Views: 0
Stumbled on this thread, because I found same error in logs.

Besides that, I had issues with the webinterface sometimes not reacting, or reacting with an error, vm's not starting, and several others. When ssh-ing in and trying to do some disktests I found several programs segfaulting.

Turned out to be a faulty memory-stick. Have replaced it and everything seems to be back to normal now.

Maybe do some hardware checks to be sure that this os ok.

Rgds
L1
 
kth5 said:
We have the same problem (losing network connectivity) since long before 1.7 with all Debian Lenny & Squeeze guests (default Installations / custom Java applications). Usually a /etc/init.d/networking restart fixes the problem but sooner or later it will happen again and again. Sometimes in such short intervals that it really becomes annoying. No messages in dmesg on either host or guest. The unhandled instruction warnings do not seem to be related though.

This happens on a DL380G5 (1xQuad E5420, 12GB RAM, 146-SAS HP) set up like so:

As well as on a DL360G6 (2xQuad E5504, 24GB RAM, 8x500GB SAS-HP) set up like so:

This issue has been reported before and wasn't solved nor could the Proxmox stuff reproduce this. On our side this problem has been around for a very long time but for some strange reason doesn't happen "as often" to the VMs we do use in production only on our testing and staging system where we virtualize, bad enough... :(
Click to expand...

This sounds like the problems associated with broadcom network adapters. I manage bunch of DL3XXG5 G6 and G7 boxes and at least G5 generation has been quite unusable with 2.6.3x kernels. We took the easiest way out and installed Intel NICs on all boxes. No problems at all afterwards.

Broadcom NICs doesn't always behave well with tagged vlans with MTU 9000 either. Yet another reason to switch to Intel.

Current kernel versions SHOULD be OK in terms of NIC freezes, but your milage may vary.
 
Warod said:
This sounds like the problems associated with broadcom network adapters. I manage bunch of DL3XXG5 G6 and G7 boxes and at least G5 generation has been quite unusable with 2.6.3x kernels. We took the easiest way out and installed Intel NICs on all boxes. No problems at all afterwards.

Broadcom NICs doesn't always behave well with tagged vlans with MTU 9000 either. Yet another reason to switch to Intel.

Current kernel versions SHOULD be OK in terms of NIC freezes, but your milage may vary.
Click to expand...

Hello everyone,

I think Warod is right.

I've encountered huge ploblems, is BroadCom BCM 5709 on ESXi as well. It don't support MTU 9000 with iSCSI offload.
And it isn't a driver problem, but a firmware one ...

We don't have the budget to switch to intel network card so MTU 1500 and VmWare iSCSI software.

Hope it helps. ;)
 
Last edited:
I am having the same error with these nics

Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

kvm: 1822: cpu0 unhandled rdmsr: 0x345
kvm: 1822: cpu0 unhandled wrmsr: 0x40 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x60 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x41 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x61 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x42 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x62 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x43 data 0
kvm: 1822: cpu0 unhandled wrmsr: 0x63 data 0
kvm: 1822: cpu1 unhandled wrmsr: 0x40 data 0

pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-74
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-14-pve: 2.6.32-74
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-49
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-31
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-8
ksm-control-daemon: 1.1-1
 
This is with virtio for the drive

kvm: 15001: cpu0 unhandled rdmsr: 0x345
kvm: 15001: cpu0 unhandled wrmsr: 0x40 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x60 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x41 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x61 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x42 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x62 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x43 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x63 data 0
kvm: 15001: cpu1 unhandled wrmsr: 0x40 data 0
tap102i0: no IPv6 routers present
device tap101i0 entered promiscuous mode
vmbr0: port 4(tap101i0) entering forwarding state
tap101i0: no IPv6 routers present
__ratelimit: 7 callbacks suppressed
kvm: 15001: cpu0 unhandled rdmsr: 0x345
kvm: 15001: cpu0 unhandled wrmsr: 0x40 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x60 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x41 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x61 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x42 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x62 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x43 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x63 data 0
kvm: 15001: cpu1 unhandled wrmsr: 0x40 data 0
__ratelimit: 7 callbacks suppressed
kvm: 15001: cpu0 unhandled rdmsr: 0x345
kvm: 15001: cpu0 unhandled wrmsr: 0x40 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x60 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x41 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x61 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x42 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x62 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x43 data 0
kvm: 15001: cpu0 unhandled wrmsr: 0x63 data 0
kvm: 15001: cpu1 unhandled wrmsr: 0x40 data 0
 
Hi everyone!

I tried to reconstruct the error and found a working solution (at least at my testing machines) :)

ERROR Definition

Nov 15 09:27:32 px0 kernel: __ratelimit: 7 callbacks suppressed
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled rdmsr: 0x345
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x40 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x60 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x41 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x61 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x42 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x62 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x43 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu0 unhandled wrmsr: 0x63 data 0
Nov 15 09:27:32 px0 kernel: kvm: 5555: cpu1 unhandled wrmsr: 0x40 data 0

After that the VM shuts down immediatly.

Information Gathering
I looked up the internet to gather some details about the errors. This is what I found:

MSR - Model Specific Registers. This registers can be used to look up supported features of the processor model or to configure features. Do not fuck around with these registers, if you have no idea what you are doing.
RDMSR - Read from MSR. Look up supported features of the processor
WRMSR - Write to MSR. Set a featurebit to enable it on the processor
0x345 - IA32_PERF_CAPABILITIES. It defines the performance monitoring feature of the processor.

I didn't find anything about the oher registers.

Testing

I tried to narrow down the problem by using different VM setups.

General
Proxmox VE 2.2
Images are located on an iSCSI target under the LVMs!

Setup 1
Memory: 2.00 GB
Processor: 1 (1 socket, 2 cores)
[*]
HDD: SCSI, no-cache
CDROM: local(live-cd)
Network: VIRTIO

[Core2Duo] -> MSR Errors
[qemu-64bit] -> No MSR Errors, but VM didn't boot
[cpu64-rhel6] -> No MSR Errors, but VM didn't boot
[host](Intel) -> MSR Errors

These tests led to the result, that it seems to be an Intel Processor problem, but wait there is more ...

Setup 2
Memory: 2.00 GB
Processor: 1 (1 socket, 2 cores)
[*]
HDD: SATA, no-cache
CDROM: local(live-cd)
Network: VIRTIO

[Core2Duo] -> MSR Errors, but VM boots and I can perform an installation of the os
[qemu-64bit] -> No MSR Errors, VM boots
[cpu64-rhel6] -> MSR ( CPu0 disabled perfctr wrmsr: 0xc1 data 0xabcd), VM boots
[host](Intel) -> MSR Errors, Vm boots

All VMs boot up correctly, with or without MSR errors.

Conclusion

Changing the storage driver doesn't solve the MSR error problem but it solves the aftereffects of it, in my case.



PS: Is somebody able to explain why the SCSI driver doesn't work?
 
Is there any progress about this issue?

I have installed proxmox 3.1.17 on my Hp Microserver N40L and on the host console seen a message like below,

- kvm: 2578: cpu0 unhandled rdmsr: 0xc0010112
- kvm: 2578: cpu0 unhandled rdmsr: 0xc0010001

Does anyone still have any messages like this ?
 
Hi,

yes, I am having this issue too. Especially when there is (heavy) load on the (virtual) network interface, e.g. while doing backup using NFS from the proxmox host to a vm (openmediavault). This leads often to a server hang up, because NFS connection seems to go to nirvana and hosts ist waiting endlessly for NFS to come back. The problem also occurs when using SMB.

Code: 
root@proxmc:/var/log# grep -ri unhand .
./kern.log:Nov 25 21:01:59 proxmc kernel: kvm: 3159: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 25 21:40:09 proxmc kernel: kvm: 3137: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 25 21:40:20 proxmc kernel: kvm: 3363: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 25 23:47:49 proxmc kernel: kvm: 3169: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 25 23:50:03 proxmc kernel: kvm: 3626: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 26 09:02:42 proxmc kernel: kvm: 3024: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 26 09:06:14 proxmc kernel: kvm: 3481: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 26 09:06:30 proxmc kernel: kvm: 3515: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 26 09:06:30 proxmc kernel: kvm: 3515: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 26 22:23:12 proxmc kernel: kvm: 3024: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 26 23:40:58 proxmc kernel: kvm: 30349: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 27 10:48:16 proxmc kernel: kvm: 49958: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 27 10:48:16 proxmc kernel: kvm: 49958: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 28 09:39:00 proxmc kernel: kvm: 3062: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 28 09:42:23 proxmc kernel: kvm: 3068: cpu0 unhandled rdmsr: 0xc0010001
./kern.log:Nov 28 09:47:07 proxmc kernel: kvm: 3534: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 28 09:47:10 proxmc kernel: kvm: 3561: cpu0 unhandled rdmsr: 0xc0010112
./kern.log:Nov 28 09:47:10 proxmc kernel: kvm: 3561: cpu0 unhandled rdmsr: 0xc0010001
./syslog:Nov 28 09:39:00 proxmc kernel: kvm: 3062: cpu0 unhandled rdmsr: 0xc0010001
./syslog:Nov 28 09:42:23 proxmc kernel: kvm: 3068: cpu0 unhandled rdmsr: 0xc0010001
./syslog:Nov 28 09:47:07 proxmc kernel: kvm: 3534: cpu0 unhandled rdmsr: 0xc0010112
./syslog:Nov 28 09:47:10 proxmc kernel: kvm: 3561: cpu0 unhandled rdmsr: 0xc0010112
./syslog:Nov 28 09:47:10 proxmc kernel: kvm: 3561: cpu0 unhandled rdmsr: 0xc0010001

Code: 
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-25-pve: 2.6.32-113
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back