Hi all
I think I'm hitting the same issue mentioned in http://forum.proxmox.com/threads/5046-Error-kvm-cpu0-unhandled-wrmsr-amp-unhandled-rdmsr but I'm starting a new thread because I'm not sure it's related to the "cpu0 unhandled wrmsr: 0x198" message.
I have a box with 2 x Xeon 5300, for a total of 8 cores, and I'm trying to run a number of VMs on this: 2 with 8 cores, 2 with 1 core, plus the odd other one, all running Ubuntu 10.04 LTS with kernel 2.6.32-32-virtual. Often when booting the 8-core VMs they will hang halfway throught the initialization process: seemingly at a different point each time.
The problem seems related to the number of cores in the VM. If I'm running 1x8 core and 1x1 core guests, and I try to start another 8 core guest, it hangs. If I reduce it to 4 cores, it starts. The magic number seems to be about twice the number of actual cores I have, after which point it fails about 80% of the time.
This is with 2.6.35-1, but if I keep the same setup then downgrade kernel to 2.6.24 it works. Here's the output of pveversion -v when it fails:
pve-manager: 1.8-17 (pve-manager/1.8/5948)
running kernel: 2.6.35-1-pve
pve-kernel-2.6.35-1-pve: 2.6.35-11
pve-kernel-2.6.24-12-pve: 2.6.24-25
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.26-1pve4
vzdump: 1.2-12
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
it's a fresh install (over top of a fresh Debian Lenny rather than direct from your boot disk).
I was also trying this under a fresh install of Ubuntu Natty as well, and seeing the same results - start a VM as "kvm -M 512 -smp 8 guest.qcow2" and it hangs, but change to "-smp 4" and it's fine. I can also reproduce it in Proxmox without the UI by running the same command.
More info.
* This happens with both 2.6.32-32-virtual on the guest and also the "linux-generic-pae" kernel.
* If the guest gets through the kernel loading, it seems stable - I can thrash all the cores on the running image while attempting to load the other one, and the running one will operate correctly.
* When the guest hangs, it's KVM process is using a lot of CPU (busy loop of some sort)
* ksmtuned was running under Proxmox 2.6.32, and ksmd under Ubuntu.
* Not a resource issue - guest images are stored locally on SSD, there's no swap and plenty of RAM to spare, no message in server logs (other than the "cpu0 unhandled wrmsr: 0x198" one on starting guest, but I see that regardles of success or failure), and when the guests start they're idle, so the chips are definitely NOT too busy to handle a VM with this number of cores.
Also I should point out that I did have this working for a while on a previous build, which was an awful hybrid of Ubuntu kernel and components with Proxmox 1.7 merged into it (with various --force options). Not really maintainable and probably not much use for a bug report, but I mention it for completeness.
Hope this is enough for you guys to reproduce it this time. I have some kernel boot logs of the hung instances under Ubuntu (not Proxmox, as I said the symptoms are the same) - let me know if you want them.
Cheers... Mike
I think I'm hitting the same issue mentioned in http://forum.proxmox.com/threads/5046-Error-kvm-cpu0-unhandled-wrmsr-amp-unhandled-rdmsr but I'm starting a new thread because I'm not sure it's related to the "cpu0 unhandled wrmsr: 0x198" message.
I have a box with 2 x Xeon 5300, for a total of 8 cores, and I'm trying to run a number of VMs on this: 2 with 8 cores, 2 with 1 core, plus the odd other one, all running Ubuntu 10.04 LTS with kernel 2.6.32-32-virtual. Often when booting the 8-core VMs they will hang halfway throught the initialization process: seemingly at a different point each time.
The problem seems related to the number of cores in the VM. If I'm running 1x8 core and 1x1 core guests, and I try to start another 8 core guest, it hangs. If I reduce it to 4 cores, it starts. The magic number seems to be about twice the number of actual cores I have, after which point it fails about 80% of the time.
This is with 2.6.35-1, but if I keep the same setup then downgrade kernel to 2.6.24 it works. Here's the output of pveversion -v when it fails:
pve-manager: 1.8-17 (pve-manager/1.8/5948)
running kernel: 2.6.35-1-pve
pve-kernel-2.6.35-1-pve: 2.6.35-11
pve-kernel-2.6.24-12-pve: 2.6.24-25
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.26-1pve4
vzdump: 1.2-12
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
it's a fresh install (over top of a fresh Debian Lenny rather than direct from your boot disk).
I was also trying this under a fresh install of Ubuntu Natty as well, and seeing the same results - start a VM as "kvm -M 512 -smp 8 guest.qcow2" and it hangs, but change to "-smp 4" and it's fine. I can also reproduce it in Proxmox without the UI by running the same command.
More info.
* This happens with both 2.6.32-32-virtual on the guest and also the "linux-generic-pae" kernel.
* If the guest gets through the kernel loading, it seems stable - I can thrash all the cores on the running image while attempting to load the other one, and the running one will operate correctly.
* When the guest hangs, it's KVM process is using a lot of CPU (busy loop of some sort)
* ksmtuned was running under Proxmox 2.6.32, and ksmd under Ubuntu.
* Not a resource issue - guest images are stored locally on SSD, there's no swap and plenty of RAM to spare, no message in server logs (other than the "cpu0 unhandled wrmsr: 0x198" one on starting guest, but I see that regardles of success or failure), and when the guests start they're idle, so the chips are definitely NOT too busy to handle a VM with this number of cores.
Also I should point out that I did have this working for a while on a previous build, which was an awful hybrid of Ubuntu kernel and components with Proxmox 1.7 merged into it (with various --force options). Not really maintainable and probably not much use for a bug report, but I mention it for completeness.
Hope this is enough for you guys to reproduce it this time. I have some kernel boot logs of the hung instances under Ubuntu (not Proxmox, as I said the symptoms are the same) - let me know if you want them.
Cheers... Mike
I must have missed the the 2.6.32 while looking for the kernel versions to downgrade to, will try that this evening and let you know the results.