Win2003R2 in KVM VM is slow in PVE 2.2 when multiply CPU cores allowed

A

achekalin

Member
Jun 17, 2011
74
0
6
Hello!

we run Windows 2003 R2 x32 server for quite a long time now (since first 2.x release I believe) in KVM VM and all that time OS was running smooth and fine. Tha machine is like this:

Code: 
boot: c
bootdisk: virtio0
cores: 2
cpu: host
memory: 4096
name: Server
net0: virtio=CA:DE:F7:DD:A2:02,bridge=vmbr0
ostype: wxp
sockets: 2
virtio0: local:101/vm-101-disk-1.qcow2,cache=writeback,size=32G
virtio1: local:101/vm-101-disk-2.raw,size=12G

The 32G disk is system one, the 12G is for swap file (so no qcow2 was chosen, just old plain raw).

If I run the server with only one core and one socket (1 core total), seems no problem. If I try to run 2x2 or more (tried up to 4x4, got no difference), the VM suddenly become very slow. The same picture was seen when I installed Linux distro into second VM with 4 cores (btw, that was latest Debian with out-of-box kernel, not that I used old kernel or old Linux OS).

I find that by adding
Code: 
args: -machine pc,kernel_irqchip=off
to the .conf file someone was able to fix these symptoms (in fact this was the way to set cpu0 only to process IRQs, if I got that right), so I tried that trick. As I can see, Linux machine feel much better that way, but Windows 2003 won't able to even start.

Then I edit sources.conf and added pvetest repo name to proxmox line, and installed all the updates apt found, the 'slowness' issue have disappeared (or may be I just wont' see it as far), so finally server run as intended.

But the question is: are there any plans to fix that soon (as running production server with pvetest's kernel and software isn't the best practice, isn't it?)

Or maybe I should downgrade packages (then - which ones and what way the safest one?)

Thank you in advance, I love Proxmox and hope this issue can be solved in easy way.
 
I assume the behavior changed because qemu/kvm 1.3 (already pvetest) and not because of the kernel change. test it. but the kernel currently available in pvetest is a quite stable one.

qemu 1.3 is already released as stable (see qemu.org) and we plan to release these packages to the stable proxmox ve repository before Christmas. if you can´t wait, run pvetest and change back to pve after release.
just follow the announcements in this forum.

back to your "slowdown" problem:
how can I reproduce this here in our test lab? please give detailed instruction to build such a VM - and how do you benchmark?
 
Thanks for pointing out that qemu/kvm 1.3 is already stable one, I feel much better now knowing it, no jokes!

To reproduce... The machine is HP DL360G5, 2 x Xeons, 410i RAID with battery, 2 x SAS disks in mirror - just an ordinary (while branded) server. We got some of such servers so I used backup one to fresh install PVE 2.2.

Windows 2003 R2 SP2 on it is the once you can find everywhere, nothing special with it. x86 version was chosen "for historical reasons". Oh, I forget, we've changed cache mode for the VM drives to test if it helps (no luck) so writeback is not mandatory for us, we used to use default cache.

I've installed it as is, then added virtio drivers from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-30.iso , that's, too, very ordinarily. The Windows server itself is a terminal server, so the only things that enabled on it was AD support and terminal server service. We've tested both out-of-box and fully updated - the same picture.

How it looks like? When VM have at least 2 or more cores, the Windows boot process slow down 3-4 times and VM CPU constantly at 90-100%. Moreover, when server finally up, users feel like they use Pentium 166 and not Xeons. Even worse, some users reported they lost their network connection to the server (very rarely, but anyway).

I think that symptoms are nothing like it should be, that why I write that.

By the way, as I've upgraded to pvetest my colleagues and users reported that server run faster than it run before moving to 2.2. So looks like pvetest introduced cure for the problem, so hope you'll promote it to pve soon (won't go for pvetest on every server).

Thank you Tom for dealing with that, hope my description will help you but I see nothing "special" in my words.
 
Last edited:
achekalin said:
Thanks for pointing out that qemu/kvm 1.3 is already stable one, I feel much better now knowing it, no jokes!

To reproduce... The machine is HP DL360G5, 2 x Xeons, 410i RAID with battery, 2 x SAS disks in mirror - just an ordinary (while branded) server. We got some of such servers so I used backup one to fresh install PVE 2.2.

Windows 2003 R2 SP2 on it is the once you can find everywhere, nothing special with it. x86 version was chosen "for historical reasons". Oh, I forget, we've changed cache mode for the VM drives to test if it helps (no luck) so writeback is not mandatory for us, we used to use default cache.

I've installed it as is, then added virtio drivers from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-30.iso , that's, too, very ordinarily. The Windows server itself is a terminal server, so the only things that enabled on it was AD support and terminal server service. We've tested both out-of-box and fully updated - the same picture.

How it looks like? When VM have at least 2 or more cores, the Windows boot process slow down 3-4 times and VM CPU constantly at 90-100%. Moreover, when server finally up, users feel like they use Pentium 166 and not Xeons. Even worse, some users reported they lost their network connection to the server (very rarely, but anyway).

I think that symptoms are nothing like it should be, that why I write that.

By the way, as I've upgraded to pvetest my colleagues and users reported that server run faster than it run before moving to 2.2. So looks like pvetest introduced cure for the problem, so hope you'll promote it to pve soon (won't go for pvetest on every server).

Thank you Tom for dealing with that, hope my description will help you but I see nothing "special" in my words.
Click to expand...

can you try last virtio from here :
http://people.redhat.com/vrozenfe/build-48/

(updated 7 december 2012). This is the last (dev) version, but they are very stable. (and they are a lot of bugfixes since build 30)

We have seen same problem with linux with 2.6.32 kernel (seem to be a virtio bug with old virtio driver and new qemu-kvm). (3.2 kernels don't have the problem)
Maybe last windows virtio can help ...?
 
spirit said:
We have seen same problem with linux with 2.6.32 kernel (seem to be a virtio bug with old virtio driver and new qemu-kvm). (3.2 kernels don't have the problem)
Maybe last windows virtio can help ...?
Click to expand...
Will try to check if that hepls ut do you really think virtio drivers may help with that 'extra cores = extra slow" problem? I see that slowness (Windows) even when it run the Windows installation from iso (and the devices were not virtio, I've installed on IDE and Realtek, and changed to virtio after the setup finished) , so I'd say the problem if not virtio-related, as for me.

Anyway, thank you for pointing out the new version! Frankly, we've never seen any bugs in virtion we're using, but we'll give it a try as soon as we can manage to update one of our servers. Thanks!
 
achekalin said:
Will try to check if that hepls ut do you really think virtio drivers may help with that 'extra cores = extra slow" problem? I see that slowness (Windows) even when it run the Windows installation from iso (and the devices were not virtio, I've installed on IDE and Realtek, and changed to virtio after the setup finished) , so I'd say the problem if not virtio-related, as for me.

Anyway, thank you for pointing out the new version! Frankly, we've never seen any bugs in virtion we're using, but we'll give it a try as soon as we can manage to update one of our servers. Thanks!
Click to expand...

for kernel 2.6.32, yes, it's a virtio network driver problem, using vhost=on and new in-kernel irqchip from qemu 1.2.

But maybe your windows problem is not related. glad to see it's works fine with qemu 1.3 !

 
I got some really strange news...

As I've upgraded PVE to the pvetest, and try to run two Windows 2003 R2 x86 KVM VMs (they are the same, except for MACs and IPs) on it, I see pretty unusual behavior: as I run first VM, it run well. The host machine load is about 20-30%, the host memory is free enough (host's RAM is 20 Gb, and the VM ram set to 4 Gb), disks not under load.

Ok, then I run second VM. As it starts, the first VM's load jumps up to 60-70% (in fact the load on 1st VM looks like VM2 load adds up to VM1 load). The host load run to 100% and holds that way for 3-5 minutes, and both VMs are almost freezers - once I see windows screensaver that won't off as I moved the mouse over it!

We've updated virtio to new version, it won't help. I also played with KSM on host, no visible effect.

Looks like we doing something wrong. Now should we go with such a system (if so, what to do next?), or install PVE 2.1 instead (which is proved to work fine on the server) and simple disable any upgrades?

P.S. By the end of the day we'll try to run ubuntu livecd in separate VM on that server instead of second Windows VM, to test if it related to Windows itself.
 
no idea whats wrong on your side but we run a lot of windows boxes in our test lab with latest packages with pvetest, so far I cannot see issues like you described.
 
tom said:
no idea whats wrong on your side but we run a lot of windows boxes in our test lab with latest packages with pvetest, so far I cannot see issues like you described.
Click to expand...

I also don't have problem (proxmox 2.2 or pvetest), but with win2003 x64.


Do you have tried with cpu: qemu64 ?


also maybe can you try to install irqbalance:

#apt-get install irqbalance.

maybe it can help ?

 
spirit said:
can you try to choose win2008 as ostype ?
(this add -no-hpet , disabling hpet timer)
Click to expand...

Tried to, looks promising! Should I use win2008 or win2008r2 as ostype?

And... if there any way I can use to disable hpet on all VMs at once?
 
achekalin said:
Tried to, looks promising! Should I use win2008 or win2008r2 as ostype?
Click to expand...
this is the same, no difference in proxmox code

And... if there any way I can use to disable hpet on all VMs at once?
Click to expand...
no.... :(

But maybe it should be the default for win2003, now that kernel_irqchip=on is enable by default ? (so we can change proxmox code).

Can you do extensive tests ? (maybe also with x64 win2003 if possible ?)
 
spirit said:
this is the same, no difference in proxmox code


no.... :(

But maybe it should be the default for win2003, now that kernel_irqchip=on is enable by default ? (so we can change proxmox code).

Can you do extensive tests ? (maybe also with x64 win2003 if possible ?)
Click to expand...


I have just redone test, win win2003 R2 x64, with qemu 1.3 (pvetest), I have a blue screen at boot with hpet enable. (it was working fine with qemu-kvm 1.2).
Booting without hpet works fine.

I'll submit a mail to to qemu mailing list.
 
Good. Just curiosity... are you a "senior member" as stated in your forum profile, or have you been promoted "Proxmox staff"? If you can change proxmox packaged I think you are the latter.
 
mmenaz said:
Good. Just curiosity... are you a "senior member" as stated in your forum profile, or have you been promoted "Proxmox staff"? If you can change proxmox packaged I think you are the latter.
Click to expand...

I'm just contributor (sending patches or features to proxmox dev mailing list) and not in proxmox staff. (proxmox doesn't paid me ;).

But I try to help proxmox users when I have time. (Proxmox have a great community :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back