After upgrade version 1.9 to 3.2(3.1), Windows 2008 R2 performance issue

[findfast]

Hi all!

Long time 1.9 version was satisfy performing for VM virtualisation around 10 VM (some Linux Ubuntu 10/12) other Windows (XP, 2003, 2008, 2008 R2).
Faced with difficulties to get up windows 2012 R2 on proxmox 1.9. Got to know then easiest way - upgrade to 3.x.
Also got information on this forum that better option is 3.2 version (no problems with backup as it was with 3.1).

The way of upgrade:
- Install on new clear SSD from latest ISO Proxmox 3.2 (old disk removed from the system)
- recreate new config file in /etc/pve/nodes/pro/qemu-server from old conf files
- setup as before all vmbr's
- config same backup place

Starting all VMs was successfull (Linux and Widows). The only problems was with windows 2008 R2 Enterprise.
3 of them did some drivers renew (COM, CPU, other) and after that started normally.
1 of them didn't start in any way. After researching found two courses:
args: -no-hpet -no-kvm-pit-reinjection - shouldn't be used
and
cpu type coundn't be win7/2008r2 nor 2008, only started with some xp/2003 or Linux 3.X/2.6 Kernel .

During working those 4 VM (win2008 r2) got significant performance regression. All have terminal server role.
Slow windows switching, long opening files.
=============
Currently what I have.
I used all performance tricks:
tablet: 0
scsihw: virtio-scsi-pci
args: -no-hpet -no-kvm-pit-reinjection (yes and no)
tried E1000 and IDE
Then I've reinstalled Proxmox 3.1 and test all once again.
I did almost all tests with different Virtio drivers combination.

I'm testing with DPC latency checker. It shows exactly when responce of system is not acceptable.
1) Chart of one Windows 2008 R2 Standart (Domain Controller) hasn't impacted after upgrade :

Using virtio 0.1-65. No nigh load. No change after restarting VM.
2) Chart of problematic Windows 2008 R2 Enterprise (terminal server, virtio).
Just after start:

During no load:

Under regular load:

Same with E1000 under load:

System responce very poor for users activity. VM CPU showing 40-70%, Proxmox shows 60-100% of CPU using.
All others VM with windows XP, 2003, 2008 have nice charts and performance, but non of them have terminal role.

Pveverion:
proxmox-ve-2.6.32: 3.1-109 (running kernel: 2.6.32-23-pve)
pve-manager: 3.1-3 (running version: 3.1-3/dc0e9b0e)
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-7
qemu-server: 3.1-1
pve-firmware: 1.0-23
libpve-common-perl: 3.0-6
libpve-access-control: 3.0-6
libpve-storage-perl: 3.0-10
pve-libspice-server1: 0.12.4-1
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.0-2


Comeback to 1.9 didn't performed because needs in upgrade all 2008R2 to 2012R2. Rollback left for no chance to solve this issue.
Seems my issue is related to kernel versions or maybe some bios settings?

Has anybody faced with such difficulties?

 

[findfast]

I've tried to install clear Windows 2008 R2 Enterprise (same distr) with no load, no roles and virtio drivers as wiki recommend. But same behave:

Not possible to get nice green chart or even get it lower.
I didn't try to install clear Windows 2008 R2 Standart.

But I've tried to start with one core. It was long starting but got such chart:

=================
Want to admit. I've installed on different production server Proxmox 3.1 a few Windows 2008 R2 Standart for same tasks. one for terminal services, one for AD, one for firebird DB, one for PostgreSQL, and some Windows 7 Pro. All of them works fine but low load, currently very low loads. Charts all the time in green area.

 

[findfast]

Today I'm following testing. I've installed new windows server 2008 r2 standart (both virtio drivers 0.1-65).
And got almost same chart as 2008 r2 enterprise (on same host)


Here is config file:
balloon: 0
boot: c
bootdisk: virtio0
cores: 8
ide2: local:iso/virtio-win-0.1-65.iso,media=cdrom,size=60558K
ide3: local:iso/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008_R2_64Bit_English_w_SP1_MLF_X17-22580.ISO,media=cdrom,size=3092500K
memory: 32768
name: termo16
net0: virtio=4E:25:1A:2B:38:49,bridge=vmbr0
ostype: win7
scsihw: virtio-scsi-pci
sockets: 2
tablet: 0
virtio0: m5:1116/vm-1116-disk-1.raw,format=raw,size=40G


Can somebody perform same test on server 2008 R2 with DPC latency checker utility?
What version of virtio drivers do you use? Net+Stor.

My # pveperfCPU BOGOMIPS: 640011.20
REGEX/SECOND: 806136
HD SIZE: 29.28 GB (/dev/mapper/pve-root)
BUFFERED READS: 184.26 MB/sec
AVERAGE SEEK TIME: 0.14 ms
FSYNCS/SECOND: 1603.84
DNS EXT: 102.90 ms
DNS INT: 29.71 ms (a.local)

 

[hb22]

Hello,

I have the same issue with migration VM Windows 2008 Enterprise from Proxmox 1.9 to Proxmox 3.3.5.
VMs with Windows 2008 standard is ok.
You have found the solution?

Hervé.

 

[Whatever]

Do you setup fixed or dynamical memory allocation (for VM)?

I've seem almost the same behavior with dynamical allocation and memory driver installed (inside VM)

 

[hb22]

This VM is configured with fixed memory.
With disk virtio ou IDE driver, it's the same issue.
With ethernet virtio or E1000 driver, it's the same issue.
VM Windows 2008 STD on the same host, it's ok.

 

[Whatever]

In my case an issue was in VirtIO Balloon driver (check device manager for unrecognized device).

Anyway, you can try to follow next tutorial in order to find out what is going on on your system: http://www.msfn.org/board/topic/140263-how-to-get-the-cause-of-high-cpu-usage-by-dpc-interrupt/

 

[findfast]

Hi,
I've found workaround for this issue.
I have also fixed memory size and played disabling balloon: 0 in config.
It is also confirmed with Enterprise edition and new installation Standart is ok.
Every night performing backup with suspend VM.
In the morning I've found that DPC latancy is good (150-1000 depend on VM, I've 4 enterprise).
So:
1) shutdown VM
2) start VM
3) after some time (hour(s)) perform backup with suspend
4) after some time - windows will perform normal

My thoughts - it is selfoptimizing widows features, deep in system kernel.
CPU type - proxmox default.

I had also some other tips but not ready to give it as manual now.
Just try it and reply.
In a few days I cound get my knowings together