Win2008R2 exaggeratedly slow with 256GB RAM and strange behaviours in PVE

[cesarpk]

Hi to all.

Anybody that can help me?

My Setup:
In Hardware i have a DELL R720, two processors Intel Xeon CPU E5-2690 v2 @ 3.00GHz., 256GB RAM.
In the Hardware Bios, i have configured Numa, and processor optimized for random access to the RAM.

In Software recently installed: PVE 3.3-5 (from his ISO and upgraded), and as a VM, Win2008R2 SP1 (no other program more)

The Problem:
When i turn on the Win2008R2 VM, the cpu go to the 100% in use for each thread that have configured this VM, and when the VM is configured for example with 4GB RAM, this VM works perfectly.

I did some test, always without the use of tablet for pointer (i use vmmouse of VMware)

Here the symptoms and my actions performed:

1) Doing some tests, i have changed manually the size the page file of Windows Server to 10 GB, and configured the VM with 62 GB. RAM (63488 MB. RAM), now i can see that the Windows Server start with 100% of CPU, and after of two minutes more or less, the processor returns to a normal state.

2) Seeing the task manager of Win2008R2 VM, i get this:
Image Name: System
User Name: System
CPU: 99%
memory (Private workspace): 52 KB
Description: NT Kernel & System

All other processes are consuming 0% of CPU


Here comes the more important information of side of PVE:

3) In paralel to this strange behavior, htop is showing that the use of the memory is growing in each second that elapses, and when his memory bar say that have used "63964/257912MB", the consumption of threads of processors of this VM returns to normal state.

4) As a second test, after of see all these behaviours, I log to Windows Server, and htop show me a high consumption of many threads of processors (+/- 50%), but after that the session was initiated, the consumption of processors returns to normal state.

5) In htop, i see the same behaviour of consumption of processor while that a session of windows is closing, i guess that any thing that i do in this VM will consume processor resources extra needlessly.

6) Moreover, while that the task manager of Win2008R2 VM says that have +/- 60GB free, PVE in his tag "Summary" says exactly the conversely.

Maybe KVM or PVE have problems for manage lots RAM memory with numa enabled in the bios of the Server, but in am not sure.
Only as a reference, i show this link:
https://bugzilla.redhat.com/show_bug.cgi?id=872524

I will be very grateful to anyone that may help me.

Best regards
Cesar

 

[NdK73]

It's not clear from your post how much RAM you gave to the VM, but it seems 256G.
If you have 256G RAM on the host, you shouldn't give more than 128G to a single VM. You can experiment giving it more, but never overcommit RAM or you're looking for troubles. And remember to account for "some" memory for the host system (probably 1G could be enough, but you need to experiment).
An high load during login/logout is quite normal: many processes are started, but that should be temporary.

 

[cesarpk]

It's not clear from your post how much RAM you gave to the VM, but it seems 256G.
If you have 256G RAM on the host, you shouldn't give more than 128G to a single VM. You can experiment giving it more, but never overcommit RAM or you're looking for troubles. And remember to account for "some" memory for the host system (probably 1G could be enough, but you need to experiment).
An high load during login/logout is quite normal: many processes are started, but that should be temporary.

Hi NdK73.

Thanks for your reply

The final target is use the RAM of this manner (The 256GB RAM of Server):
For PVE:
a) Use the OS (PVE) with 8 GB. RAM.

For the VM:
b) 8 or 16 GB. RAM for the OS (the VM share some necessary files)
c) 248 or 240 GB. RAM for the MS-SQL Server.

Question:
Why If i have 256G RAM on the host, i shouldn't give more than 128G to a single VM?
( I want to use lots of RAM for MS-SQL Server... it will have more reading cache enabled )


Moreover, any suggestions?

 

[NdK73]

The "max half of the memory for a single VM" is an old rule of thumb I learnt long ago, so might be outdated.

I never used MS-SQL server (and tend avoiding anything MS anyway!), but your scheme could be good enough (if the rule above doesn't apply). Are you sure you're applying it correctly? And what do you mean with the second point? And, most important: if you're going to use ONLY this VM, why don't you install it bare-metal?

 

[cesarpk]

I have solved the problem with the latest version kernel and modifying some codes.

The "max half of the memory for a single VM" is an old rule of thumb I learnt long ago, so might be outdated.

I never used MS-SQL server (and tend avoiding anything MS anyway!), but your scheme could be good enough (if the rule above doesn't apply). Are you sure you're applying it correctly? And what do you mean with the second point? And, most important: if you're going to use ONLY this VM, why don't you install it bare-metal?

Because i will enjoy of the all features that has the virtualization, as HA, online Backup, etc., and for don't have speed degradation problems, the hardware setup must be very robust, also i have configured and optimized for the use that will give to the Bios Hardware, RAID controller, OSs and PVE.

Anyway, many thanks for your attention and suggestions, it is much appreciated.

Best regards
Cesar

 

[e100]

Cesar,

When windows starts it writes zeros to all the RAM, read on to understand why this is significant.

On the 2.6.32 kernel transparent huge pages is disabled by default.
On the 3.10 kernel they are enabled by default

I'll quote myself from here:http://forum.proxmox.com/threads/15849-Huge-Pages-Support

For example, the page table for 16GB of RAM using 4k pages is 32MB, using hugepages it is 64KB
Keeping 64KB in the CPU Cache is much easier than 32MB

In the 2.6.32 kernel, that setting is located in:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]

On 3.10 kernel its in:

# /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

Be careful with a single VM using tons of RAM.
If the OOM Killer ever kicks in it will be first to get killed.

See Trace Flag 834, it might help improve performance of SQL server: http://support.microsoft.com/kb/920093

 

[cesarpk]

Hi e100.

Many Thanks for your tips, you are a great companion.

I have the 3.10 Kernel installed in my PVEs, and his option of automatic configuration of huge page in online mode is a idea very likable.

And about of the tips on SQL server, this information is new for me, but as we have a DBA, i will give the web link to him for be sure that such information be taken into account.

Moreover, I was reading in the Web link, the "Trace Flag 834", and i have a doubt: If the linux kernel will do the huge pages for the VM, will he need the SQL server the configuration of "Trace Flag 834"?

Moreover, i need to download DRBD 8.4.5 from the git repository, but i have a problem, that is this:
shell> git clone git://git.drbd.org/drbd-8.4.git
Cloning into 'drbd-8.4'...
fatal: unable to connect to git.drbd.org:
git.drbd.org[0: 212.69.161.111]: errno=Connection timed out

And since some days ago that i can not download it, do you or anybody have a copy of this image downloaded for send me by email?
or tell me how can i download it?

Best regards
Cesar

 

[e100]

I recently got a copy of the repo, I will try to remember to tar it up for you tomorrow.

With THP on the host, the guest has no idea THP is enabled so it will still use small pages and have it's own giant page tables.

Using large pages in the guest should improve performance. I've not tried trace flag 834 myself as I only recently discovered it and have not had any mssql servers with lots of RAM.

I am currently upgrading some servers to 128GB of RAM so will be investigating these issues soon.

 

[cesarpk]

I recently got a copy of the repo, I will try to remember to tar it up for you tomorrow.

With THP on the host, the guest has no idea THP is enabled so it will still use small pages and have it's own giant page tables.

Using large pages in the guest should improve performance. I've not tried trace flag 834 myself as I only recently discovered it and have not had any mssql servers with lots of RAM.

I am currently upgrading some servers to 128GB of RAM so will be investigating these issues soon.

Many thanks e100 !!!, i will wait your email.

Moreover, if you have news about of SQL + hugepages + trace flag 834 + some other information relevant, please let me know, that from me part although i will not work in MS-SQL (that a DBA is involved), if i have good news, i can tell you about that.

Best regards
Cesar

 

[cesarpk]

Hi e100

As also i have problems for download this program:
git clone git://git.linbit.com/drbd-utils.git

I would like ask you to send me by email a copy of this download.

Ahh, please, that both programs (DRBD v8.4.5 and utils v8.9.1) be from the git, due to that if are of "http://oss.linbit.com/drbd/", i will have problems.

Note: The files can be in tar.gz file (the mail will be more short).

I very much appreciate your help.

Best regards
Cesar

 

[spirit]

Hi,

about trace flag 834, it's improve cpu interrupts, because sqlserver since 2005sp3, have a very precise timer.
with a lot of cores, without any sql traffic (simply start sqlserver), you'll see cpu usage increase (on the host side) on each core.

I think this could be solved with the new hyper-v timer enlightment(-cpu,hvtime).
But it's works only with kernel 3.10.

 

[cesarpk]

Hi e100

I got the files of DRBD from the git repository, so i don't need that you send me the files.

Many thanks again for your fine attention, which is very appreciated.

Best regards
Cesar

 

[cesarpk]

Hi,

about trace flag 834, it's improve cpu interrupts, because sqlserver since 2005sp3, have a very precise timer.
with a lot of cores, without any sql traffic (simply start sqlserver), you'll see cpu usage increase (on the host side) on each core.

I think this could be solved with the new hyper-v timer enlightment(-cpu,hvtime).
But it's works only with kernel 3.10.

Thanks for the tip

"hv_time" works for me (at least I do not see any error messages), and more later i will prove "hv_vapic"

Best regards
Cesar

 

[half_life]

Thanks for the tip

"hv_time" works for me (at least I do not see any error messages), and more later i will prove "hv_vapic"

Best regards
Cesar

Maybe late to the table but I hope this info can help someone that is experiencing trouble on an R720. You might want to examine your BIOS settings where it concerns power management. Where I work we run thousands of R720s and therefore have seen this quite a bit. Please refer to Dells documents on the subject for details. ftp://ftp.dell.com/Manuals/all-prod...dge/poweredge-1655mc_White Papers12_en-us.pdf

 

[e100]

Trace flag 8038 was created specifically to disable the high precision timer.
834 enables large pages which I suspect would also reduce CPU usage.

With 2008R2 and 2012R2 I have found that for our particular application that trace flag 4199 made a huge difference in performance.

I first discovered that from this post: http://www.sqlservercentral.com/Forums/Topic1012851-392-1.aspx#bm1013388

 

[e100]

cesarpk, Whats your email address?

Nevermind, I found it on the mailing list.

 

[cesarpk]

Maybe late to the table but I hope this info can help someone that is experiencing trouble on an R720. You might want to examine your BIOS settings where it concerns power management. Where I work we run thousands of R720s and therefore have seen this quite a bit. Please refer to Dells documents on the subject for details. ftp://ftp.dell.com/Manuals/all-prod...dge/poweredge-1655mc_White Papers12_en-us.pdf

Hi half_life.

Thanks for the document, i will read it.

Moreover, i have other document of Dell: "BIOS Performance and Power Tuning Guidelines for Dell PowerEdge 12th Generation Servers", but unfortunately i don't remember the url, but i am sure that you will can find it easily.

Best regards
Cesar

 

[cesarpk]

cesarpk, Whats your email address?

Nevermind, I found it on the mailing list.

Many thanks e100 for your fine attention (which I value very much), but as i have downloaded these programs, i will not need of your gentility, if I have caused some inconvenience, please excuse me, it's very nice for people as me, that there are people like you in this forum.

Many thanks again, and the best of success for you.

Best regards
Cesar

 

[macday]

Thanks for the tip

"hv_time" works for me (at least I do not see any error messages), and more later i will prove "hv_vapic"

Best regards
Cesar

How could i use it in the qemu config file ?

 

[cesarpk]

How could i use it in the qemu config file ?

These was the words of Spirit that he said me:

Proxmox use some hyper-v features to help for some case, including high memory.

But currently, 2 features are missing.

in /usr/share/perl5/PVE/QemuServer


> if ($ost eq 'win7' || $ost eq 'win8' || $ost eq 'w2k8' ||
> $ost eq 'wvista') {
> push @$globalFlags, 'kvm-pit.lost_tick_policy=discard';
> push @$cmd, '-no-hpet';
> #push @$cpuFlags , 'hv_vapic" if !$nokvm; #fixme, my win2008R2 hang at boot with this
> push @$cpuFlags , 'hv_spinlocks=0xffff' if !$nokvm;
> }

> if ($ost eq 'win7' || $ost eq 'win8') {
> push @$cpuFlags , 'hv_relaxed' if !$nokvm;
> }


maybe can your try to uncomment
> #push @$cpuFlags , 'hv_vapic" if !$nokvm; #fixme, my win2008R2 hang at boot with this

and restart
/etc/init.d/pvedaemon restart

and start your vm again.

(I think they was a bug in previous kernel, but maybe it's fixed now).


Another missing feature is hv_time, paravirtualized clock, but AFAIK it's only work with 3.10 kernel.

so, you can try

> if ($ost eq 'win7' || $ost eq 'win8') {
> push @$cpuFlags , 'hv_relaxed' if !$nokvm;
> push @$cpuFlags , 'hv_time' if !$nokvm;
> }

___________________________
Good luck with your work !!!

Best regards
Cesar