Huge Pages Support?

[e100]

Now that I have some systems with 128GB of RAM it seems that using HugePages would be very beneficial.

For example, the page table for 16GB of RAM using 4k pages is 32MB, using hugepages it is 64KB
Keeping 64KB in the CPU Cache is much easier than 32MB

Looking at change logs I see that openvz at one point set transparent huge pages to disabled by default (enabled by default in upstream redhat)
But I can not find an explination as to why it was disabled.

Is there some issue between openvz and transparent huge pages?
If so, can I use transparent huge pages with KVM if I do not use any openvz containers?

 

[e100]

I have enabled Transparent Huge Pages on a couple of servers, not noticed any negative impacts on openvz so far.

Anyone with more than 8GB of RAM should be interested in this, from what I have read it can lead to a 10%-20% performance improvement.
Here are some slides with detailed information: http://www.linux-kvm.org/wiki/images/9/9e/2010-forum-thp.pdf

 

[Raymond Burns]

That document says it is not good for KVM.
Essentially, you would need an OpenVZ only node, and a KVM node

 

[e100]

That document says it is not good for KVM.
Essentially, you would need an OpenVZ only node, and a KVM node

No, it says Hugetlbfs is bad for KVM, not Transparent Huge Pages.
There are two ways to use Huge Pages:
1. Transparent Huge Pages (THP), as the name implies, transparent to the process.
2. Hugetlbfs works with applications written specifically to use Huge Pages and access the huge pages via the pre-allocated Hugetlbfs

Hugetlbfs should perform the best but you loose things like ballooning, KSM, ability to swap VM when memory pressure is high, etc.
Since I do not want to loose all those KVM features I am only interested in THP.

One of the benchmarks in that document were faster using KVM with THP used in the Host and the Guest than when running on bare metal. see page 25

 

[spirit]

Now that I have some systems with 128GB of RAM it seems that using HugePages would be very beneficial.

For example, the page table for 16GB of RAM using 4k pages is 32MB, using hugepages it is 64KB
Keeping 64KB in the CPU Cache is much easier than 32MB

Looking at change logs I see that openvz at one point set transparent huge pages to disabled by default (enabled by default in upstream redhat)
But I can not find an explination as to why it was disabled.

Is there some issue between openvz and transparent huge pages?
If so, can I use transparent huge pages with KVM if I do not use any openvz containers?

Hi,
yes it was disable by default by openvz kernel, don't remember when exactly, maybe 1 year ago.
But it's safe to enable it for kvm. (I'm using 512GB systems with transparaent hugepage enabled)

echo always > /sys/kernel/mm/transparent_hugepage/enabled 
echo always > /sys/kernel/mm/redhat_transparent_hugepage/enabled

(don't known why we have 2 entries, maybe it's related to openvz patches)

 

[Raymond Burns]

Is it possible to get a Tuning Forum Category?
No way I will remember all this for all the Proxmox systems I deploy. But I want as much optimal performance as possible, especially if the servers will be on 24/7

 

[e100]

Do you know if it causes issues with openvz or not?

The change log had no explination just said something like 'disable the by default'

That is all the info my googlefu was able to find.

 

[e100]

We should put stuff like this in the wiki once we have solid information on what we can/cannot do.

Another way to never forget is to automate it with Chef (or salt, puppet, etc) thats what I do.

 

[e100]

I think it is only necessary to set one, not both.
ran this:

#echo always > /sys/kernel/mm/transparent_hugepage/enabled

Then this:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] never

 

[spirit]

Do you know if it causes issues with openvz or not?

The change log had no explination just said something like 'disable the by default'

That is all the info my googlefu was able to find.

I don't remember exactly, but it was related to this

http://kb.parallels.com/en/113568
* Transparent hugepages support has been disabled by default. (PSBM-11811)
If necessary, you can enable it by running this command:
# echo always > /sys/kernel/mm/redhat_transparent_hugepage/enabled

But I can't found anymore the reference of PSBM-11811.

I think it's was a problem for very specific case.
(I need to check the proxmox pve-devel mailing archive, we have discussed about this when the change has been done)

 

[e100]

Thanks for the hint to look in the pve-devel.

Looks like some discussion here:
http://pve.proxmox.com/pipermail/pve-devel/2012-April/002627.html

Maybe something about defrag? Not read the whole thread yet.

 

[e100]

Found it!
http://kb.parallels.com/en/111808

Also, Proxmox has a bug about this still open:
https://bugzilla.proxmox.com/show_bug.cgi?id=163

It says this issue was fixed, I wonder if the fix was them disabling THP?
Also, it seems like it was only an issue when the host gets low on free RAM, that is a situation I actively work to avoid anyway.