How to enable 1G hugepages?

H

huggy3233

New Member
May 29, 2023
10
0
1
I've been trying to enable 1G hugepages on my proxmox server (R7 5700X, 48GB RAM, Proxmox 7.4, Kernel: 6.2.11-2-pve) but haven't been able to get it to work. I am aiming to allocate 16 1Gibibyte hugepages to a vm for a total of 16 Gibibytes. So far I have added: hugepagesz=1G hugepages=16 default_hugepagesz=1G to my grub boot parameters, and added: hugepages: 1024 to my vms config. When running hugeadm --explain it appears that the pages have been created successfully. But when I run this command: grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' to check which processes are using hugepages my vm is still using 2M pages.

What am I doing wrong here? or is that check command not the proper way to check hugepage allocation?
 
huggy3233 said:
grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' to check which processes are using hugepages my vm is still using 2M pages.
Click to expand...
You grep for anonymous hugepages, If you configure hugepages, they are not anonymous anymore.
 
  • Like
Reactions: gseeley
What is the difference between anonymous and normal hugepages?
How do I check for normal hugepage usage?
 
huggy3233 said:
What is the difference between anonymous and normal hugepages?
Click to expand...
normal hugepages are allocated manually and anonymous hugepages are allocated in the background and on demand. They are a little bit unpredictable and often people advice in deactivating them completely. Once allocated, they cannot be used by normal processes and only a fraction of all programs can use hugepages. One of them is KVM, so therefore it may be an advantage of using them.

I personally work a lot with hugepages, yet with databases, not especially with VMs. Hugepages are often only allocatable before the memory is filled and fragmented, so you may require a reboot on online/live hugepage increase.
huggy3233 said:
How do I check for normal hugepage usage?
Click to expand...
Normalle via /proc/meminfo (general) or via smap similar as you did:

Code: 
 grep -B 11 -E 'KernelPageSize:\s+2048 kB' /proc/$PID/smaps | grep "^Size:" | awk 'BEGIN{sum=0}{sum+=$2}END{print sum/2048}'
 
  • Like
Reactions: gseeley
/proc/meminfo shows that I have 16 1Gib hugepages, in my vms .conf file I have added hugepages: 1024.
 
That flag is enabled, I think the problem here is that I am unsure how to check if hugepages are working properly.
 
There are no free hugepages so I guess they are being used, if I restart the vm will the hugepages still work?
 
Changing the command to the correct sizes returns 16, which I assume is the correct result. Hugepages are also working after a restart. Thanks for your help.
 
huggy3233 said:
Changing the command to the correct sizes returns 16, which I assume is the correct result. Hugepages are also working after a restart. Thanks for your help.
Click to expand...
In my case, I configure on Proxmox server (v8.2.7) (two CPU Xeon Gold 6152, 384GB memory, 3.4TB ssd) GRUB_CMDLINE_LINUX_DEFAULT="quite default_hugepagesz=2M hugepages=165888", and on each VM I set hugepages: 2, memory: 32768. When I restart Proxmox and first bulk start 10 VM, it's ok. But I stop and start any 1 VM I get error "start failed: hugepage allocation failed at /usr/share/perl5/PVE/QemuServer/Memory.pm line 663". /proc/meminfo => HugePages_Free: 18432 still enough for 1 VM. I don't know why. Do you have any idea?
 
Last edited:
jaynguyen said:
In my case, I configure on Proxmox server (v8.2.7) (two CPU Xeon Gold 6152, 384GB memory, 3.4TB ssd) GRUB_CMDLINE_LINUX_DEFAULT="quite default_hugepagesz=2M hugepages=165888", and on each VM I set hugepages: 2, memory: 32768. When I restart Proxmox and first bulk start 10 VM, it's ok. But I stop and start any 1 VM I get error "start failed: hugepage allocation failed at /usr/share/perl5/PVE/QemuServer/Memory.pm line 663". /proc/meminfo => HugePages_Free: 18432 still enough for 1 VM. I don't know why. Do you have any idea?
Click to expand...
I have this with 1 VM only as am implementing hugepages at the moment, so try a clean built vm and see if the problem persists
 
Sorry to disturb, my problem is not about finetuning hugepages but creating them.
I've been configuring hugepages for a decade under debian.
1GB hugepages are required for big VMs and we have no time to spare in memory ballooning.

I had checked that the system supported 2MB hugepages:
Code: 
grep pse /proc/cpuinfo | uniq
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512_vnni md_clear flush_l1d arch_capabilities

and even 1GB hugepages:
Code: 
grep pdpe1gb /proc/cpuinfo | uniq
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512_vnni md_clear flush_l1d arch_capabilities

both confirmed by:
Code: 
ls /sys/devices/system/node/node0/hugepages/
hugepages-1048576kB  hugepages-2048kB

My 1st reflex was to configure hugepages the same way under ProxMox, creating directory, group, adding config in /etc/fstab...
No available hugepages appeared after reboot.

Just in case, I then restarted on a brand new installation of ProxMox,
used the simple (simplistic?) recipe repeated on various forum and tuto:
edited /etc/default/grub to have those 2 kernel parameters configured as follow:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet default_hugepagesz=1G hugepagesz=1G hugepages=16"
GRUB_CMDLINE_LINUX="quiet default_hugepagesz=1G hugepagesz=1G hugepages=16"

Recipe said to just run
Code: 
update-grub
but I had a doubt about the need to rebuilt initramfs, so I instead ran this:
Code: 
update-initramfs -u -k all 
update-grub

After reboot still no available hugepages in /proc/meminfo:
Code: 
...

HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0

...

dmesg says:
Code: 
dmesg | grep -i huge
[    0.339494] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.339497] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.339500] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.339502] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    8.768945] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    8.824747] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.

journal says this:
Code: 
journalctl --since "6 hour ago" | grep -i huge
Jul 30 22:33:12 proxmox5820 kernel: HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
Jul 30 22:33:12 proxmox5820 kernel: HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
Jul 30 22:33:12 proxmox5820 kernel: HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
Jul 30 22:33:12 proxmox5820 kernel: HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
Jul 30 22:33:12 proxmox5820 systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
Jul 30 22:33:12 proxmox5820 systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Jul 30 22:33:18 proxmox5820 lxcfs[1202]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jul 30 22:39:05 proxmox5820 kernel: HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
Jul 30 22:39:05 proxmox5820 kernel: HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
Jul 30 22:39:05 proxmox5820 kernel: HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
Jul 30 22:39:05 proxmox5820 kernel: HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
Jul 30 22:39:05 proxmox5820 systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
Jul 30 22:39:05 proxmox5820 systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Jul 30 22:39:11 proxmox5820 lxcfs[1223]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jul 30 22:48:21 proxmox5820 kernel: HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
Jul 30 22:48:21 proxmox5820 kernel: HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
Jul 30 22:48:21 proxmox5820 kernel: HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
Jul 30 22:48:21 proxmox5820 kernel: HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
Jul 30 22:48:21 proxmox5820 systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
Jul 30 22:48:21 proxmox5820 systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Jul 30 22:48:27 proxmox5820 lxcfs[1184]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jul 30 23:04:36 proxmox5820 kernel: HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
Jul 30 23:04:36 proxmox5820 kernel: HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
Jul 30 23:04:36 proxmox5820 kernel: HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
Jul 30 23:04:36 proxmox5820 kernel: HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
Jul 30 23:04:36 proxmox5820 systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
Jul 30 23:04:36 proxmox5820 systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Jul 30 23:04:42 proxmox5820 lxcfs[1211]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc

then repeated the whole process with variants like
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=25"
with various combinations of the additional parameters
  • intel_iommu=on
  • iommu=pt
  • transparent_hugepage=never
What is the magic ingredient missing?
Things look like I'm putting settings at one place and ProxMox is using parameters from elsewhere!
There is a prerequisite or package missing?

Using ProxMox 8.4
on a Dell Precision T5820x
cpu: Intel Core i9-10940X CPU @ 3.30GHz
RAM: 64GB
System installed on ZFS(Raid1) with 2TB disks
 
Hmm ... that is VERY strange, I just confirmed in a VM in PVE 8.4 that the steps you mentioned just works. As you said, the configuration is as it always has been for decades:

Code: 
root@proxmox8-ext4 ~ > grep -i huge /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet default_hugepagesz=1G hugepagesz=1G hugepages=4"

root@proxmox8-ext4 ~ > update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.12-2-pve
Found initrd image: /boot/initrd.img-6.8.12-2-pve
Found memtest86+x64 image: /boot/memtest86+x64.bin
done

root@proxmox8-ext4 ~ > reboot

wait and check:

Code: 
root@proxmox8-ext4 ~ > grep -i huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       4
HugePages_Free:        4
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:         4194304 kB

root@proxmox8-ext4 ~ > dmesg | grep -i huge
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet default_hugepagesz=1G hugepagesz=1G hugepages=4
[    0.090804] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet default_hugepagesz=1G hugepagesz=1G hugepages=4
[    0.441986] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc hugepage)
[    0.445519] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
[    0.503207] HugeTLB: registered 1.00 GiB page size, pre-allocated 4 pages
[    0.503212] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.503221] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.503222] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    4.923086] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    4.954423] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.

Are you sure the options are actually used? Have you checked /proc/cmdline?
Are you per change using UEFI-boot with systemd? If this is true, you need to change another file /etc/kernel/cmdline.
 
Essentially exploring, I made a very simple installation, 1 raid1 set with ZFS and everything on it.
Some sources say there might be a compatibility problem using hugepages with root on a zfs filesystem.
But those reports mention symptoms are a failure to boot!
So, I was going to ask if, in your experience, you had a more sophisticated setup with root isolated on another filesystem?

But if I finally make a good boy of myself and read your comment to the end...
BINGO! I now get the 16 hugepages of 1GB I had configured.
Thank You!!!

I had seen no forum, post... mentioning the specificity of UEFI-boot.
Considering this deficiency, let me summarize for posterity.

My settings in /etc/default/grub:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16"
GRUB_CMDLINE_LINUX="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages16"
were useless because my boot method is not grub but UEFI.

I must know that /proc/cmdline is read-only, just an echo of parameters used to start current session.
As you recommended, I checked in it and realized my current session had not used settings made in /etc/default/grub:
Code: 
initrd=\EFI\proxmox\6.8.12-9-pve\initrd.img-6.8.12-9-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

I then edited /etc/[U]kernel[/U]/cmdline (the editable template for /proc/cmdline) that initially had this content:
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs
and instead made it this:
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16

I applied those changes with:
Code: 
proxmox-boot-tool refresh
and rebooted.
I now have the hugepages I configured.

Thanks again!!!
 
Last edited:
You must log in or register to reply here.
Top Bottom