Boot hangs after “Loading initial ramdisk” on kernels 7.0.14-24/25-pve with 2M hugepages on the cmdline (Dell, ~1 TB RAM). Kernel 6.17.13-21-pve boots

J

jeffblima

Member
Sep 12, 2022
1
0
15
After updating a non-production Proxmox VE node, the host hangs at boot right after:


initial-ramdisk.png


Bash: 
Loading Linux 7.0.14-24-pve ...
Loading initial ramdisk ...


There is no further output, even after waiting several minutes [confirm wait time: ___ min].


Environment

  • Hardware: Dell [model], BIOS [version], Intel CPU [model], [1 or 2] sockets
  • RAM: 1,031,426 MiB total (~1007 GiB)
  • Proxmox VE: pve-manager 9.2.21
  • Failing kernels: 7.0.14-24-pve and 7.0.14-25-pve
  • Working kernel: 6.17.13-21-pve
  • Root on LVM (/dev/mapper/pve-root), booting with GRUB
  • All VMs on this host use hugepages (no non-hugepage VMs)

Cmdline that hangs on the 7.0 kernels

Bash: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on default_hugepagesz=2M hugepagesz=2M hugepages=499712"

hugepages=499712 × 2 MiB = 999,424 MiB (~976 GiB), which leaves about 32 GiB for the host.


What I tested

  • Kernel 6.17.13-21-pve with the same cmdline (IOMMU + hugepages=499712): boots normally [confirm HugePages_Total = 499712].
  • Kernels 7.0.14-24-pve and 7.0.14-25-pve with the same cmdline: both hang after “Loading initial ramdisk”.
  • Kernel 7.0.14-25-pve without intel_iommu=on and without the hugepages parameters: boots.
  • Kernel 7.0.14-25-pve with intel_iommu=on only: boots. dmesg shows DMAR: IOMMU enabled and 287 entries under /sys/kernel/iommu_groups/, so IOMMU itself looks fine.
  • Pending: [result with hugetlb_free_vmemmap=on] / [result with hugepages=450000] / [result with loglevel=7 earlyprintk=vga,keep or Serial Over LAN].

My suspicion
Since 6.17.13-21 boots with the same cmdline and RAM, I suspect a regression or behavior change in boot-time hugepage allocation in the 7.0 series. It could also be extreme slowness allocating ~500k 2M pages that early in boot, but I haven’t been able to confirm that yet.


Questions

  1. Is there any known change in the 7.0 series (hugetlb, vmemmap, struct page init) affecting cmdline hugepage reservation on hosts with ~1 TB of RAM?
  2. Is allocating via vm.nr_hugepages in /etc/sysctl.d/ (after boot) the recommended approach in this scenario?
  3. Is there a recommended cmdline parameter for the 7.0 series in this case?

I can provide dmesg, journalctl -b, numactl -H, /proc/meminfo and the /boot/config-* settings if needed.
 
could you try with a smaller number of hugepages? e.g. 10000? removing `quiet` might also show more progress..
 
Hello,
I am actually battling with the exact same issue. In my case I do not have the hugepages set as option, I just wanted to share that here. I have went through a full hardware validation and after fixing few things I told were related, I am still unable to boot my server. I have moved since on the OS itself and I am at my 8 day trying to figure this out. I will share some feedback if I find anything, I just wanted to let you know that hugepages may not be related.
Thanks!
 
Hello,
Turns out my case was related to hardware issue, my motherboard seems to have an issue with the video output.
To be honest, I am not sure my case is similar to the case you have, in my case the machine was reseting, which it does not seems to be the case here.
Seems like we had somewhat similar behaviour which does not seems to be to be caused by the same thing.
Have a nice one!
 
You must log in or register to reply here.
Top Bottom
Back