PVE 7.2 w/ Kernel 5.15 fails to boot past 'Loading initial ramdisk...'

Maurice

New Member
Mar 11, 2021
6
1
3
Hi all,

After apt update / upgrade my kernel went from 5.11.22-7-pve to 5.15.64-1-pve but now gets stuck at 'Loading initial ramdisk...'
Adding the nomodeset makes no difference, still no messages are shown.
Starting PVE with the 5.11.22-7-pve kernel works without issues.

I run PVE on:
- Supermicro SuperServer A2SDi-16C-HLN4F
- 16 Core Intel Atom C3955 (Denverton) processor
- 64GB DDR4 ECC memory
- 1TB Samsung 980 PRO (with Heatsink) M.2 NVMe SSD.

Any suggestions?
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
2,140
173
68
29
Vienna
shop.maurer-it.com
Hello,

The upgrade process went ok, i.e. did not appear errors during the upgrade process?

May I ask you what exactly command you used for the upgrade? It's an apt update && apt upgrade or apt update && dist-upgrade?

EDIT: Can you also provide us with the output of pveversion -v
 

nirokato

Member
Mar 1, 2014
8
3
23
Hi Moayad,
I have the same issue on a A2SDi-8C-HLN4F. Did apt update followed by apt dist-upgrade:

Here is the output you're looking for:
Code:
proxmox-ve: 7.2-1 (running kernel: 5.11.22-7-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
Code:
   93  apt update
   94  apt dist-upgrade
   95  reboot
   96  pveversion -v
   97  history
Code:
root@pve:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
2E85-8F0E is configured with: uefi (versions: 5.11.22-4-pve, 5.11.22-7-pve, 5.15.64-1-pve)
2E86-0B2B is configured with: uefi (versions: 5.11.22-4-pve, 5.11.22-7-pve, 5.15.64-1-pve)
Code:
root@pve:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.11.22-7-pve\initrd.img-5.11.22-7-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
 
Last edited:

nirokato

Member
Mar 1, 2014
8
3
23
For some basic troubleshooting I added "acpi=off" to /etc/kernel/cmdline as below
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs acpi=off

Refreshed with proxmox-boot-tool refresh and rebooted.

Now kernel 5.15.64 boots, but but no ACPI is unacceptable. What are my next steps?

Code:
root@pve:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs acpi=off

Code:
root@pve:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
2E85-8F0E is configured with: uefi (versions: 5.11.22-7-pve, 5.15.64-1-pve)
2E86-0B2B is configured with: uefi (versions: 5.11.22-7-pve, 5.15.64-1-pve)

Code:
root@pve:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
 

nirokato

Member
Mar 1, 2014
8
3
23
Hello,

The upgrade process went ok, i.e. did not appear errors during the upgrade process?

May I ask you what exactly command you used for the upgrade? It's an apt update && apt upgrade or apt update && dist-upgrade?

EDIT: Can you also provide us with the output of pveversion -v

Please let me know what troubleshooting steps are next, thank you!
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
2,140
173
68
29
Vienna
shop.maurer-it.com
Hello,

What does the bellow commands saying:

Bash:
update-initramfs -k all -u
proxmox-boot-tool refresh
If no error resulting from the above commands, try to reboot the server afterward.
 

RolandK

Active Member
Mar 5, 2019
414
70
33
49
>Adding the nomodeset makes no difference, still no messages are shown.

i cannot remember that i have seen a linux system crash/hang on boot at that stage

did you remove the "quiet" param from the grub commandline ?
 

nirokato

Member
Mar 1, 2014
8
3
23
Hello,

What does the bellow commands saying:

Bash:
update-initramfs -k all -u
proxmox-boot-tool refresh
If no error resulting from the above commands, try to reboot the server afterward.

Thank you for the quick reply. Sorry for the delay, I've been flying between destinations and haven't had the best connectivity.

Code:
root@pve:~# update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-5.15.64-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2E85-8F0E
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/2E86-0B2B
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
update-initramfs: Generating /boot/initrd.img-5.11.22-7-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2E85-8F0E
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/2E86-0B2B
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
update-initramfs: Generating /boot/initrd.img-5.11.22-4-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2E85-8F0E
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/2E86-0B2B
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
root@pve:~# proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2E85-8F0E
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/2E86-0B2B
        Copying kernel and creating boot-entry for 5.11.22-7-pve
        Copying kernel and creating boot-entry for 5.15.64-1-pve
root@pve:~#

Screenshot attached, no success after reboot without "acpi=off"
 

Attachments

  • Screenshot_20221113_110853.png
    Screenshot_20221113_110853.png
    10.7 KB · Views: 12

nirokato

Member
Mar 1, 2014
8
3
23
>did you remove the "quiet" param from the grub commandline ?
Yep, this system uses UEFI Sytemd-boot and ZFS. Below, I included the /proc/cmdline I pasted from earlier. I also tried with "no quiet". There is no additional output after the screenshot from my previous post.

Code:
root@pve:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs acpi=off
 

nirokato

Member
Mar 1, 2014
8
3
23
Thanks! I am already reading through this thread (page 5 ish now?). Edit: Sorry, I'm on mobile, it was this thread...https://forum.proxmox.com/threads/o...15-for-proxmox-ve-7-x-available.100936/page-6

I'm at another airport right now, tomorrow I plan to test simplefb and nofb kernel options and also any newer kernels in the pve-test repo.

Also, I'm not doing anything with PCIe or GPU passthrough, but I did try with the following iommu options, but no dice.
Code:
intel_iommu=on
iommu=pt

I'll keep reading through that thread and see if inspiration strikes.

Any other suggestions are welcome! I'll add them to the pile for tomorrow's testing.
 
Last edited:

nirokato

Member
Mar 1, 2014
8
3
23
I had the same problem on my A2SDi-8C+-HLN4F after upgrading BIOS to the latest version 1.7. Boot process stucked at Loading initial ramdisk...

After downgrading BIOS to the previous version, PVE booted normally.

Thank you Radim, that definitely resolved the initrd boot issue!

I was able to successfully boot kernel 5.15.64-1-pve after downgrading to 1.6.

BIOS downgraded to 1.6 using A2SDICH1.A27.zip

Pointer link: https://www.supermicro.com/support/resources/getfile.php?SoftwareItemID=15374
Final link: https://www.supermicro.com/Bios/softfiles/15374/A2SDICH1.A27.zip
File: A2SDICH1.A27.zip
SHA256: 4232c1d5eaec4d7a3805496e6fe75781bee87af026378930fd98155a351f5bfa

Credit to Serve The Home forum user RolloZ170 for the BIOS file link above.

Here is the source for that post: https://forums.servethehome.com/ind...problems-after-bios-upgrade.37706/post-350548

I tried uploading the file to the forum here (in case Supermicro ever became unavailable), but the file was too large at 5923311 bytes.

I also tried repacking with tar/zstd (--ultra -22), but only got it down to 4952197 bytes, which was still too large.

Note: I'll try additional kernel options at a later point on BIOS 1.7 to see if anything resolves the issue on that version and report back.

Thank you all for the help!
 
  • Like
Reactions: Fisxko and Neobin

Maurice

New Member
Mar 11, 2021
6
1
3
Dear all,
I am sorry for not replying earlier, I have been on some business trips.

Thank you for this solution Nitokato and Radim, much appreciated!

Is this 1.6 BIOS that is posted here also suitable for my 16 core Atom C3955 motherboard A2SDI-16C-HLN4F ?
 

Maurice

New Member
Mar 11, 2021
6
1
3
This worked for me too. Thanks a million!

Does anyone know if the root cause being investigated by Proxmox?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!