Proxmox Upgrade 8.2 made my system not booting

[golruck]

Hi everyone,

First thread and newbie here.

I have just upgraded from 8.1 to 8.2 and when I restarted, my server got stuck. I logged to IPMI and saw the cursor blinking on y black window. I reset the server and could see that booting on the new kernel 6.8.4-3-pve was the cause. Booting from the same kernel in recovery mode works but I am not sure what is the impact on running the recovery mode all the time. I found several threads but pined the kernel 6.5.12-5pve so now it boot correctly.

Now the next step is to understand why it does not like the new kernel. I understand that it is better to run the latest one for security and compatibility but I am not sure how to debug this. My server is old but functioning so I don't really want to upgrade unless it is really necessary. It is a production server for my home usage. Only one node.

As I am not sure what do you need to help me so here is some information:







Main Hardware:
Main Board ASRock E3C224-4L latest firmware
CPU INTEL "Haswell", 4x 3.4GHz, Socket 1150, Boxed (BX80646E31240V3)
Memory KINGSTON ValueRAM KVR16E11K4/32

you will find the attached files results of few different command of my system (I reached the character limit)

I have also noticed this error at boot but it still boots, as there is no more firmware update on that motherboard, I assume I cannot do anything about it.


Thank you in advance for help and the great PVE and PBS products (I come from ESXi 6.0.0 and migrated a few month ago to Proxmox)!

 

[justinclift]

... as there is no more firmware update on that motherboard, I assume I cannot do anything about it.

You could install the intel-microcode package, which can be added to an existing Proxmox server pretty easily:

# apt -y install intel-microcode

 

[golruck]

Thank you for the tip.

I tried to run the command and it was not installing (I don't remember the command result) after I searched on Internet I found this which I ran and installed the 3.20240514.1 version. Then I rebooted (leaving the pinned kernel). I ran the same script to see if it now detects the installed microcode but to see if the microcode is applied but it states "Not found" while when I run the apt -y install intel-microcode I get intel-microcode is already the newest version (3.20240514.1):

root@pve:~# apt -y install intel-microcode
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
intel-microcode is already the newest version (3.20240514.1).
The following packages were automatically installed and are no longer required:
  proxmox-kernel-6.5.11-8-pve-signed proxmox-kernel-6.5.13-1-pve-signed
  proxmox-kernel-6.5.13-3-pve-signed
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@pve:~# dmesg | grep -i microcode
[    0.060042] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x22 (or later)
[    0.190759] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.190759] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.190761] SRBDS: Vulnerable: No microcode
[    0.880358] microcode: Microcode Update Driver: v2.2.

root@pve:~# grep -E 'family|model|stepping|microcode' /proc/cpuinfo | head -5
cpu family      : 6
model           : 60
model name      : Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz
stepping        : 3
microcode       : 0x9

Nothing when I run

dpkg -s intel-microcode | grep version

I am not sure how to install the microcode. Any obvious thing I missed?

Thanks a lot

 

[justinclift]

Oh. It looks like the intel-microcode package specifically stops itself from being automatically loaded at boot time!

# cat /etc/modprobe.d/intel-microcode-blacklist.conf
# The microcode module attempts to apply a microcode update when
# it autoloads.  This is not always safe, so we block it by default.
blacklist microcode

That file comes with the intel-microcode package itself:

# dpkg-query -S /etc/modprobe.d/intel-microcode-blacklist.conf
intel-microcode: /etc/modprobe.d/intel-microcode-blacklist.conf

Not sure if it gets loaded by some other process instead though, rather than at boot time.

Will need to investigate this properly later on.

 

[justinclift]

Looking at the README.Debian.gz file that comes with the package:

# zmore /usr/share/doc/intel-microcode/README.Debian.gz

It seems like it's added to the early bootup sequence (initramfs) instead.

On my test systems it seems to be applied ok:

# grep -E 'family|model|stepping|microcode' /proc/cpuinfo | head -5
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
stepping        : 4
microcode       : 0x42e

and:

# dmesg | grep -i microcode
[    2.057786] microcode: Current revision: 0x0000042e

 

[justinclift]

Looking at the file in the intel-microcode package, my cpus seem to have a matching microcode file under /lib/firmware/intel-ucode/.

Selecting the info from the /proc/cpuinfo output:

cpu family      : 6
model           : 62   -> converting that to hex gives "3E"
stepping        : 4

Then using those values to look for matching entries in the package:

# dpkg-query -L intel-microcode | grep -i "06-3e-04"
/lib/firmware/intel-ucode/06-3e-04

Yours seems to have an "initramfs" file instead:

# dpkg-query -L intel-microcode | grep -i "06-3c-03"
/lib/firmware/intel-ucode/06-3c-03.initramfs

I'm not sure how that works, though the intel-microcode package does include a Debian specific README with useful info:

# zmore /usr/share/doc/intel-microcode/README.Debian.gz
...
Using Debian to apply microcode updates:

Debian can apply microcode updates to the system processors during the
operating system boot when a correctly configured Linux kernel (such as
the standard Debian Linux kernels), and a small set of extra packages from
"non-free" and "contrib" are installed.

You must have "contrib" and "non-free" repositories enabled in apt's
sources list (either in /etc/apt/sources.list, or in a file inside
/etc/apt/sources.list.d/).

On a default Debian system (which uses a Debian kernel, the grub
bootloader, and initramfs-tools to create the initramfs for the kernel),
install the "intel-microcode" package and its dependencies, and reboot.
...

There's further info in that file. Maybe read over that and do some further experimenting?

 

[golruck]

Thank you for the analyse and help with this!

I am a bit clueless. I googled a bit and could see posts mentioning my cpu 06-3c-03 but I am not sure how I can really go further. I have one node and it is my home production server. I don't have a spare one to play with so I am a bit anxious to play too much with this especially that I am an linux newbie.

I wonder if it is also linked the the microcode but I see that the hyper-threading enabled option in the BIOS is not recognized in Proxmox. I am not sure how much I loose but I wanted to mention it.

Would you know where I could seek for further help? Another Debian forum maybe? As this is maybe not link to Proxmox itself but a more generally issue with the new kernel?

Thank you for your thought.

 

[justinclift]

I have one node and it is my home production server. I don't have a spare one to play ...

Ahhh, yeah. For quite a lot of practice stuff you can install Proxmox as a virtual machine inside your production node, then do creative experimentation on that practice VM without hurting other stuff.

That doesn't work real well when you need to do things with the cpu itself though.

Would you know where I could seek for further help?

Debian seems to have official "Debian User Forums", which looks potentially suitable:

https://forums.debian.net

Maybe give that a go?

 

[golruck]

Hi again,

I opened the following ticket to Debian forum and basically they think it is a Proxmox issue and the microcode may not be the culprit but something else.

https://forums.debian.net/viewtopic.php?t=159313

Would it be a problem if I continue to run the the Kernel 6.5.12-5-pve instead or the new one? Is it still supported? Can I still upgrade with the next coming updates?

Shall I re-install the host from scratch or simply wait for a new kernel update? I see some other posts having some issues too so there will be maybe a fix in the next update?

Sorry for the many questions and for now it is running so I am not in a hurry.

Thank you in advance

 

[justinclift]

For your situation I'd probably just drop back to the 6.5.x kernel for now (using the method in the Promox 8.2 release notes), at least until the Proxmox team fixes the problems people are having with the newer kernel series.

You won't be alone in doing that either.

 

[theyost]

First had this problem about a month ago with my last update. Reverting to 6.5.13-5-pve resolved the issue.

Ran another update this evening hoping the bug had been resolved. Unfortunately it ha not (at least for me). Reverting to 6.5.13-5-pve again resolved the issue for me.