Does Proxmox ship their own CPU microcode packages? For example: amd64-microcode or intel-microcode? Or is it necessary to add non-free-firmware to sources.list for PVE8/bookworm and newer, so we can get the latest microcode for CPUs?
Proxmox VE 8.0+ cpu microcode
- Thread starter Wolvez
- Start date
In my experience, you need to add
non-free-firmware to the existing Debian repository lines, and you need to install it manually.
Exactly, like leesteken wrote, you can add the
non-free-firmware and use that.We will evaluate if it's practicable and workable to upload the microcode packages to our repositories too in the future, as that would be compatible with having the
non-free-firmware repository added anyway.Edited for clarity.
Last edited:
Thanks for the reply. I do not regret paying the Proxmox team 100 Euros a year.
I figured out why I was confused. It was because of the pve-firmware package. Not sure what firmware is included in that package.
I decided to live life in the danger zone and pulled amd64-microcode from unstable:
/etc/apt/sources.list.d/unstable-firmware.list:
deb http://deb.debian.org/debian unstable non-free-firmwareDO NOT DO THIS if you don't know what you are doing.
I have my server on EPYC rome/ZEN 2 and another machine on an Intel 4770k. I'll update when/if things go wrong.
Sorry, I cannot, as I wanted to write the exact opposite. That was, us (re-)uploading the exact same microcode package would be fully compatible with having Debian's non-free-firmware repo active.
Sorry for the confusion, seems I'm ripe for either more coffee in the morning or a vacation
Debian migrated the amd64-microcode into bookworm and bullseye security repo overnight, so as long as you have that repo active either an upgrade, or a new installation should pull in the version with the fix for Zenbleed, namely: "3.20230719.1"
https://tracker.debian.org/news/144...202307191deb12u1-source-into-stable-security/
https://tracker.debian.org/news/144...deb11u1-source-amd64-into-oldstable-security/
Is there a trick, how to install the microcode correctly in Proxmox?
The installation of "3.20230719.1" runs through without errors. But the microcode is not used after a reboot.
The installation of "3.20230719.1" runs through without errors. But the microcode is not used after a reboot.
Code:
root@pve2:~# apt install amd64-microcode
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
amd64-microcode
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 116 kB of archives.
After this operation, 257 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security bookworm-security/non-free-firmware amd64 amd64-microcode amd64 3.20230719.1~deb12u1 [116 kB]
Fetched 116 kB in 0s (1,309 kB/s)
Selecting previously unselected package amd64-microcode.
(Reading database ... 62377 files and directories currently installed.)
Preparing to unpack .../amd64-microcode_3.20230719.1~deb12u1_amd64.deb ...
Unpacking amd64-microcode (3.20230719.1~deb12u1) ...
Setting up amd64-microcode (3.20230719.1~deb12u1) ...
update-initramfs: deferring update (trigger activated)
amd64-microcode: microcode will be updated at next boot
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.2.16-5-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP syncSorry, I cannot, as I wanted to write the exact opposite. That was, us (re-)uploading the exact same microcode package would be fully compatible with having Debian's non-free-firmware repo active.
Sorry for the confusion, seems I'm ripe for either more coffee in the morning or a vacation
Hehe, thank you.
How are you checking to see if the new firmware was being loaded at boot? I haven't done anything special to load microcode.
https://wiki.debian.org/Microcode
Checking the microcode version of your CPU:
Code:
dmesg | grep "microcode updated early to"
journalctl -b -k | grep "microcode updated early to"
zgrep "microcode updated early to" /var/log/kern.log*This is for my AMD EPYC ROME CPU:
Code:
root@prox:~# dmesg | grep "microcode updated early to"
root@prox:~# journalctl -b -k | grep "microcode updated early to"
Jul 25 15:08:07 prox kernel: microcode: microcode updated early to new patch_level=0x0830107a
root@prox:~# zgrep "microcode updated early to" /var/log/kern.log
2023-07-24T16:08:21.203298-07:00 prox kernel: [ 2.282659] microcode: microcode updated early to new patch_level=0x08301072
2023-07-25T15:08:15.502253-07:00 prox kernel: [ 2.268750] microcode: microcode updated early to new patch_level=0x0830107a
root@prox:~# zgrep "microcode updated early to" /var/log/kern.log*
/var/log/kern.log:2023-07-24T16:08:21.203298-07:00 prox kernel: [ 2.282659] microcode: microcode updated early to new patch_level=0x08301072
/var/log/kern.log:2023-07-25T15:08:15.502253-07:00 prox kernel: [ 2.268750] microcode: microcode updated early to new patch_level=0x0830107a
Last edited:
Sorry, I've never messed with these microcodes. Is this a safe way to install these?
Code:
deb http://deb.debian.org/debian bookworm main contrib non-free
deb http://security.debian.org/debian-security bookworm-security main contrib non-free
deb http://deb.debian.org/debian bookworm-updates main contrib non-freeOn this site https://wiki.debian.org/Microcode it only mentions Bullseye, which is Debian 11. I'm on Proxmox 8, though (after upgrading from 7 with kernel 5.17, but now my VMs in my N5105 mini PC are crashing again).
Last edited:
For Debin 12/Bookworm+ they moved firmware blobs to:
Code:
non-free-firmware
Code:
non-free
Code:
non-free-firmwareI'm having a weird issue getting the firmware loaded onto a system and I'm not sure if it's related or not.
I installed all updates on proxmox-ve 8.0.2 (pve-no-subscription repo), rebooted and I'm still seeing some warnings in my logs.
On other distributions (Alma9 specifically), I might manually trigger a firmware (even prior to a reboot) with the following, but it doesn't seem to work on proxmox-ve.
Is there something special about the PVE kernel that doesn't include microcode reloading? Or is the Debian microcode package somehow not recognizing the PVE kernels? Or is something else going on?
I'll dig around to see what I can find, but I'm kind of stuck here.
I installed all updates on proxmox-ve 8.0.2 (pve-no-subscription repo), rebooted and I'm still seeing some warnings in my logs.
Code:
root@pve2:~# journalctl -k --grep=microcode
Aug 14 16:08:09 pve2 kernel: Zenbleed: please update your microcode for the most optimal fix
Aug 14 16:08:09 pve2 kernel: microcode: CPU2: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU3: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU1: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU4: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU5: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU6: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU8: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU7: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU9: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU10: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU11: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU12: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU0: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU13: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU14: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: CPU15: patch_level=0x08701030
Aug 14 16:08:09 pve2 kernel: microcode: Microcode Update Driver: v2.2.
root@pve2:~# zgrep "microcode udpated" /var/log/kern.log*
root@pve2:~#On other distributions (Alma9 specifically), I might manually trigger a firmware (even prior to a reboot) with the following, but it doesn't seem to work on proxmox-ve.
Code:
root@pve2:~# echo 1 > /sys/devices/system/cpu/microcode/reload
-bash: /sys/devices/system/cpu/microcode/reload: Permission deniedIs there something special about the PVE kernel that doesn't include microcode reloading? Or is the Debian microcode package somehow not recognizing the PVE kernels? Or is something else going on?
I'll dig around to see what I can find, but I'm kind of stuck here.
Did you ever find a solution to this issue? I updated to the 6.5 kernel via the pvetest-repo and updated the amd64-microcode package from Debian-Sid manually. After reboot the microcode is reported as 0xa50000d, but should be 0xa50000f (Ryzen 5 5600G). I don't have any microcode-update related messages in dmesg, just like you.
The hook is called by update-initramfs...
The hook is called by update-initramfs...
The microcode not loading could be related to the boot loader in use. I'm on a ZFS root and Proxmox is using Systemd-boot. If your install is using GRUB you may need to change the grub config to load microcode updates. I'm not sure as I only have this one Proxmox install currently.
Last edited:
I haven't, I got busy with other tasks and I also figured "time may fix it".
We have a shiny new proxmox cluster I might be able to try things out on (that will have support on it). I'll report what happens, though it may be a few weeks in the future.
Rebooting the nodes got the new microcode into the Kernel. Or at least it seems to have...
Unfortunately it's looking like hot-patching CPUs isn't supported in Proxmox-VE. The Proxmox-VE kernel is compiled without CONFIG_MICROCODE_LATE_LOADING.
The short version is that microcode can be loaded into the CPU at two times:
This is my best understanding of the issue as of today. It's less than desirable, but I now respect and accept why PVE has it turned off.
Unfortunately it's looking like hot-patching CPUs isn't supported in Proxmox-VE. The Proxmox-VE kernel is compiled without CONFIG_MICROCODE_LATE_LOADING.
The short version is that microcode can be loaded into the CPU at two times:
- Early Loading: This happens in the boot process before the kernel hands over control to /bin/init and the microcode is stored in the initrd.
- Late Loading: This happens when the system is fully booted and is running real workloads.
This is my best understanding of the issue as of today. It's less than desirable, but I now respect and accept why PVE has it turned off.