Meltdown and Spectre Linux Kernel fixes

martin

Proxmox Staff Member
Staff member
Apr 28, 2005
644
372
63
We just published new kernels for Proxmox VE 4.x and 5.x, addressing Meltdown and Spectre in the kernel.

Please upgrade your Proxmox VE hosts via "apt update && apt dist-upgrade".

Proxmox VE 5.x: pve-kernel (4.13.13-34)
  • cherry-pick / backport of KPTI / Meltdown fixes (from Ubuntu-4.13.0-23.25)
  • add Google Spectre PoC fix for KVM
  • fix objtool build regression
-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:19:58 +0100

Proxmox VE 4.x: pve-kernel (4.4.98-102)
  • cherry-pick / backport of KPTI / Meltdown fix (based on Ubuntu-4.4.0-107.130)
  • add Google Spectre PoC fix for KVM
-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:15:19 +0100

__________________
Best regards,

Martin Maurer
Proxmox VE project leader
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
you can verify whether KPTI is active on your system after rebooting by checking the kernel log ("dmesg" or "journalctl -b -k") for the following string:

kernel: Kernel/User page tables isolation: enabled
note that the current kernel buiilds do not (yet) contain the automatic disabling of KPTI on AMD systems (it will be included in the first round of follow-up updates).

you can control whether KPTI is enabled by setting the kernel parameter "pti" (e.g., in /etc/default/grub or manually by editing the kernel command line in Grub when booting):

Documentation/admin-guide/kernel-parameters.txt said:
pti= [X86_64] Control user/kernel address space isolation:
on - enable
off - disable
auto - default setting
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
Fixes CVE-2017-5753 (variant #1/Spectre) ?
like the changelog says, it fixes the specific PoC described by Google's Project Zero (to access host memory via Spectre from a KVM guest). it does not fix Spectre in general (see below).

I guess there will be a next-to-come fix for CVE-2017-5715 (variant #2/Spectre), associating microcode update and IBRS kernel patch, correct ?
yes (for both Spectre variants actually). (most of) those are currently
  • not yet publicly available (microcode updates)
  • not yet finalized upstream (IBRS/IBPB/.. and RETPOLINE patches)
once they are, we will release Qemu and kernel updates accordingly (the microcode updates need to be provided by your system/mother board/CPU vendor, either via BIOS/UEFI updates or via intel-microcode/amd-microcode packages from Debian non-free).

same goes for regression fixes - we are of course monitoring upstream developments in all those areas.

note that Spectre-style exploits are much harder to pull off in practice, and need to be much more customized to the concrete binary / system under attack compared to Meltdown. Spectre (and new exploits based on similar mechanisms) and the associated counter measures will likely keep security researchers and kernel developers (of all operating systems) busy for the weeks and months (if not years) to come. KPTI will likely also see more changes to allow for more fine-grained control to gain back some of the performance lost in some use cases (e.g., allowing specific processes or cgroups to be exempt from KPTI because they already have all the privileges to dump any memory anyhow), but those have to wait for the dust to settle.

if you experience issues with the new kernels, please include as much information as possible in order to narrow down possible culprits and find fixes fast:
  • CPU and mainboard including firmware and microcode versions
  • pveversion -v
  • kernel stack traces / journal logs / error messages
  • clear description of the issue and how it is triggered
 
  • Like
Reactions: gkovacs and chrone

mir

Well-Known Member
Apr 14, 2012
3,489
97
48
Copenhagen, Denmark
you can verify whether KPTI is active on your system after rebooting by checking the kernel log ("dmesg" or "journalctl -b -k") for the following string:



note that the current kernel buiilds do not (yet) contain the automatic disabling of KPTI on AMD systems (it will be included in the first round of follow-up updates).

you can control whether KPTI is enabled by setting the kernel parameter "pti" (e.g., in /etc/default/grub or manually by editing the kernel command line in Grub when booting):
I am running with AMD Opteron processors so the newly released kernel are almost useless unless changing boot settings. This information might have come handy in the release notes! Now I will have to rescheduled a new reboot :-(
 

stef1777

Member
Jan 31, 2010
178
8
18
Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.

kernel_crash_DL120G7.png

Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,
 
Last edited:

anigwei

Member
Nov 25, 2015
33
1
8
Barcelona (Catalunya)
about.me
Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.

View attachment 6615

Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,
I've exactly the same issue on an HP Proliant server! Warning :eek::eek:
 

markusn

New Member
Jan 31, 2016
1
0
1
33
When will the patch be available in the No-Subscription Repository? The latest kernel available to me is "pve-kernel-4.4.98-3-pve". Please, don't forget us home-users, or offer a more affordable access the the enterprise repository for home users.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
When will the patch be available in the No-Subscription Repository? The latest kernel available to me is "pve-kernel-4.4.98-3-pve". Please, don't forget us home-users, or offer a more affordable access the the enterprise repository for home users.
that one is the right kernel. the patched kernels are available in all repositories for both PVE 4 and PVE 5.

the kernel packages have two versions - one is the "ABI", which is 4.4.98-3-pve and part of the package name, and doesn't have to change for every update. the other is the package version, which is 4.4.98-102, and changes with every update.
 
Nov 3, 2017
45
4
8
Crashed my home testing server – well, prevented it from booting to be more precise. Guess I'll wait for some more testing before rolling it out in productive
 

pmoxuser7

New Member
Apr 13, 2017
2
0
1
30
Can you please check if the patch is correctly applied to 4.4.98-102?

On 4.13.13-34 i see a correctly applied patch:

> uname -v
#1 SMP PVE 4.13.13-34 (Sun, 7 Jan 2018 13:19:58 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs : cpu_insecure


On 4.4.98-102 i do not see the bug in /proc/cpuinfo

> uname -v
#1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs :


Edit: I can boot both kernels without any issues. It just seems to me that the workaround is not applied on 4.4.98-102? Both Servers share the same CPU.
 

stef1777

Member
Jan 31, 2010
178
8
18
Tried on an old HP DL360-G5 with Proxmox VE 4.x with pve-kernel 4.4.98-102. Boot fine.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.

View attachment 6615

Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,
I found a possible culprit, currently building an updated kernel.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
Can you please check if the patch is correctly applied to 4.4.98-102?

On 4.13.13-34 i see a correctly applied patch:

> uname -v
#1 SMP PVE 4.13.13-34 (Sun, 7 Jan 2018 13:19:58 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs : cpu_insecure


On 4.4.98-102 i do not see the bug in /proc/cpuinfo

> uname -v
#1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs :


Edit: I can boot both kernels without any issues. It just seems to me that the workaround is not applied on 4.4.98-102? Both Servers share the same CPU.
the patch sets for 4.13 and 4.4 are very very different, including how config options and flags are called internally, and what is exposed where. to check whether KPTI is on, check the kernel log like I described in my first post in this thread.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
3,472
540
113
We just tried installing these updates, but the apt process is stuck at 99%, "Setting up pve-manager".
We're on the community repo. Is there a problem currently with that pve-manager..?
likely unrelated, please open a new thread if the problem persists
 

mouk

Member
May 3, 2016
30
0
6
47
yes, some more attempts at apt dist-upgrade got this resolved. Not sure why this happens.Second host finished just fine.... Thanks Fabian, and sorry for the noise.
 
Last edited:
Aug 14, 2013
121
3
18
It would be helpfull to post some cpu usage metrics here or in another threadm to check the performance impact in various CPUs.

Updating all of our clusters, migrating vms and rebooting nodes its a quite time hungry procedure. Do we have an estimation for next kernel? Maybe we can wait if its 2-3 days max.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!