Meltdown and Spectre Linux Kernel fixes

[martin]

We just published new kernels for Proxmox VE 4.x and 5.x, addressing Meltdown and Spectre in the kernel.

Please upgrade your Proxmox VE hosts via "apt update && apt dist-upgrade".

Proxmox VE 5.x: pve-kernel (4.13.13-34)

• cherry-pick / backport of KPTI / Meltdown fixes (from Ubuntu-4.13.0-23.25)
• add Google Spectre PoC fix for KVM
• fix objtool build regression

-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:19:58 +0100

Proxmox VE 4.x: pve-kernel (4.4.98-102)

• cherry-pick / backport of KPTI / Meltdown fix (based on Ubuntu-4.4.0-107.130)
• add Google Spectre PoC fix for KVM

-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:15:19 +0100

__________________
Best regards,

Martin Maurer
Proxmox VE project leader

 

[fabian]

you can verify whether KPTI is active on your system after rebooting by checking the kernel log ("dmesg" or "journalctl -b -k") for the following string:

kernel: Kernel/User page tables isolation: enabled

note that the current kernel buiilds do not (yet) contain the automatic disabling of KPTI on AMD systems (it will be included in the first round of follow-up updates).

you can control whether KPTI is enabled by setting the kernel parameter "pti" (e.g., in /etc/default/grub or manually by editing the kernel command line in Grub when booting):

pti= [X86_64] Control user/kernel address space isolation:
on - enable
off - disable
auto - default setting

 

[Symbol]

cherry-pick / backport of KPTI / Meltdown fixes (from Ubuntu-4.13.0-23.25)

Fixes CVE-2017-5754 (variant #3/Meltdown).

add Google Spectre PoC fix for KVM

Fixes CVE-2017-5753 (variant #1/Spectre) ?

I guess there will be a next-to-come fix for CVE-2017-5715 (variant #2/Spectre), associating microcode update and IBRS kernel patch, correct ?

 

[fabian]

Fixes CVE-2017-5753 (variant #1/Spectre) ?

like the changelog says, it fixes the specific PoC described by Google's Project Zero (to access host memory via Spectre from a KVM guest). it does not fix Spectre in general (see below).

I guess there will be a next-to-come fix for CVE-2017-5715 (variant #2/Spectre), associating microcode update and IBRS kernel patch, correct ?

yes (for both Spectre variants actually). (most of) those are currently

• not yet publicly available (microcode updates)
• not yet finalized upstream (IBRS/IBPB/.. and RETPOLINE patches)

once they are, we will release Qemu and kernel updates accordingly (the microcode updates need to be provided by your system/mother board/CPU vendor, either via BIOS/UEFI updates or via intel-microcode/amd-microcode packages from Debian non-free).

same goes for regression fixes - we are of course monitoring upstream developments in all those areas.

note that Spectre-style exploits are much harder to pull off in practice, and need to be much more customized to the concrete binary / system under attack compared to Meltdown. Spectre (and new exploits based on similar mechanisms) and the associated counter measures will likely keep security researchers and kernel developers (of all operating systems) busy for the weeks and months (if not years) to come. KPTI will likely also see more changes to allow for more fine-grained control to gain back some of the performance lost in some use cases (e.g., allowing specific processes or cgroups to be exempt from KPTI because they already have all the privileges to dump any memory anyhow), but those have to wait for the dust to settle.

if you experience issues with the new kernels, please include as much information as possible in order to narrow down possible culprits and find fixes fast:

• CPU and mainboard including firmware and microcode versions
• pveversion -v
• kernel stack traces / journal logs / error messages
  
• clear description of the issue and how it is triggered

 

[mir]

you can verify whether KPTI is active on your system after rebooting by checking the kernel log ("dmesg" or "journalctl -b -k") for the following string:



note that the current kernel buiilds do not (yet) contain the automatic disabling of KPTI on AMD systems (it will be included in the first round of follow-up updates).

you can control whether KPTI is enabled by setting the kernel parameter "pti" (e.g., in /etc/default/grub or manually by editing the kernel command line in Grub when booting):

I am running with AMD Opteron processors so the newly released kernel are almost useless unless changing boot settings. This information might have come handy in the release notes! Now I will have to rescheduled a new reboot :-(

 

[morph027]

Awesome, thank you, we highly appreciate the overtime you presumably had in last days

 

[stef1777]

Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.



Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,

 

[anigwei]

Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.

View attachment 6615

Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,

I've exactly the same issue on an HP Proliant server! Warning

 

[markusn]

When will the patch be available in the No-Subscription Repository? The latest kernel available to me is "pve-kernel-4.4.98-3-pve". Please, don't forget us home-users, or offer a more affordable access the the enterprise repository for home users.

 

[fabian]

When will the patch be available in the No-Subscription Repository? The latest kernel available to me is "pve-kernel-4.4.98-3-pve". Please, don't forget us home-users, or offer a more affordable access the the enterprise repository for home users.

that one is the right kernel. the patched kernels are available in all repositories for both PVE 4 and PVE 5.

the kernel packages have two versions - one is the "ABI", which is 4.4.98-3-pve and part of the package name, and doesn't have to change for every update. the other is the package version, which is 4.4.98-102, and changes with every update.

 

[anigwei]

Hi,

True: In pve-no-subscription you have pve-kernel-4.4.98-3-pve which is 4.4.98-102 (patched one).

Regards,

 

[morph027]

Updated 2 serves (my old lab HP MicroServer G8 plus one recent Skylake) wihtou any boot problems. Will keep an eye on the monitoring

I feel that this story will cause us a lot of trouble.

Jip...the whole it world is having a "funny" new year 2018 i guess...

 

[speedbird]

Crashed my home testing server – well, prevented it from booting to be more precise. Guess I'll wait for some more testing before rolling it out in productive

 

[pmoxuser7]

Can you please check if the patch is correctly applied to 4.4.98-102?

On 4.13.13-34 i see a correctly applied patch:

> uname -v
#1 SMP PVE 4.13.13-34 (Sun, 7 Jan 2018 13:19:58 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs : cpu_insecure

On 4.4.98-102 i do not see the bug in /proc/cpuinfo

> uname -v
#1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs :

Edit: I can boot both kernels without any issues. It just seems to me that the workaround is not applied on 4.4.98-102? Both Servers share the same CPU.

 

[stef1777]

Tried on an old HP DL360-G5 with Proxmox VE 4.x with pve-kernel 4.4.98-102. Boot fine.

 

[fabian]

Hello!

Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

I got this just after Grub selection and screen refresh.

View attachment 6615

Reverted back to 4.4.98-101 with Grub, it work.

I feel that this story will cause us a lot of trouble.

Sincerely,

I found a possible culprit, currently building an updated kernel.

 

[fabian]

Can you please check if the patch is correctly applied to 4.4.98-102?

On 4.13.13-34 i see a correctly applied patch:

> uname -v
#1 SMP PVE 4.13.13-34 (Sun, 7 Jan 2018 13:19:58 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs : cpu_insecure

On 4.4.98-102 i do not see the bug in /proc/cpuinfo

> uname -v
#1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100)

> cat /proc/cpuinfo | grep bugs | uniq -c
32 bugs :

Edit: I can boot both kernels without any issues. It just seems to me that the workaround is not applied on 4.4.98-102? Both Servers share the same CPU.

the patch sets for 4.13 and 4.4 are very very different, including how config options and flags are called internally, and what is exposed where. to check whether KPTI is on, check the kernel log like I described in my first post in this thread.

 

[fabian]

We just tried installing these updates, but the apt process is stuck at 99%, "Setting up pve-manager".
We're on the community repo. Is there a problem currently with that pve-manager..?

likely unrelated, please open a new thread if the problem persists

 

[mouk]

yes, some more attempts at apt dist-upgrade got this resolved. Not sure why this happens.Second host finished just fine.... Thanks Fabian, and sorry for the noise.

 

[Sakis]

It would be helpfull to post some cpu usage metrics here or in another threadm to check the performance impact in various CPUs.

Updating all of our clusters, migrating vms and rebooting nodes its a quite time hungry procedure. Do we have an estimation for next kernel? Maybe we can wait if its 2-3 days max.