1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Meltdown and Spectre Linux Kernel fixes

Discussion in 'Proxmox VE: Installation and configuration' started by martin, Jan 7, 2018.

  1. martin

    martin Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    612
    Likes Received:
    235
    We just published new kernels for Proxmox VE 4.x and 5.x, addressing Meltdown and Spectre in the kernel.

    Please upgrade your Proxmox VE hosts via "apt update && apt dist-upgrade".

    Proxmox VE 5.x: pve-kernel (4.13.13-34)
    • cherry-pick / backport of KPTI / Meltdown fixes (from Ubuntu-4.13.0-23.25)
    • add Google Spectre PoC fix for KVM
    • fix objtool build regression
    -- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:19:58 +0100

    Proxmox VE 4.x: pve-kernel (4.4.98-102)
    • cherry-pick / backport of KPTI / Meltdown fix (based on Ubuntu-4.4.0-107.130)
    • add Google Spectre PoC fix for KVM
    -- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:15:19 +0100

    __________________
    Best regards,

    Martin Maurer
    Proxmox VE project leader
     
    JOduMonT, chrone, EuroDomenii and 5 others like this.
  2. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    you can verify whether KPTI is active on your system after rebooting by checking the kernel log ("dmesg" or "journalctl -b -k") for the following string:

    note that the current kernel buiilds do not (yet) contain the automatic disabling of KPTI on AMD systems (it will be included in the first round of follow-up updates).

    you can control whether KPTI is enabled by setting the kernel parameter "pti" (e.g., in /etc/default/grub or manually by editing the kernel command line in Grub when booting):

     
    chrone, EuroDomenii and danmac like this.
  3. Symbol

    Symbol Member
    Proxmox VE Subscriber

    Joined:
    Mar 1, 2017
    Messages:
    36
    Likes Received:
    1
    Fixes CVE-2017-5754 (variant #3/Meltdown).

    Fixes CVE-2017-5753 (variant #1/Spectre) ?

    I guess there will be a next-to-come fix for CVE-2017-5715 (variant #2/Spectre), associating microcode update and IBRS kernel patch, correct ?
     
  4. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    like the changelog says, it fixes the specific PoC described by Google's Project Zero (to access host memory via Spectre from a KVM guest). it does not fix Spectre in general (see below).

    yes (for both Spectre variants actually). (most of) those are currently
    • not yet publicly available (microcode updates)
    • not yet finalized upstream (IBRS/IBPB/.. and RETPOLINE patches)
    once they are, we will release Qemu and kernel updates accordingly (the microcode updates need to be provided by your system/mother board/CPU vendor, either via BIOS/UEFI updates or via intel-microcode/amd-microcode packages from Debian non-free).

    same goes for regression fixes - we are of course monitoring upstream developments in all those areas.

    note that Spectre-style exploits are much harder to pull off in practice, and need to be much more customized to the concrete binary / system under attack compared to Meltdown. Spectre (and new exploits based on similar mechanisms) and the associated counter measures will likely keep security researchers and kernel developers (of all operating systems) busy for the weeks and months (if not years) to come. KPTI will likely also see more changes to allow for more fine-grained control to gain back some of the performance lost in some use cases (e.g., allowing specific processes or cgroups to be exempt from KPTI because they already have all the privileges to dump any memory anyhow), but those have to wait for the dust to settle.

    if you experience issues with the new kernels, please include as much information as possible in order to narrow down possible culprits and find fixes fast:
    • CPU and mainboard including firmware and microcode versions
    • pveversion -v
    • kernel stack traces / journal logs / error messages
    • clear description of the issue and how it is triggered
     
    gkovacs and chrone like this.
  5. mir

    mir Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,417
    Likes Received:
    87
    I am running with AMD Opteron processors so the newly released kernel are almost useless unless changing boot settings. This information might have come handy in the release notes! Now I will have to rescheduled a new reboot :-(
     
  6. morph027

    morph027 Active Member

    Joined:
    Mar 22, 2013
    Messages:
    367
    Likes Received:
    42
    Awesome, thank you, we highly appreciate the overtime you presumably had in last days ;)
     
  7. stef1777

    stef1777 Member

    Joined:
    Jan 31, 2010
    Messages:
    176
    Likes Received:
    8
    Hello!

    Be carefull, I just tried on one of our server with Proxmox VE 4.x with pve-kernel (4.4.98-102) on an HP DL120 G7 and the server crash at boot.

    I got this just after Grub selection and screen refresh.

    kernel_crash_DL120G7.png

    Reverted back to 4.4.98-101 with Grub, it work.

    I feel that this story will cause us a lot of trouble.

    Sincerely,
     
    #7 stef1777, Jan 8, 2018
    Last edited: Jan 8, 2018
    chrone, Andrei ZeeGiant and Xabi like this.
  8. anigwei

    anigwei Member

    Joined:
    Nov 25, 2015
    Messages:
    33
    Likes Received:
    1
    I've exactly the same issue on an HP Proliant server! Warning :eek::eek:
     
  9. markusn

    markusn New Member

    Joined:
    Jan 31, 2016
    Messages:
    1
    Likes Received:
    0
    When will the patch be available in the No-Subscription Repository? The latest kernel available to me is "pve-kernel-4.4.98-3-pve". Please, don't forget us home-users, or offer a more affordable access the the enterprise repository for home users.
     
  10. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    that one is the right kernel. the patched kernels are available in all repositories for both PVE 4 and PVE 5.

    the kernel packages have two versions - one is the "ABI", which is 4.4.98-3-pve and part of the package name, and doesn't have to change for every update. the other is the package version, which is 4.4.98-102, and changes with every update.
     
  11. anigwei

    anigwei Member

    Joined:
    Nov 25, 2015
    Messages:
    33
    Likes Received:
    1
    Hi,

    True: In pve-no-subscription you have pve-kernel-4.4.98-3-pve which is 4.4.98-102 (patched one).

    Regards,
     
  12. morph027

    morph027 Active Member

    Joined:
    Mar 22, 2013
    Messages:
    367
    Likes Received:
    42
    Updated 2 serves (my old lab HP MicroServer G8 plus one recent Skylake) wihtou any boot problems. Will keep an eye on the monitoring ;)

    Jip...the whole it world is having a "funny" new year 2018 i guess...
     
    anigwei likes this.
  13. speedbird

    speedbird Member
    Proxmox VE Subscriber

    Joined:
    Nov 3, 2017
    Messages:
    45
    Likes Received:
    2
    Crashed my home testing server – well, prevented it from booting to be more precise. Guess I'll wait for some more testing before rolling it out in productive
     
  14. pmoxuser7

    pmoxuser7 New Member

    Joined:
    Apr 13, 2017
    Messages:
    2
    Likes Received:
    0
    Can you please check if the patch is correctly applied to 4.4.98-102?

    On 4.13.13-34 i see a correctly applied patch:

    > uname -v
    #1 SMP PVE 4.13.13-34 (Sun, 7 Jan 2018 13:19:58 +0100)

    > cat /proc/cpuinfo | grep bugs | uniq -c
    32 bugs : cpu_insecure


    On 4.4.98-102 i do not see the bug in /proc/cpuinfo

    > uname -v
    #1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100)

    > cat /proc/cpuinfo | grep bugs | uniq -c
    32 bugs :


    Edit: I can boot both kernels without any issues. It just seems to me that the workaround is not applied on 4.4.98-102? Both Servers share the same CPU.
     
  15. stef1777

    stef1777 Member

    Joined:
    Jan 31, 2010
    Messages:
    176
    Likes Received:
    8
    Tried on an old HP DL360-G5 with Proxmox VE 4.x with pve-kernel 4.4.98-102. Boot fine.
     
  16. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    I found a possible culprit, currently building an updated kernel.
     
    GadgetPig and chrone like this.
  17. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    the patch sets for 4.13 and 4.4 are very very different, including how config options and flags are called internally, and what is exposed where. to check whether KPTI is on, check the kernel log like I described in my first post in this thread.
     
  18. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    2,913
    Likes Received:
    445
    likely unrelated, please open a new thread if the problem persists
     
  19. mouk

    mouk Member

    Joined:
    May 3, 2016
    Messages:
    30
    Likes Received:
    0
    yes, some more attempts at apt dist-upgrade got this resolved. Not sure why this happens.Second host finished just fine.... Thanks Fabian, and sorry for the noise.
     
    #19 mouk, Jan 8, 2018
    Last edited: Jan 8, 2018
  20. Sakis

    Sakis Member
    Proxmox VE Subscriber

    Joined:
    Aug 14, 2013
    Messages:
    117
    Likes Received:
    2
    It would be helpfull to post some cpu usage metrics here or in another threadm to check the performance impact in various CPUs.

    Updating all of our clusters, migrating vms and rebooting nodes its a quite time hungry procedure. Do we have an estimation for next kernel? Maybe we can wait if its 2-3 days max.
     

Share This Page