Meltdown and Spectre Linux Kernel fixes

sommarnatt

Member
Mar 20, 2014
22
0
21
Sweden
https://www.qemu.org/2018/01/04/spectre/
Right now, there are no public patches to KVM that expose the new CPUID bits and MSRs to the virtual machines, therefore there is no urgent need to update QEMU; remember that updating the host kernel is enough to protect the host from malicious guests. Nevertheless, updates will be posted to the qemu-devel mailing list in the next few days, and a 2.11.1 patch release will be released with the fix.

https://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg01386.html

I am wondering why the +pcid is not added to KVM boot.

I have done some testing with updated linux kernels, without this parameter the new patched Debian 9 Kernels are much slower compared to if +pcid is added for starting kvm.


-cpu SandyBridge,+pcid,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel -m 8192
vs
-cpu SandyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel

If cpu "host" or default is used, pcid is added to the guest system, but migrating between different HW Servers (HP gen8, gen9 and gen10) is not working anymore.

I would suggest adding +pcid for starting kvm
 

Sebastian2000

Member
Oct 31, 2017
80
1
8
41
@fabian I know that all we HAVE to upgrade kernel, but because there is a lot of KVM in some production HOST, it's dificult for us to do it yet and had to wait sometime more. The question is, if all KVM are updated and noone have access to GUI / ssh host proxmox, I understand that risk is very limited about the HOST server as none of guest will can use the vulnerabilitie?
 
Last edited:

sommarnatt

Member
Mar 20, 2014
22
0
21
Sweden
@fabian I know that all we HAVE to upgrade kernel, but because there is a lot of KVM in some production HOST, it's dificult for us to do it yet and had to wait sometime more. The question is, if all KVM are updated and noone have access to GUI / ssh host proxmox, I understand that risk is very limited about the HOST server as none of guest will can use the vulnerabilitie?

You should at the very least patch Meltdown, as it is the easiest one to "use". So that means patching guest, rebooting it.

Spectre needs microcode update, intel has release some already to certain companies like HP, Dell, Supermicro and they've created bios updates for some cpus / servers. It will probably be release to microcode-intel package soon, so you can patch it through the OS. It also means another reboot though afaik.

Spectre PoC from Google was mitigated afaik with the Patch from Proxmox (kudos, good job) and if you have a cluster of Proxmox servers with shared storage (think Ceph, SANs etc) you can just move the guests around, reboot hosts as you like.
 

stef1777

Active Member
Jan 31, 2010
178
8
38
Meltdown available on Jessie.

No yet tried.

Get:1 http://security.debian.org/ jessie/updates/main linux-image-amd64 amd64 3.16+63+deb8u1 [5,844 B]
Get:2 http://security.debian.org/ jessie/updates/main linux-image-3.16.0-5-amd64 amd64 3.16.51-3+deb8u1 [34.0 MB]
Get:3 http://httpredir.debian.org/debian/ jessie/main libnuma1 amd64 2.0.10-1 [32.5 kB]
Get:4 http://httpredir.debian.org/debian/ jessie/main firmware-linux-free all 3.3 [19.1 kB]
Get:5 http://httpredir.debian.org/debian/ jessie/main irqbalance amd64 1.0.6-3+deb8u1 [31.2 kB]
 

JOduMonT

Active Member
Jan 20, 2016
59
5
28
Bangkok
jdumont.consulting
this will tell you if you're kernel is patched :

dmesg | grep "Kernel/User page tables isolation: enabled" && echo "patched :)" || echo "unpatched :("
found on : askubuntu / questions /992137 / how-to-check-that-kpti-is-enabled-on-my-ubuntu

for non paying user (non entreprise) you need to active the pve-no-subscription repo before the 'apt update && apt dist-upgrade'
in /etc/apt/source.list you must have something like :
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription
replace stretch by jessie or wheezy to reflect your version; as you must know

also if you patch your kernel with KPTI and try cat /proc/cpuinfo | grep bugs
you must see one cpu_insecure per core
this is scary and sad but that means the KPTI patch is applied.



We just published new kernels for Proxmox VE 4.x and 5.x, addressing Meltdown and Spectre in the kernel.

Please upgrade your Proxmox VE hosts via "apt update && apt dist-upgrade".

Proxmox VE 5.x: pve-kernel (4.13.13-34)
  • cherry-pick / backport of KPTI / Meltdown fixes (from Ubuntu-4.13.0-23.25)
  • add Google Spectre PoC fix for KVM
  • fix objtool build regression
-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:19:58 +0100

Proxmox VE 4.x: pve-kernel (4.4.98-102)
  • cherry-pick / backport of KPTI / Meltdown fix (based on Ubuntu-4.4.0-107.130)
  • add Google Spectre PoC fix for KVM
-- Proxmox Support Team <support@proxmox.com> Sun, 7 Jan 2018 13:15:19 +0100

__________________
Best regards,

Martin Maurer
Proxmox VE project leader
 

stef1777

Active Member
Jan 31, 2010
178
8
38
Warning: the first kit (PoC) for Spectre 2 is out. Say to run on one KVM and can read memory of other KVM on same host.
 
Last edited:

JOduMonT

Active Member
Jan 20, 2016
59
5
28
Bangkok
jdumont.consulting
actually a easy to understand status is provided by OVH
@ docs.ovh.com/fr/dedicated/meltdown-spectre-kernel-update-per-operating-system/

so mostly Meltdown could be patched
when Spectre still a vulnerability for most OS
 

Volodimir

New Member
Jan 10, 2018
8
0
1
41
Can someone suggest why new 103 kernel (previous one 102 have it) have not KPTI patch:

# uname -a
#1 SMP PVE 4.4.98-103 (Mon, 8 Jan 2018 10:15:44 +0100) x86_64 GNU/Linux
# dmesg | grep "page tables isolation"
# dmesg |grep iso
#
 

stef1777

Active Member
Jan 31, 2010
178
8
38
Works for me with PVE 4.x.

# dmesg | grep "User page tables isolation"
[ 0.000000] Kernel/User page tables isolation: enabled

# pveversion -v
proxmox-ve: 4.4-103 (running kernel: 4.4.98-3-pve)
pve-manager: 4.4-20 (running version: 4.4-20/2650b7b5)
pve-kernel-4.4.98-2-pve: 4.4.98-101
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.95-1-pve: 4.4.95-99
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-54
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.1-5~pve4
pve-container: 1.0-104
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
 

Volodimir

New Member
Jan 10, 2018
8
0
1
41
For me unfortunately not

# dmesg | grep "User page tables isolation"
# pveversion -v
proxmox-ve: 4.4-103 (running kernel: 4.4.98-3-pve)
pve-manager: 4.4-20 (running version: 4.4-20/2650b7b5)
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.95-1-pve: 4.4.95-99
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-54
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.1-5~pve4
pve-container: 1.0-104
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
openvswitch-switch: 2.6.0-2

can anyone suggest what I am do wrong?
 

Volodimir

New Member
Jan 10, 2018
8
0
1
41
Dell PowerEdge M630

# dmidecode | grep 630
Product Name: PowerEdge M630
SKU Number: SKU=NotProvided;ModelName=PowerEdge M630

at same time same model with kernel 102 have not such problems

# dmidecode | grep 630
Product Name: PowerEdge M630
SKU Number: SKU=NotProvided;ModelName=PowerEdge M630
# uname -a
Linux proxmox18 4.4.98-3-pve #1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100) x86_64 GNU/Linux
# dmesg | grep "User page tables isolation"
[ 0.000000] Kernel/User page tables isolation: enabled
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,451
1,115
164
Dell PowerEdge M630

# dmidecode | grep 630
Product Name: PowerEdge M630
SKU Number: SKU=NotProvided;ModelName=PowerEdge M630

at same time same model with kernel 102 have not such problems

# dmidecode | grep 630
Product Name: PowerEdge M630
SKU Number: SKU=NotProvided;ModelName=PowerEdge M630
# uname -a
Linux proxmox18 4.4.98-3-pve #1 SMP PVE 4.4.98-102 (Sun, 7 Jan 2018 13:15:19 +0100) x86_64 GNU/Linux
# dmesg | grep "User page tables isolation"
[ 0.000000] Kernel/User page tables isolation: enabled

same model, or same system? what does /proc/cpuinfo contain on both hosts? there is nothing in the diff between -102 and -103 that should cause this, and the only way in our 4.4 kernel to silently disable pti is if the kernel detects it is booted as PV XEN guest..
 

Volodimir

New Member
Jan 10, 2018
8
0
1
41
Same models and same systems, severs also have same CPU

103 kernel - 32 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2 Sockets)
102 kernel - 32 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2 Sockets)

/proc/cpuinfo have differences only for CPU frequency

# diff 102.txt /root/103.txt
8c8
< cpu MHz : 1872.773
---
> cpu MHz : 2474.718
22c22
< bogomips : 4190.35
---
> bogomips : 4190.21
....
< cpu MHz : 2628.445
---
> cpu MHz : 2366.601
859c859
< bogomips : 4191.63
---
> bogomips : 4191.45

Also server not "booted as PV XEN guest", because this is physical servers, dmidecode also show this.
Related to bios version

103 kernel:
BIOS Version 2.4.2
Firmware Version 2.50.50.50
Lifecycle Controller Firmware 2.50.50.50

102 kernel:
BIOS Version 2.4.2
Firmware Version 2.50.50.50
Lifecycle Controller Firmware 2.50.50.50

so as you can see servers have same hardware/os except kernel.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!