Question about vulnerabilities and mitigations

J

jaytee129

Member
Jun 16, 2022
144
10
23
I loaded proxmox on what is a 2-3 year old system but with a CPU that was first launched in 2012 (so fairly old model in computer-years). using the lscpu command I see there are a number of vulnerabilities. Is it correct to say that where the command reports "Mitigation", proxmox has implemented the stated mitigation? Or is it just a suggestion as to what to do, and if it doesn't say mitigation there is none?

Code: 
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Vulnerable: No microcode
Vulnerability Tsx async abort:   Not affected
 
jaytee129 said:
Is it correct to say that where the command reports "Mitigation", proxmox has implemented the stated mitigation?
Click to expand...
Yes, there are "on". The "Vulnerable" are the ones, that you need to address if possible, yet I do not know where to get the microcode to fix this.
 
I find that this script gives a nice detailed overview. I think that SMT vulnerable means you need to disable SMT/Hyper-threading for the mitigation to be fully effective (which might not be so bad as they only count for 20-40% of a real core and disabling will give more cache per core).
 
@leesteken, @Neobin, Thanks so much. This is great stuff to have/know.

One of my two systems has quite a few vulnerabilities, despite my having upgraded the BIOS with latest version just a couple of weeks ago. It's interesting because the system was sold as ideal for a firewall, e.g. pfsense. I'm going to research these vulnerabilities to try to assess the risk but I don't think I want this thing connected to the internet. Question is would it okay if it was behind a firewall.

I tried turning off Hyperthreading but it really slowed the system down. It's not the quickest cpu to start with.
 
What are the factors at play here in terms of mitigation? It looks like a number of mitigations are handled by the OS, but, in theory, are most or all of these vulnerabilities something that can (should) be addressable by the system BIOS? Is it fair to say OS mitigations are there because these are not addressed at lower level in many cases?

In terms of future product selection - especially for something directly connected/listening to the internet, is it advisable to not go back further than a certain launch year or generation of CPU's? If so, what year/generation should be the cutoff? I generally buy intel because I have the perception there's more support when used with hypervisors. (My previous experience was with VMware, and I also have TrueNAS, for which I use Intel)
 
The problems are in the hardware design and some can be fixed in firmware (not the BIOS but the updatable microcode inside the CPU). Some can only be worked-around/mitigated via complex software solutions until new hardware is designed (which may have new issues in the future).
There is no known safe product on the market and new related issues keep popping up. Although newer CPUs are often less vulnerable because the designers know about some of the issues for some time now, researchers find new ways to exploit the hardware designs and the hardware and software fixes.
And some "sure" fixes in hardware, like no speculative execution at all. would currently be so slow that it can't compete and won't sell in large enough volumes. Like everything else it is a balancing act and without mitigations even browsers can be vulnerable to malicious websites (it's not only a problem software running as root).

Interesting as all of this is, none of this is Proxmox (or virtualization or OS) specific but it does hit harder on hypervisors that run untrusted VMs by unkown people.
 
I checked a third computer (with an i5-5300U CPU launched 2015) that I was thinking of using with proxmox. I installed the basic stuff, ran the script you (@leesteken) pointed to above three times: Once before doing any OS updates, then after doing updates, then after installing intel microcode that @Neobin pointed to. There was no difference between the first two - both had the same 5 vulnerabilities - but after updating the microcode everything red turned green - no vulnerabilities. Even the ones due to SMT/HT are no longer issues. (I thought the only way to fix those ones was to turn off Hyperthreading.)

The microcode is obviously a key security fix needed so thanks again for pointing to it. I had heard about it but read somewhere on this forum that this was only available (at least from Proxmox staff) if one paid for support. It's a bit disappointing to find out such a fundamental security patch is not provided for free and also strange. If you're going to charge for fundamental security fixes, then charge for your product. Don't offer it for free. There are enough cyber threats out there that increasingly cost individuals, corporations, and society in general. Security should be no 1 priority for internet connected products. But I digress....

One thing that caught my eye was the following info:

Code: 
CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: usercopy/swapgs barriers and __user pointer sanitization)
* Kernel has array_index_mask_nospec:  UNKNOWN  (couldn't check (missing 'readelf' tool, please install it, usually it's in the 'binutils' package))
* Kernel has the Red Hat/Ubuntu patch:  UNKNOWN  (missing 'strings' tool, please install it, usually it's in the binutils package)
* Kernel has mask_nospec64 (arm64):  UNKNOWN  (couldn't check (missing 'readelf' tool, please install it, usually it's in the 'binutils' package))
* Kernel has array_index_nospec (arm64):  UNKNOWN  (couldn't check (missing 'readelf' tool, please install it, usually it's in the 'binutils' package))
* Checking count of LFENCE instructions following a jump in kernel...  UNKNOWN  (couldn't check (missing 'readelf' tool, please install it, usually it's in the 'binutils' package))
> STATUS:  NOT VULNERABLE  (Mitigation: usercopy/swapgs barriers and __user pointer sanitization)

Although bottom line says "not vulnerable" to this variant of Spectre, it's telling me to install binutuls (this, I presume) so the script can use 'readelf'. My first thought was do I really want to do this when there's a clear statement my system is not vulnerable? Will it change the outcome? Do I want to add the binutils toolbox when I only want one tool for one task (though, admittedly, I may want to perform this task again in the future)? Is that tool or other ones in the toolbox a risk to have on my system if a bot with bad intentions (is there any other kind?) does get into it with root access?

Also, I'm now wondering if it's a bad idea to use proxmox to run an internet firewall - Sophos UTM in my case - along with another VM. While I'm logically funneling my WAN traffic to the firewall VM, I'm wondering if proxmox vulnerabilities are either somewhat or fully exposed to the internet and my other VM ahead of the firewall. While for all I know it might do a better job at mitigating vulnerabilities than Sophos where it's sufficiently abstracted from the h/w, does it just add openings that wouldn't be there if the box was only running firewall software?

Any comments on this?
 
The extra utilities to install is only for the script to tell you whether mitigations are implemented in the kernel, not whether they are active and working. It will not change the outcome. Yes at lot depends on the hardware and the microcode from the manufacturer of your CPU. The Linux kernel will do its best regardless. So please complain to the hardware vendors.
jaytee129 said:
Also, I'm now wondering if it's a bad idea to use proxmox to run an internet firewall - Sophos UTM in my case - along with another VM. While I'm logically funneling my WAN traffic to the firewall VM, I'm wondering if proxmox vulnerabilities are either somewhat or fully exposed to the internet and my other VM ahead of the firewall. While for all I know it might do a better job at mitigating vulnerabilities than Sophos where it's sufficiently abstracted from the h/w, does it just add openings that wouldn't be there if the box was only running firewall software?
Click to expand...
It's not Proxmox that's the problem, its the (fast) CPU. Yes, a firewall hardware device might have less (or different vulnerabilities). Many of the problems also apply to fast ARM processors because of speculative execution of instructions. Maybe a slow (MIPS?) firewall is safer, but most devices tend to get very few updates (if ever), while the Linux kernel tries to keep on top of things. You could run a separate VM with PCIe passthrough of the network device for an extra layer of defense.
I guess it depends on whether you trust the hardware vendors to provide safe hardware that doesn't need updates, or the (free, open source) software vendors that keep providing updates (sometimes only for a while, like Android phones). Or maybe do both if you think that's safer.
My preference is OpenWrt which I run on separate hardware (WiFi routers with ARM or MIPS) and which gets regular software updates (including the Linux kernel and drivers).

PS: Nobody can give you guarantees or clearly defined guidelines as everything is constantly changing and nothing is perfect. You'll have to do your own research and decide for yourself what to trust and what extra defense to implement.
 
Last edited:
jaytee129 said:
I had heard about it but read somewhere on this forum that this was only available (at least from Proxmox staff) if one paid for support. It's a bit disappointing to find out such a fundamental security patch is not provided for free and also strange. If you're going to charge for fundamental security fixes, then charge for your product. Don't offer it for free.
Click to expand...

Please provide a reference/link to this. I have the feeling, that there might be a misunderstanding.
 
  • Like
Reactions: leesteken
jaytee129 said:
I had heard about it but read somewhere on this forum that this was only available (at least from Proxmox staff) if one paid for support. It's a bit disappointing to find out such a fundamental security patch is not provided for free and also strange. If you're going to charge for fundamental security fixes, then charge for your product. Don't offer it for free.
Click to expand...
Neobin said:
Please provide a reference/link to this. I have the feeling, that there might be a misunderstanding.
Click to expand...
The enterprise repository (for which you pay) is only slower in updates. Unless this is true (and please provide evidence), there is no functionality that is present in the enterprise repository that is not present in the no-subscription repository: actually, without a paid subscription (and for free), you get updates, improvement and features earlier.
 
  • Like
Reactions: guletz and Neobin
I tried to find the posts I ran into about getting microcode fix and can't find them. Could well have been my misunderstanding of difference between 'non-free' (open source code) and paid/subscription support. I will say there wasn't any mention of where/how to get the microcode otherwise I would have followed that thread - and that certainly would have helped avoid any misunderstanding. Instead I went to vendor and manufacturer with the issue and manufacturer replied "But we don't have microcode for the BIOS. can you update our latest bios pls ?". I upgraded BIOS but it didn't get rid of the message the kernel was giving me about being a few version behind (18 vs 22 in my case), as I expected from the reply. I clearly remember thinking, 'oh s*** what do I do now and looking up the cost/benefit of support vs cost/benefit of buying new h/w.

Was letting time go by but the reality for the unit in question is that it has numerous vulnerabilities after all the possible updates so I'm thinking it's going to have to be relegated to a role firmly behind the firewall, and that I'll need newer h/w.
 
leesteken said:
Maybe a slow (MIPS?) firewall is safer, but most devices tend to get very few updates (if ever), while the Linux kernel tries to keep on top of things. You could run a separate VM with PCIe passthrough of the network device for an extra layer of defense.
I guess it depends on whether you trust the hardware vendors to provide safe hardware that doesn't need updates, or the (free, open source) software vendors that keep providing updates (sometimes only for a while, like Android phones). Or maybe do both if you think that's safer.
My preference is OpenWrt which I run on separate hardware (WiFi routers with ARM or MIPS) and which gets regular software updates (including the Linux kernel and drivers).
Click to expand...

Hi,

IMO, is allways better to mix security based on software AND hardware. I use for any LAN when I can, at least 1 or 2 Mikrotik devices(for ex. HEX S - MMIPS arch, 4 virtual core cpu, 256 MB RAM, and is very cheap). The first one is a border ruter firewall, and the second is a transparent bridge:

WAN/Internet ----- Border FW ---- Bridge FW ----- LAN

On the bridge device I am running various firewall rules(with no IP), so even in case that Border is compromise, the Bridge firewall can not be attaked(no system service on bridge) as the next step of the attack development. As for updates, you have quite often updates, sometimes 3-4 in the same month.


Good luck / Bafta!
 
You should also consider, that those CPU-based vulnerabilities are often only locally exploitable and a lot of vendors stated that their software is not remotely vulnerable if you have not set up the mitigations (which cost A LOT of performance). So for a firewall, this could still be totally fine.
 
  • Like
Reactions: guletz
LnxBil said:
You should also consider, that those CPU-based vulnerabilities are often only locally exploitable and a lot of vendors stated that their software is not remotely vulnerable if you have not set up the mitigations (which cost A LOT of performance). So for a firewall, this could still be totally fine.
Click to expand...
Hmmm. interesting. 'bit counterintuitive. at least to someone who works at level way above CPU and OS code. isn't most if not all the CPU/OS functionality (and vulnerabilities) ultimately exploited through the network port, whether used for LAN or WAN?

And even if the WAN port is somehow less vulnerable than the LAN port, anyone getting through malware or other internal weakness (admittedly perhaps a bit harder to do) would have access to the firewall through the LAN port to add to it's jump off points or whatever the particular vulnerability exploited allows. Is that a correct way to look at it?

Still trying to figure out if I'm going to buy some new hardware to replace the vulnerable unit I now used as a firewall.
 
guletz said:
Hi,

IMO, is allways better to mix security based on software AND hardware. I use for any LAN when I can, at least 1 or 2 Mikrotik devices(for ex. HEX S - MMIPS arch, 4 virtual core cpu, 256 MB RAM, and is very cheap). The first one is a border ruter firewall, and the second is a transparent bridge:

WAN/Internet ----- Border FW ---- Bridge FW ----- LAN

On the bridge device I am running various firewall rules(with no IP), so even in case that Border is compromise, the Bridge firewall can not be attaked(no system service on bridge) as the next step of the attack development. As for updates, you have quite often updates, sometimes 3-4 in the same month.


Good luck / Bafta!
Click to expand...

Would you know if Mikrotik could be used not just to provide an extra layer of protection up front but also take care of WAN failover such that it's transparent to proxmox box? That would certainly add value as I hope to add that capability and haven't found how to set up network failover in proxmox yet. Also I have an older Mikrotik HEX I got working a long time ago (with a lot of effort) but ended up replacing it with something else and put it on the shelf. They seem quite versatile but are not for the faint of heart. Also I seem to recall the unit I had could be configured to use a cellular USB WAN stick, meaning not just wired ISP modem failover options.
 
jaytee129 said:
Hmmm. interesting. 'bit counterintuitive. at least to someone who works at level way above CPU and OS code. isn't most if not all the CPU/OS functionality (and vulnerabilities) ultimately exploited through the network port, whether used for LAN or WAN?

And even if the WAN port is somehow less vulnerable than the LAN port, anyone getting through malware or other internal weakness (admittedly perhaps a bit harder to do) would have access to the firewall through the LAN port to add to it's jump off points or whatever the particular vulnerability exploited allows. Is that a correct way to look at it?

Still trying to figure out if I'm going to buy some new hardware to replace the vulnerable unit I now used as a firewall.
Click to expand...
You have to look at the service the machine provides, e.g. a server that has only one out-to-the-world port providing database services via SQL. How should that be exploitable by just running SQL queries? On the other hand, on machines with a lot of virtual machines on it, it is totally different and you should also abstract as much as possible (no KSM, no CPU type host, etc.)
 
  • Like
Reactions: jaytee129
jaytee129 said:
Would you know if Mikrotik could be used not just to provide an extra layer of protection up front but also take care of WAN failover such that it's transparent to proxmox box?
Click to expand...

Yes, it could be done. I use it in some situation when a dual wan is abailable. Mikrotik is very versatile.
It could be done also in Linux, like this:

- you have 2 interfaces(A= main ISP, B is secondary, default gw is A)
- create 2 diiferent "good" route:

route add 9.9.9.9 gw A
route add 8.8.4.4 gw B

- create a script that will test if A is available:

if ping 9.9.9.9 is OK AND IF default gw = A then exit
else
route del default gw
route add default gw B

.... see netwatch(script-up/script-down) in Mikrotik.


Good luck / Bafta !
 
Last edited:
  • Like
Reactions: jaytee129
You must log in or register to reply here.
Top Bottom