Having a thought in regards to running SentinelONE on the PVE host itself. Would definatley want to exclude the storage locations that have the VM/Container disks running from real-time scanning, but any other gotchas from running a EDR or AV in general on the PVE host to detect and stop malicious code?
Running EDR on PVE Host
- Thread starter Undivided0519
- Start date
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
We are doing something similar with MS Defender. We took inspiration from MS's documentation on excluded files for Hyper-V https://learn.microsoft.com/en-us/d...crosoft-defender-antivirus#hyper-v-exclusions
We Exclude all VM files eg *.qcow2, the cluster filesystem /etc/pve/* and Proxmox processes like kvm, pmxcfs.
Not running EDR is not an option for us, so would love to get some offical guidance from proxmox
We Exclude all VM files eg *.qcow2, the cluster filesystem /etc/pve/* and Proxmox processes like kvm, pmxcfs.
Not running EDR is not an option for us, so would love to get some offical guidance from proxmox
What are you protecting yourself from? The host itself doesn’t run client loads and any management interfaces should be isolated in a proper environment. You can even use Secure Boot to make sure you do not run “bad” code.
Moreover most EDR software does not “do” anything on Linux, most of it requires you to pipe in a binary kernel module, but I’ve never seen an EDR detect and block anything on Linux - other than when you try to unload the EDR because it crashes the kernel.
I have had bad experiences with software that can interfere with the operation of mission critical systems. Eg. CrowdStrike issue not too long ago.
Perfectly fine to run the EDR in your guests though. If you really need to and your EDR supports any current kernel (also a joke with most EDR providers - Linux support is generally an afterthought), it will work until your kernel needs upgrading, then you will have to bug their support for 3 months to build a new one.
If you just need to check a box for an audit, install ClamAV, just as, if not more effective than CrowdStrike, Cylance, Sophos, Symantec, Defender (that’s a joke even in Windows) and any other commercial EDR on Linux I have ever worked with. Note VMware also does not run EDR on its cluster hosts, nor does any other commercial clustered storage or virtualization system support or recommend it, RedHat, Nutanix, Citrix etc
True security tools: Ansible, Tripwire, Nessus, Snort, above mentioned SecureBoot etc
Moreover most EDR software does not “do” anything on Linux, most of it requires you to pipe in a binary kernel module, but I’ve never seen an EDR detect and block anything on Linux - other than when you try to unload the EDR because it crashes the kernel.
I have had bad experiences with software that can interfere with the operation of mission critical systems. Eg. CrowdStrike issue not too long ago.
Perfectly fine to run the EDR in your guests though. If you really need to and your EDR supports any current kernel (also a joke with most EDR providers - Linux support is generally an afterthought), it will work until your kernel needs upgrading, then you will have to bug their support for 3 months to build a new one.
If you just need to check a box for an audit, install ClamAV, just as, if not more effective than CrowdStrike, Cylance, Sophos, Symantec, Defender (that’s a joke even in Windows) and any other commercial EDR on Linux I have ever worked with. Note VMware also does not run EDR on its cluster hosts, nor does any other commercial clustered storage or virtualization system support or recommend it, RedHat, Nutanix, Citrix etc
True security tools: Ansible, Tripwire, Nessus, Snort, above mentioned SecureBoot etc
Last edited:
Ransomware crews love getting onto hypervisors and encrypting then deleting VMs on the host – this was one of the advantages of Proxmox over ESXi for us that we could run a security agent on the host. Yes we network segment hosts and have MFA/SSO for both the web and SSH and so on, but this is the last line of defense.
Again, does your EDR detect activity on Ceph? Simple test: write a Python script that overwrites encrypted data to a file. Does your EDR detect or stop it? How would it detect the difference of ransomware and a guest using something like Bitlocker?
For protection against ransomware you need immutable snapshots and backups on your storage.
For protection against ransomware you need immutable snapshots and backups on your storage.
It's not detecting ransomware activity on the guest VM's, it is detecting it on the Proxmox node itself. The attacker gets onto your Proxmox node, stops the VMs via CLI, then encrypts then qcow2 files. At the moment, It's largley happening on ESXi and Hyper-V eg https://www.microsoft.com/en-us/sec...hypervisor-vulnerability-for-mass-encryption/.
Last edited:
Yes, I understand, but for the exploit you linked to, is there any EDR that would detect it on a hypervisor today? If you think so, please test it. Windows Server also has at least Defender built in, and on Windows you should always run an EDR, so why doesn’t it on its own platform?
I think most enterprises have EDR these days, yet these types of live-of-the-land ransomware does not get detected.
I think most enterprises have EDR these days, yet these types of live-of-the-land ransomware does not get detected.
I guess it migh not detect the start of a LOLBin attack, but it should detect other ransomware artifacts such as file encyrption, file deletions etc for example we get alerts on our linux file servers when large amountf of files are altered. It should also detect/block network traffic to C&C servers and the like. We even run it on our HPC servers without any issue, so I'm not sure what the down side is.
Yes, but the definition of a hypervisor is that a large amount of blocks get altered on a continuous basis. Moreover, I highly doubt the EDR is even aware of Ceph-type traffic and can integrate into it. Even a QCOW2 file is just 1 file, you only need to encrypt a few blocks to make it completely unusable. Again, what you're describing is on Linux clients where people with browsers do dumb shit, not the hypervisor, and for those, yes, there are EDR. Traffic to C&C servers - it would have to already know (blacklist) those and that seems to me should be an issue that is handled at the edge by IDS integrated into firewalls.
My contention is that once a ransomware gets sufficient rights to even start its attack, it's already too late, it has deep access in your network and root access at that point, if that has happened on your hypervisor, you've been compromised, you need to disconnect your datacenter, wipe your admins computers, you need to wipe your jump hosts, you need to wipe your entire cluster and recover from ... snapshots and backups.
I don't have a problem with EDR as a detection tool, but what you're describing can be picked up by any SIEM as well, and likely when the attacker is just knocking at the door. Allowing it to interfere (kill VMs, hang kernels, add measurable latency to read/write calls etc) has always been an issue for us, especially in clusters, you're potentially hanging the entire cluster because ClownStrike uses its customers to test a dynamic update - this has happened twice on Linux before the notorious Windows outage, it's why, even on VMs we no longer allow EDR to run as a kernel module - it has to use BPF instead.
I would love to hear about a good EDR tool for Linux, one that a) builds the kernel interface regardless of kernel version b) doesn't interfere with legitimate operations and c) can actually detect 'bad behavior' both on the host and in VMs and containers. Right now, it seems most EDR vendors are just there to tick a box that they also have a Linux (and same can be said about Mac) client, but they only detect Windows threats and Windows-like behavior, replicate the same attack in Python, and it won't detect anything.
My contention is that once a ransomware gets sufficient rights to even start its attack, it's already too late, it has deep access in your network and root access at that point, if that has happened on your hypervisor, you've been compromised, you need to disconnect your datacenter, wipe your admins computers, you need to wipe your jump hosts, you need to wipe your entire cluster and recover from ... snapshots and backups.
I don't have a problem with EDR as a detection tool, but what you're describing can be picked up by any SIEM as well, and likely when the attacker is just knocking at the door. Allowing it to interfere (kill VMs, hang kernels, add measurable latency to read/write calls etc) has always been an issue for us, especially in clusters, you're potentially hanging the entire cluster because ClownStrike uses its customers to test a dynamic update - this has happened twice on Linux before the notorious Windows outage, it's why, even on VMs we no longer allow EDR to run as a kernel module - it has to use BPF instead.
I would love to hear about a good EDR tool for Linux, one that a) builds the kernel interface regardless of kernel version b) doesn't interfere with legitimate operations and c) can actually detect 'bad behavior' both on the host and in VMs and containers. Right now, it seems most EDR vendors are just there to tick a box that they also have a Linux (and same can be said about Mac) client, but they only detect Windows threats and Windows-like behavior, replicate the same attack in Python, and it won't detect anything.