Security Hardening

[deepcloud]

We have seen recent Ransomware attacks on vSphere ESXi Hypervisors and are very concerned about Proxmox being targeted too.

We are planning on doing the hardening of Proxmox hosts and implementing a security audit using lynis.

During the course of this audit I am sure to hit many roadblocks and will seek help and guidance here in this thread.

Any recommendations on protection of the Proxmox hosts, any Antivirus recommended, like Eset, Kaspersky Hybrid Cloud Security or anything else better ?

Any other suggestions like App amour with a list of only ceph and proxmox services being whitelisted and locked down

Thanks in Advance

 

[leesteken]

This section of the PBS documentation might interest you: Ransomware Protection & Recovery.

 

[Dunuin]

I also remember a case where the BMC was publicly available and vulnerable so the PVE host got hit by ransomware that way. So don't forget the BMC as a backdoor and not only focus on PVE itself.

 

[LnxBil]

Any recommendations on protection of the Proxmox hosts, any Antivirus recommended, like Eset, Kaspersky Hybrid Cloud Security or anything else better ?

Oh boy .... this is a money-maker ... those tools are either total useless or will slow down everything so much that you won't work with your infrastructure. And "security" - for a security-minded-person is to NOT USE ANY CLOUD AT ALL, so why would I build a reverse tunnel to any shady company that's claiming they securing my infrastructure with some hybrid cloud security stuff? Get proper software for your ingress and egress points that does e.g. deep packet inspection, proxy, mailfilter and stuff like that and properly secure your hypervisor (that's why we're here).

For PVE this means (some points have been said before):

• Have a good PVE firewall at datacenter level. If your PVE boxes and BMC cannot be reached, they cannot be attacked and any audit will yield nothing. We use it all the time to shield legacy from auditors ... and everyone is happy.
• PVE has also this very nice feature of security groups, that can be applied to each VM turning them into single-host-DMZs, which is one of the big security selling points of PVE.
• have a proper management lan with a lot of loops to jump through for admins, they're attacked first. Consider e.g. using hardened VPN inside of your company for the management LAN.
• scheduled updates (OS, firmware)
• proper backup strategy including backup hardware for a self-sufficient restore outside of your current infrastructure, maybe regular restore tests
• good backend storage with regular out-of-band snapshots to easly restore on ransomware attacks (e.g. ZFS over NFS or iSCSI), can also combined with replication
• SSO on PVE with a short retention / lifetime
• sounds maybe a kind of weird, but use management machines that have exotic operating system to reduce the attack surface, e.g. *BSD, Plan 9, etc.
• This pains me to say, but disable SSH for administrators and use a machine that only has GUI access, e.g. via PVE VNC console and do everything administrative from there (maybe also using 4-eye-principle). It's harder to get this hacked and is the IT-form of galvanic separation.

 

[deepcloud]

Oh boy .... this is a money-maker ... those tools are either total useless or will slow down everything so much that you won't work with your infrastructure. And "security" - for a security-minded-person is to NOT USE ANY CLOUD AT ALL, so why would I build a reverse tunnel to any shady company that's claiming they securing my infrastructure with some hybrid cloud security stuff? Get proper software for your ingress and egress points that does e.g. deep packet inspection, proxy, mailfilter and stuff like that and properly secure your hypervisor (that's why we're here).

For PVE this means (some points have been said before):

• Have a good PVE firewall at datacenter level. If your PVE boxes and BMC cannot be reached, they cannot be attacked and any audit will yield nothing. We use it all the time to shield legacy from auditors ... and everyone is happy.
• PVE has also this very nice feature of security groups, that can be applied to each VM turning them into single-host-DMZs, which is one of the big security selling points of PVE.
• have a proper management lan with a lot of loops to jump through for admins, they're attacked first. Consider e.g. using hardened VPN inside of your company for the management LAN.
• scheduled updates (OS, firmware)
• proper backup strategy including backup hardware for a self-sufficient restore outside of your current infrastructure, maybe regular restore tests
• good backend storage with regular out-of-band snapshots to easly restore on ransomware attacks (e.g. ZFS over NFS or iSCSI), can also combined with replication
• SSO on PVE with a short retention / lifetime
• sounds maybe a kind of weird, but use management machines that have exotic operating system to reduce the attack surface, e.g. *BSD, Plan 9, etc.
• This pains me to say, but disable SSH for administrators and use a machine that only has GUI access, e.g. via PVE VNC console and do everything administrative from there (maybe also using 4-eye-principle). It's harder to get this hacked and is the IT-form of galvanic separation.

thanks @LnxBil

 

[deepcloud]

Should we not look at securing services on Proxmox

systemd-analyze security - gives out most as unsafe... any ideas

 

[LnxBil]

Should we not look at securing services on Proxmox

systemd-analyze security - gives out most as unsafe... any ideas

It's all running as root without a sandbox, what would you expect? As long as you're not exposing any service on the network, it does not matter that much. There are normally no ordinary users running stuff on your PVE host, so this is generally not so bad.

 

[Johannes S]

Should we not look at securing services on Proxmox

systemd-analyze security - gives out most as unsafe... any ideas

You will need to understand what systemd-analyze security does and what's not. It's NOT a general purpose vulnerabilities scanner like lynis but it checks whether the systemd files use certain sandboxing features of systemd. The idea is that you lock your service in an environment where a lot of things are not possible even if in theory the service (or an attacker taking over the service) has the permission to use them. For example I might setup a webserver which runs under the user www-data. The served files are inside the /srv/www directory, which has read/write permissions for the user www-data and every member of the group www-data. Although the users needs write permissions (otherwise they couldn't add new pages) the webserver don't need them, he only needs to read them to serve them to the internet. Thus I could use a systemd directive that this folder should be read-only for the webserver. Now if an attacker manages to manipulate the web server he still wouldn't be able to write files to /srv/www

Of course this will only work as long as there isn't a vulnerability in this part of systemds sandboxing mechanisms.

Lennart Poettering (one of systemds developers) published a blog post on it:
https://0pointer.de/blog/projects/security.html

A shorter (and therefore less complete and detailied) introduction:
https://0pointer.de/blog/projects/security.html

And of course the manual pages of systemd

The majority are low hanging fruits that should work with most programs without causing problems. They are not enabled by default though since in the end the softwares developers should know which permissions are needed by their programs and which are not. Thus it's actually the job of the developers to provide hardened service files
Sadly most projects don't know or don't care about this and thus use still the less secure defaults. A sysadmin however can use systemd overrides, for additional hardening but he will have do do thorough testing to make sure that everything still works.

Coming back to proxmox: Propably proxmox services could also be hardened, but I doubt that there will be much gains. For example the PrivateNetworks directive bans a service for using the network. This is useful for e.G. a service which will only talks to other services on the same machine. For proxmox cluster services this wouldn't make sense obviouvsly . And I guess that this might be true for most of the remaining directives.
On the other hand there are some directives which might work even then. But this would be needed to investigated.
Guess I should open a suggestion ticket in proxmox bugzilla or write a mail to their security contact on this

In your case I wouldn't worry to much, if you followed LnxBils hint on the firewall (combined with another firewall on the network level so even in case of an security issue with the PVE firewall you will still be save), backup, sso etc you are better off than many homelabbers AND companies.