Working on hardening Proxmox hosts - looking for advice regarding some findings

[happypuffinlinuxengineer]

Hello Proxmox Community & Team,

I am working through making a hardened baseline for my teams Proxmox deployment, loosely following CIS Lvl1/2 and DISA STIG requirements for Linux operating systems. I have made good progress, and have built out a hardened PVE cluster successfully that has functioning replication, migration, etc. Just have a few questions that have come out of this process:

- The SSL private keys are kept in plaintext in the configuration database, as well as on the cluster filesystem. They are protected by being owned by root - so in my mind largely a non-issue, however for approval purposes I will need to either explain this as not security relevant or find a means of either encrypting the private keys (which creates the problem of just having another set of keys somewhere to decrypt). The idea was thrown around regarding the possibility of encrypting the configuration database on the filesystem - is this a possibility in a future release?

- Has FIPS mode encryption been attempted with a PVE installation? I don't need FIPS-accreditation, rather just the usage of FIPS-type ciphers would greatly aid my team with obtaining approvals for PVE. For now, leaving the Debian OS set to it's default cryptography configuration.

- Setting the default user shell timeout to 900 seconds in /etc/bash.bashrc should be OK?

- How to disable TLSv1.2 for the pveproxy 8006 interface? I created a file in /etc/defaults/ for pveproxy and set the following line, but TLSv1.2 is still on (need it to be turned off and run in TLSv1.3 only mode):

CIPHERS="HIGH:!TLSv1:!TLSv1_2:!SSLv3:!aNULL:!MD5"

- Any known problems with setting maximum SSHD sessions to 3 (project req), and setting the client alive interval and client alive count max to 600 as per CIS guidelines? I presume it would limit the cluster to have only 3 migrations in flight between hosts at a time? So far, it is behaving normally.

Thanks for any input.

 

[nicobos]

I am very interested in the steps you have taken to harden your Promox Datacenter. We are running Proxmox cluster 8.3 built on the standard ISO image as provided by proxmox.com IT auditing requires me to come up with a procedure to harden the Proxmox nodes. Basically I would like to follow CIS benchmarks as close as possible, but I am not sure if implementing such benchmark on my Proxmox cluster nodes could lead to issues. So any advise on safely making the Proxmox cluster nodes more secure than the default installation is appreciated.

 

[chrcoluk]

Setup a Nginx reverse proxy, there is guides on the net, then you can configure Nginx which is far more configurable. In my experience Nginx configuration survives Proxmox updates.

 

[LnxBil]

I would also be interessted in the results of this CIS hardening.

The SSL private keys are kept in plaintext in the configuration database, as well as on the cluster filesystem. They are protected by being owned by root - so in my mind largely a non-issue, however for approval purposes I will need to either explain this as not security relevant or find a means of either encrypting the private keys (which creates the problem of just having another set of keys somewhere to decrypt). The idea was thrown around regarding the possibility of encrypting the configuration database on the filesystem - is this a possibility in a future release?

As you've already said ... why would adding another insecure ring of encryption help in this case? Having the encrypted data AND the key to decrypt it on the same machine does not make the system more secure. Can you try to explain this in more detail?

Has FIPS mode encryption been attempted with a PVE installation? I don't need FIPS-accreditation, rather just the usage of FIPS-type ciphers would greatly aid my team with obtaining approvals for PVE. For now, leaving the Debian OS set to it's default cryptography configuration.

I'm unware that you can set the cipher suite like on RHEL to just use FIPS. @chrcoluk has a good point, this will mitigate at least the port 8006 issue.

How to disable TLSv1.2 for the pveproxy 8006 interface? I created a file in /etc/defaults/ for pveproxy and set the following line, but TLSv1.2 is still on (need it to be turned off and run in TLSv1.3 only mode):
CIPHERS="HIGH:!TLSv1:!TLSv1_2:!SSLv3:!aNULL:!MD5"

Have you tried to set the value to ""? According to the manpage, there are two seperate settings one for < 1.3 and one for >=1.3

Setting the default user shell timeout to 900 seconds in /etc/bash.bashrc should be OK?

There is nothing run in an interactive shell where this timeout would come into play, so yes, it should be save to set this.

Any known problems with setting maximum SSHD sessions to 3 (project req), and setting the client alive interval and client alive count max to 600 as per CIS guidelines? I presume it would limit the cluster to have only 3 migrations in flight between hosts at a time? So far, it is behaving normally.

At least the alive stuff is also not relevant because the shell is not interactive so if it has a connection it is doing something and therefore you won't need a keep alive.

For the migration, you may run into problems with 3 sessions and need to check if it works. You can also switch from secure to insecure migration (best with an internal and not exposed network), which does not involve SSH at all.

You can also have special settings for special machines or your PVE cluster network in your sshd_config file, so that for all other machines except the PVE nodes, your limitation will apply. With this, you can adhere to the CIS requirements for alle cluster-external clients.

 

[bbgeek17]

For the migration, you may run into problems with 3 sessions and need to check if it works. You can also switch from secure to insecure migration (best with an internal and not exposed network), which does not involve SSH at all.

FYI: https://kb.blockbridge.com/technote/proxmox-concurrent-vm-migration/index.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

FYI: https://kb.blockbridge.com/technote/proxmox-concurrent-vm-migration/index.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Nice writeup. It's more detailed than the reference documentation

 

[Qon007]

Hei folks,
I just ran DEBIAN12-CIS benchmark https://github.com/ansible-lockdown/DEBIAN12-CIS against fresh proxmox9 host and these are the failed check. Most of them should be fine to patch, but not sure about the kernel modules or anything else proxmox actually need as it is?

CIS_ID and Titles for Unique Failed Checks:
------------------------------------------
- 1, ., 1, ., 1, ., 1: 1.1.1.1 | Ensure cramfs kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 1: 1.1.1.1 | Ensure cramfs kernel module is not available | disabled
- 1, ., 1, ., 1, ., 1, 0: 1.1.1.10 | Ensure unused filesystems kernel modules are not available | Manual Check will be needed to validate if alerts
- 1, ., 1, ., 1, ., 2: 1.1.1.2 | Ensure freevxfs kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 2: 1.1.1.2 | Ensure freevxfs kernel module is not available | disabled
- 1, ., 1, ., 1, ., 3: 1.1.1.3 | Ensure hfs kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 3: 1.1.1.3 | Ensure hfs kernel module is not available | disabled
- 1, ., 1, ., 1, ., 4: 1.1.1.4 | Ensure hfsplus kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 4: 1.1.1.4 | Ensure hfsplus kernel module is not available | disabled
- 1, ., 1, ., 1, ., 5: 1.1.1.5 | Ensure jffs2 kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 5: 1.1.1.5 | Ensure jffs2 kernel module is not available | disabled
- 1, ., 1, ., 1, ., 6: 1.1.1.6 | Ensure overlayfs kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 6: 1.1.1.6 | Ensure overlayfs kernel module is not available | disabled
- 1, ., 1, ., 1, ., 7: 1.1.1.7 | Ensure squashfs kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 7: 1.1.1.7 | Ensure squashfs kernel module is not available | disabled
- 1, ., 1, ., 1, ., 8: 1.1.1.8 | Ensure udf kernel module is not available
- 1, ., 1, ., 1, ., 8: 1.1.1.8 | Ensure udf kernel module is not available | blacklist
- 1, ., 1, ., 1, ., 9: 1.1.1.9 | Ensure usb-storage kernel module is not available
- 1, ., 1, ., 1, ., 9: 1.1.1.9 | Ensure usb-storage kernel module is not available | blacklist
- 3.2.1: 3.2.1 | Ensure dccp kernel module is not available | DCCP config
- 3.2.2: 3.2.2 | Ensure tipc kernel module is not available | tipc config
- 3.2.3: 3.2.3 | Ensure rds kernel module is not available | rds config
- 3.2.4: 3.2.4 | Ensure sctp kernel module is not available | sctp config

See the attached file for full list.

 

[LnxBil]

USB storage may be needed by some homelabbers, depending on your system, but most enterprise customers will most probably not use it.

Have you blacklisted the modules and ran it again?

 

[Qon007]

USB storage may be needed by some homelabbers, depending on your system, but most enterprise customers will most probably not use it.

Have you blacklisted the modules and ran it again?

Didn't tried to patch the system yet. I would rather disable USB in bios if really needed.

 

[i8degrees]

Didn't tried to patch the system yet. I would rather disable USB in bios if really needed.

Beware that disabling USB in BIOS may not always do quite as you expect. I have one machine where 5V power is still available even after disabling -- whether one considers this a bug is not my point; double check your ports afterwards!