Security: recommendations for going prod with pve

[doitright]

Hello dear community,

We are a small startup with two people and are currently setting up our infrastructure.

We will be active in the media industry and have a strong focus on open source, as well as the intention to support relevant projects later on as soon as cash flow comes in.

We have a few questions about the deployment of our Proxmox hypervisor, as we have experience with PVE, but not directly in production.

We would like to know if additional hardening of the PVE hypervisor is necessary. From the outset, we opted for an immutable infrastructure and place value on quality and “doing it right and properly” rather than moving quickly to market.

This means that our infrastructure currently looks something like this:

1. Debian minimal is the golden image for all VMs. Our Debian is CIS hardened and achieves a Lynis score of 80. Monitoring is currently still done via email notifications, partitions are created via LVM, and the VMs are fully CIS compliant (NIST seemed a bit too excessive to us).
2. Our main firewall is an Opnsense with very restrictive rules. VMs have access to Unbound (via Opnsense), RFC1918 blocked, Debian repos via 443, access to NTP (IP based, NIST), SMTP (via alias to our mail provider), and whois (whois.arin.net for fail2ban). PVE also has access to PVE repos.

Suricata runs on WAN and Zenarmor runs on all non-WAN interfaces on our opnsense.

3. There are honeypot files on both the VMs and the hypervisor. As soon as someone opens them, they are immediately notified via email.

4. Each VM is in its own VLAN. This is implemented via a CISCO VIC 1225 running on the pve hypervisor. This saves us SDN or VLAN management via PVE. We have six networks for public and private services, four of which are general networks, one for infrastructure (in case traffic/reverse proxy, etc. becomes necessary), and one network reserved for trunk VLAN in case more machines are added later.

5. Changes are monitored via AIDE on the VMs and, as mentioned, are currently still implemented via email.

6. Unattended upgrades, cron jobs, etc. are set up for VMs and Opnsense.

7. Backup strategy and disaster recovery: Opnsense and PVE run on ZFS and are backed up via ZFS snapshots (3 times, once locally, once on the backup server, and once in the cloud). VMs are backed up via PBS (Proxmox Backup Server).

Our question now is:

Does Proxmox need additional hardening to go into production?

We are a little confused. While our VMs achieve a Lynis score of 79 to 80, our Proxmox only achieves 65 points in the Lynis score and is not CIS hardened.

But we are also afraid of breaking things if we now also harden Proxmox with CIS.

With our setup, is it possible to:

1. Go online for private services (exposed via Cloudflare tunnel and email verification required)
2. Go online for public services, also via Cloudflare Tunnel, but without further verification – i.e., accessible to anyone from the internet?

Or do we need additional hypervisor hardening?

As I said, we would like to “do it right” from the start, but on the other hand, we also have to go to market at some point...

What is your recommendation?

Our Proxmox management interface is separate from VM traffic, TOTP is enabled, the above firewall rules are in place, etc., so our only concern that would argue for VM hardening is VM escapes. However, we have little production experience, even though we place a high value on quality, and are wondering whether we should try to harden CIS on Proxmox now or whether our setup is OK as it is?

Thank you very much for your support.

 

[Onslow]

Welcome, doitright! I like your approach (shown also in your nick ).
I'm not qualified enough to answer your question. I'm quite impressed how thoughtfully created is your infrastructure.

The only idea which has come to my mind is whether the plan for VM backups is similarly good. Maybe yes, but you haven't described it, so I've thought I'll write.

Proxmox Backup Server is a great choice. I would check if you use its features to make backups even more secure. I mean: giving to PVE only minimal required permissions - the ability only to create backups and NOT to delete nor rotate backups. See https://pbs.proxmox.com/docs/user-management.html#access-control and https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

Retension (schedules of pruning jobs) should be created in the PBS, not in the PVE. Then even if someone compromises PVE, he can't destroy the backups in the PBS.

I also suggest using "namespaces" in PBS https://pbs.proxmox.com/docs/storage.html#backup-namespaces
from the start. Then if (when) you have more PVE servers, you can separate permissions of PVE servers to their namespaces and still profit from deduplication.

I would also use another (possibly remote) PBS to copy backups from the first PBS. Whether in "push" or in "pull" direction - one must pick depending on the approach and possibilities.

But all the https://pbs.proxmox.com/docs/ is worth reading
Hope I helped a little.

 

[Johannes S]

Hello,

like Onslow I'm impressed how much effort and thought you put in your infrastructure. Most people do it the other way around, I prefer your approach

I remember some earlier discussions on this e.g. https://forum.proxmox.com/threads/proxmox-server-hardening-document-for-compliance.146961 or https://forum.proxmox.com/threads/proxmox-security-hardening.136924/ The search function will also yield some results

Basically anything you do on a regular Debian server should work on ProxmoxVE too although some measures (like disallowing root login via ssh) would break the functionality of the cluster. But it's a start.

Regarding PBS I would also suggest to have an offsite PBS which is only accessible via VPN and only from one or two management machines (maybe even another vserver not connected to your regular infrastructure) and configure it like this:

• A pull-sync pulls backups from your local PBS, each PBS is only allowed to read from the other, but not write to it
• The firewall on the remote PBS only allows connections from the management machine, not from your infrastructure.
• Retention rules are higher and different from your local PBS

So an attacker who was able to take over all your local infrastructure including the local PBS still can't access the remote PBS. If you need to recover something from the remote PBS you would create a temporary exclusion in it's firewall, restore everything you need and afterwards close the firewall again.

Regards, Johannes.

 

[doitright]

Thank you for your kind welcoming here and your advices.
And for the infra compliments
We just do not want massive rebuilds (or sleepless nightmares) later because of high technical debt (no mater if on infra, software or anything else business related), since we are like I said just two persons and think its better to invest now in a solid basis which also lasts in the long run, as maybe having sleepless nights and an undocumented nightmare later to manage with two people.
Of course where we do not have resources we also go with pragmatic approach ..

However..

The correct access control on PBS of course is crucial, also correct pruning via PBS, so backups won't be deleted by an attacker and is actually configured this way But your tips are very helpful, thank you @Onslow .
At the moment we have only the backups on PBS though, but we are actually "not online & aware" and before going online we'll have them also 3 times, preferable additional one onsite and one offsite, like stated by @Johannes S .

Unfortunately we do not have a third server, so I guess at the beginning we'll make the second VM backup pragmatic once a month as a cold backup to a drive directly via console access, even if it's some kind of "rude" and not automated.. has on the other hand also the bulletproof benefit of that it's even not connected and would be for the absolute worst case.
The other offsite backup I think we'll go with @Johannes S recommendation and deploy on a VPS for offsite backups.

Thank you also for the helpful links, we will go through each of them and check everything.
Additionally I would like to add that we also found this hardening guide, which seems to be helpful (and may help other readers here later on, too).
Be aware we still not verified it completely but we will work through it (and also your provided links here) and report how well what worked out for us.

We'll try to document the whole hardening process of course for future folks and post it here when we finished (probably in a new thread) and nothing breaked

Thank you and kind regards

 

[Johannes S]

The other offsite backup I think we'll go with @Johannes S recommendation and deploy on a VPS for offsite backups.

There are also now Proxmox Partners who offer PBS as a managed cloud service, e.G. tuxis.nl in the netherlands or Inett in Germany. I would expect that other partners (like the ones in North America) would offer something similiar. Their pricing is quite reasonable (20 Euro for 1 TB or 0,02 Euro per GB pay as you grow) so I'm considering to ditch my netcup-hosted vserver for one of them (one service less to maintain myself).

 

[Johannes S]

Unfortunately we do not have a third server, so I guess at the beginning we'll make the second VM backup pragmatic once a month as a cold backup to a drive directly via console access, even if it's some kind of "rude" and not automated.. has on the other hand also the bulletproof benefit of that it's even not connected and would be for the absolute worst case.

This is actually a quite cheap (although manual and thus error-prone) way to have a cheap offsite backup. Buy some external USB Discs and add them as "removable datastores" to your PBS. As long as you also verify the data from time to time you then have another copy which you can also store outside of your infrastrucures location. I met some guy at an event who also has a small design company (just him and two-three part-time-employees), his offsite backup are two external discs. Every week he switches between them for doing a backup from all his office workstations to it. Since one of them is always in his flat and the other one in his companys office it's quite a affordable solution. Another benefit of such an approach is, that it can be easily scaled to have more copies (e.g. two disks for weekly updates, two more for monthly updates etc).

The obvious downside is that they need manual intervention which might get forgotten or missed (for example if both of you get sick or too much else to do).