Five Newbie Questions

[Daedalus_Dance]

Hi Humans,

Proxmox newbie here – got a few questions I hope someone / some people can help answer about a single host deployment. I’m comfortable with Linux, bash and other things from managing multiple VMs in the past but moving up to the hypervisor level now. Disclaimer: Please tell me anything I might be mistaken about as I’d rather know now than find out later after a mistake – you can be as insulting as you like

Few things to get out the way:

Why Proxmox? Budget is important (I’d rather spend it on more hardware) so VMware and such wasn’t considered. I considered Xenserver, oVirt and Proxmox. Initially looked at XenServer because I liked XO but then the 7.3 Update and price gouging stopped that entirely. I’m more used to Centos / RHEL systems so looked then seriously at oVirt but found single host management to be a bit silly because you essentially need to run a VM just to manage the hypervisor – this could however be an advantage I’ve missed? Proxmox seems to support single-host deployments more readily. Also oVirt seemed to have a reputation as being a little unstable as RHEV’s testing bed. I was concerned about whether Proxmox is “business ready” but a sys-admin who currently uses it for their business told me it just works for them and they’ve not had any trouble.

Intended Config: Will be 2x E5-2603v4, 64GB RAM, 4x 480gb SSD in RAID 10, Redundant PSUs. I’m intending to use ZFS for the file system as I’ve been told it’s great for single host set ups on proxmox and obviously supports snapshotting etcetera. The machine will be running probably max 10-15 VMs including one or two with quite heavy webhosting usage (hence the SSDs)

Questions:

1. I would like the host to be installed and configured before I send it to a datacentre for colocation when I will be assigned IPs. A second option is to install the software at KVM-over-IP but this option precludes testing the host before shipping. How easy a process is it to install the hypervisor first and then assign the IPs to the VMs and Hypervisor?
   
   
2. Backups. I’m exceptionally paranoid about disaster recovery backups and would want to send VM backups to a remote location (probably a third party) – What’s the best way to send VM backups to a remote bucket (AWS, Backblaze, my own NAS, whatever)?
   
   
3. Linked to question two, security. I couldn’t find a definitive guide to hardening Proxmox anywhere. Is it just like hardening any normal Ubuntu/Debian OS? I’m comfortable working with VPNs, minimal installs and SSH keys etcetera but there’s still something “just not right” about the remote management of hypervisors?
   
   
4. Logging and monitoring. What do people use for logging and monitoring?
   
   
5. Any gotchas or insights from working with Proxmox I should know about, particularly around initial setup? It’s a really general question but its often these ones you get the best insights from
   

Thank you for anyone who takes the time to answer any of these questions!!!

 

[dcsapak]

General Remark to your setup:

4x 480gb SSD in RAID 10

I’m intending to use ZFS

64GB RAM

Make sure you do not use a raid controller for zfs (it does not like that, some users reported issues with boot,etc)
also use enterprise grade ssds or you will probably not be happy with your performance
note that by default, zfs reserves up to 50% of your ram for its ARC (this can be configured, and depending on your data amount/usage patterns, you can get away with less)

I would like the host to be installed and configured before I send it to a datacentre for colocation when I will be assigned IPs. A second option is to install the software at KVM-over-IP but this option precludes testing the host before shipping. How easy a process is it to install the hypervisor first and then assign the IPs to the VMs and Hypervisor?

The installer requires you to enter an ip address (it tries to guess based on dhcp), but this can be changed afterwards via the gui or /etc/network/interfaces (do not forget to also change /etc/hosts)

Backups. I’m exceptionally paranoid about disaster recovery backups and would want to send VM backups to a remote location (probably a third party) – What’s the best way to send VM backups to a remote bucket (AWS, Backblaze, my own NAS, whatever)?

The integrated backup mechanism (vzdump) can target any directory, so as long as you can use the target as a directory you can backup there (e.g. an nfs share)

Linked to question two, security. I couldn’t find a definitive guide to hardening Proxmox anywhere. Is it just like hardening any normal Ubuntu/Debian OS? I’m comfortable working with VPNs, minimal installs and SSH keys etcetera but there’s still something “just not right” about the remote management of hypervisors?

in general, yes normal debian instructions apply, but be careful when you want to have multiple proxmox installations in a cluster, we rely on ssh/https/multicast traffic, so a few caveats apply there
we have an integrated firewall, which is configurable via the webgui

Logging and monitoring. What do people use for logging and monitoring?

i personally (in my homelab) use the external metric server export and a few self written scripts, but this question should better be answered by the community
(although since it is just debian underneath, you could use any monitoring system which supports debian)

Any gotchas or insights from working with Proxmox I should know about, particularly around initial setup? It’s a really general question but its often these ones you get the best insights from

a good start would probably the documentation, which is integrated in the webgui (matching the installed version) but also available here (for the newest version): https://pve.proxmox.com/pve-docs/

 

[Rhinox]

Ad.3: From my understanding, Proxmox can not be hardened enough to have its management-port (and kvm-port) facing internet with no other protection. It was never designed to protect itself alone (this holds true for many hypervisors, i.e. esxi, xen, etc). If you check any common unix/linux hardening guide, there are a lot of things that can not be done on Proxmox without breaking its functionality...

If you are going to have Proxmox in colocation-datacenter, I recommend to put some firewall in front of Proxmox management and KVM-port. Do not use firewall as virtual appliance, use dedicated hw-one (i.e. some small box running pfsense or mikrotik-board, etc)...

And as a bare minimum, I recommend to have at least 3x NIC: one for kvm if your server has it (and it should, if you want to put it in colocation datacenter), one for proxmox-management, and one for all VPS. If any of your vps get ddos-ed, you can still access at least Proxmox management...

 

[Daedalus_Dance]

General Remark to your setup...

Thank you a lot for all this helpful information! Indeed, I saw the issues with the RAID controllers in the documentation so just removed it from the hardware list. SSD's will be Intel DC series. My main concern with the hardware is the processors and whether I'm being too cheap on them going for the lowest E5s... Everyone seems to be saying RAM and I/O is the bottleneck though.

On question one, from what you have said it seems setting it up first time KVM-Over-IP with a boot USB would be easier if it'll just configure the network all itself.

Ad.3: From my understanding...

Thank you for this as well! Yes, the motherboard has integrated KVM. A third NIC would be useful in DDoS instance and I hadn't thought of this. Your point about hypervisors simply not being intended to protect themselves alone is important.

My initial intentions with security were to restrict access Proxmox to my VPN and dedicated IP, as well as software firewalls and the standard debian OS hardening (disabling root for SSH, strong keys, only installing the packages required). A dedicated hardware firewall would definitely help alleviate my concerns though and provide another layer - I'll look into this .