looking for opinion of more knowledgeable ppl on a new instance

[dawidle7]

hi there

im just starting out with my 1st production proxmox instance and id like to ask for your counsel

hardware: i have one old server that i need secured and easy to backup and restore, i also have a newer workstation with ample hd space where i want my backups to go, optionally i want my server environment to grow (under load) to this computer and the cloud (i consider AWS and Azure but open to suggestions, are there european clouds with k8s?)

software: i will deploy a server program onto kubernetes using a private repo and registry, as it is single threaded for now i want to run 2 instances (i got 3 cores, 6 threads); i understand containers are faster and lighter than VMs - but are they equally secure? there were backdoors in the linux VM in the past - are these closed? (i remember it had sth to do with the floppy drive) can i even use kubernetes with you containers? it did not work on docker for me; i need memory unused by one container/VM to be available for other containers/VMs as i have only 16gb of ram and cant easily expand.

backup: i would like to use ZFS - i understand that it allows deltas to be sent to another machine and pretty much i could backup my entire proxmox every hour - is my understanding correct? is there anything that id rather use the backup server for?

firewall: use proxmox firewall or the one in the guest OS (linux)? what gives power/security? i need logs

thats all for now
keenly awaiting your reply

d

 

[aj@root]

Hybrid Cloud

Proxmox doesn't have any native support for integrating with other cloud systems and it doesn't yet support dynamic live-migration load balancing. That may become part of Proxmox VE and Proxmox Datacenter Manager in the future. Dynamic Load Scheduler is on the VE Roadmap at least: https://pve.proxmox.com/wiki/Roadmap

If you've already got a setup with kubernetes that handles hybrid cloud deployment, I'm sure it'll work, but when you consider the cost of cloud services and the extra management complexity vs getting a server, it's probably much cheaper and simpler to buy a used server that's just way more than what you need.

You can get the equivalent of what would be sold as 144 vCPUs (2 cpus x 12 cores x 2 threads x 3ghz) for less than $100 on ebay (for example, a Xeon E5-2687W v4).

Personally, I wouldn't run Proxmox without at least 2 servers where just one server has 2-3x the capacity that I expect to use. I wouldn't use less than 2 servers + qdevice because I want live migrations for maintenance, redundancy, and automatic failover.

LXC vs VM for Kubernetes

As long as you don't change the default settings for LXCs, they are secure. If you enable privileged containers, module loading, fuse, or a variety of other optional privileges, there are ways to jailbreak them.

VMs are more secure in the sense that there are far more hoops to jump through to get to the host - you have to chain more exploits.

You can also live-migrate VMs, which you cannot do with LXCs.

I'd recommend running kubernetes in a VM so that you can have full root kernel access to be able to configure it how you'd like to easily.

If you try to run that in an LXC you're almost certainly going to have to change settings that bend the security model and, if also misconfigured in kubernetes, that could lead to host access, whereas if you misconfigure kubernetes in a VM, the attacker would only get access to the VM. If you run your databases and other common services in other containers (or VMs) then you can keep those isolated from the more privileged kubernetes containers via the VM.

Backup

You could run PBS (Proxmox Backup Server) on a workstation, as long as the Proxmox VE node can access it. Because it targets virtualized guests, PBS uses a block-level backup system that's far more efficient than file-based backup systems, in many cases. It can use HDDs, but recovery can be slow (due to how the block-based backups become fragmented early on). That can be overcome by using an SSD for metadata (or all of it).

In a 2-node setup you can use ZFS replication.

In a single node you could write your own scripts to replicate zfs over ssh, but you wouldn't be able to use the built-in Proxmox tooling.

Also, you generally don't backup the host. Hosts are considered to be throwaway. If something goes wrong, you just reinstall and 10 minutes later you start restoring backups.

There are some things that you might want to backup after initial configuration - such as /etc/network/ (or all of /etc/ with tar --one-file-system for good measure), but just about anything that you run is expected to run in a guest.

PBS doesn't support backing up the host at all.

Firewall

Generally, go with the Proxmox firewall - except for things where pfSense or MikroTik or FortiGate make more sense. Generally it's a good idea to separate the host and guest networks and use a hardware or virtualized router to be the gateway, but from what it sounds like in your case, that might be a bit more config than you're looking for.

The SDN "Simple" network will give you isolation and keep internet access on a single node, but once you progress to a two-node system you'd need to progress to "VLAN" or "EVPN VXLAN" (if you need guests to be able to communicate across nodes). Both of those would require an external router setup.

The Proxmox firewall is API- and scripting-accessible and can be controlled at different scopes - datacenter, node, and guest. There's various logs and log levels and whatnot you you can dig into.

You can use SMTP credentials for whatever your mail provider is (Google, Postmark, Mailgun, etc) to get a variety of notifications, including firewall alerts.