looking for opinion of more knowledgeable ppl on a new instance

[dawidle7]

hi there

im just starting out with my 1st production proxmox instance and id like to ask for your counsel

hardware: i have one old server that i need secured and easy to backup and restore, i also have a newer workstation with ample hd space where i want my backups to go, optionally i want my server environment to grow (under load) to this computer and the cloud (i consider AWS and Azure but open to suggestions, are there european clouds with k8s?)

software: i will deploy a server program onto kubernetes using a private repo and registry, as it is single threaded for now i want to run 2 instances (i got 3 cores, 6 threads); i understand containers are faster and lighter than VMs - but are they equally secure? there were backdoors in the linux VM in the past - are these closed? (i remember it had sth to do with the floppy drive) can i even use kubernetes with you containers? it did not work on docker for me; i need memory unused by one container/VM to be available for other containers/VMs as i have only 16gb of ram and cant easily expand.

backup: i would like to use ZFS - i understand that it allows deltas to be sent to another machine and pretty much i could backup my entire proxmox every hour - is my understanding correct? is there anything that id rather use the backup server for?

firewall: use proxmox firewall or the one in the guest OS (linux)? what gives power/security? i need logs

thats all for now
keenly awaiting your reply

d

 

[aj@root]

Hybrid Cloud

Proxmox doesn't have any native support for integrating with other cloud systems and it doesn't yet support dynamic live-migration load balancing. That may become part of Proxmox VE and Proxmox Datacenter Manager in the future. Dynamic Load Scheduler is on the VE Roadmap at least: https://pve.proxmox.com/wiki/Roadmap

If you've already got a setup with kubernetes that handles hybrid cloud deployment, I'm sure it'll work, but when you consider the cost of cloud services and the extra management complexity vs getting a server, it's probably much cheaper and simpler to buy a used server that's just way more than what you need.

You can get the equivalent of what would be sold as 144 vCPUs (2 cpus x 12 cores x 2 threads x 3ghz) for less than $100 on ebay (for example, a Xeon E5-2687W v4).

Personally, I wouldn't run Proxmox without at least 2 servers where just one server has 2-3x the capacity that I expect to use. I wouldn't use less than 2 servers + qdevice because I want live migrations for maintenance, redundancy, and automatic failover.

LXC vs VM for Kubernetes

As long as you don't change the default settings for LXCs, they are secure. If you enable privileged containers, module loading, fuse, or a variety of other optional privileges, there are ways to jailbreak them.

VMs are more secure in the sense that there are far more hoops to jump through to get to the host - you have to chain more exploits.

You can also live-migrate VMs, which you cannot do with LXCs.

I'd recommend running kubernetes in a VM so that you can have full root kernel access to be able to configure it how you'd like to easily.

If you try to run that in an LXC you're almost certainly going to have to change settings that bend the security model and, if also misconfigured in kubernetes, that could lead to host access, whereas if you misconfigure kubernetes in a VM, the attacker would only get access to the VM. If you run your databases and other common services in other containers (or VMs) then you can keep those isolated from the more privileged kubernetes containers via the VM.

Backup

You could run PBS (Proxmox Backup Server) on a workstation, as long as the Proxmox VE node can access it. Because it targets virtualized guests, PBS uses a block-level backup system that's far more efficient than file-based backup systems, in many cases. It can use HDDs, but recovery can be slow (due to how the block-based backups become fragmented early on). That can be overcome by using an SSD for metadata (or all of it).

In a 2-node setup you can use ZFS replication.

In a single node you could write your own scripts to replicate zfs over ssh, but you wouldn't be able to use the built-in Proxmox tooling.

Also, you generally don't backup the host. Hosts are considered to be throwaway. If something goes wrong, you just reinstall and 10 minutes later you start restoring backups.

There are some things that you might want to backup after initial configuration - such as /etc/network/ (or all of /etc/ with tar --one-file-system for good measure), but just about anything that you run is expected to run in a guest.

PBS doesn't support backing up the host at all.

Firewall

Generally, go with the Proxmox firewall - except for things where pfSense or MikroTik or FortiGate make more sense. Generally it's a good idea to separate the host and guest networks and use a hardware or virtualized router to be the gateway, but from what it sounds like in your case, that might be a bit more config than you're looking for.

The SDN "Simple" network will give you isolation and keep internet access on a single node, but once you progress to a two-node system you'd need to progress to "VLAN" or "EVPN VXLAN" (if you need guests to be able to communicate across nodes). Both of those would require an external router setup.

The Proxmox firewall is API- and scripting-accessible and can be controlled at different scopes - datacenter, node, and guest. There's various logs and log levels and whatnot you you can dig into.

You can use SMTP credentials for whatever your mail provider is (Google, Postmark, Mailgun, etc) to get a variety of notifications, including firewall alerts.

 

[dawidle7]

ok so i finally found some time to move along with this mess

i have created a vm in virt manager and installed proxmox 8.4 on it
1. issue - i need proxmox zfs with 1 disk, setup requires 2 - can i circumvent that?
on that proxmox instance i have created a vm100 which i see immediately uses zvols - nice
2. can i have these zvols of arbitrary size? i do not want to migrate when number of users/data increases
that vm100 has a virtual hard disk and a virtiofs instance that proxmox supplies - it did not automatically create a folder for that virtiofs - why?
3. when i try to debootstrap on that virtiofs after chrooting apt fails to resolve hostnames while ping by name works - i tried twice, same result - does anybody know why? debian irc wasnt very helpful
i also installed proxmox backup server in another vm but it seems a bit more complicated than proxmox ve itself
4. is my understanding correct that both pbs and zfs offer incremental backups but only zfs allows me to backup proxmox configuration by copying /etc?

 

[aj@root]

You may want to break that down into multiple threads on the forum rather than using this thread as a catchall - it's old enough and the title is vague enough that I'm probably the only one watching it - just because I happened to be logged in and saw it when you first posted.

1. Choose RAID0 (striped) and that will allow the single-disk, single-vdev "striped" pool.



2. You can always increase the size, but you can't decrease it. You can also use Thin provisioning with Discard enabled and and run fstrim inside the guest to get it back down to just the used size.



Virtio Folders are managed under Datacenter: Directory Mappings, but Proxmox does not create any by default. What are you seeing exactly?



3. You probably still need to have a resolv.conf in the chroot.

#/etc/resolv.conf

search cluster.example.com
nameserver 1.1.1.1

4. The PVEs aren't meant to be backed up.

Anything shared across the cluster is meant to be done at the Datacenter level and any special-purpose task is intended to be done in a guest.

Something like ansible would be the "backup" for a PVE.

If you were to have a problem with a node, the recommended procedure is to fail it out of the cluster, reinstall, and reconfigure. This is typically much faster than trying to sort through various configurations and determine which to restore and which to regenerate.

That said, yes, you can certainly use ZFS snapshots or rsync on the Host.