The best OS fo Kubernetes in Proxmox

[Marx]

Hi
I want to build Kubernetes cluster on top of Proxmox. It's not decided if it'll be k8s, k3s or any other flavor of Kubernetes.
Now I'm looking for best OS for Kubernetes node.

First choice:
- LXC - I tend to use LXC, as it's my private cluster, so security is not a big concern. I also think LXC is better for resources (my cluster is not very powerful)
- full VM

Second choice:
- use normal OS like Debian - as Proxmox is using Debian, it's delivering ready-to-use template, everything will work, but I need to take care of this OS (every node mean another OS to take care of users, passwords, updates etc)
- specific OS for running Kubernetes. I can name a few: Flatcar (as a successor for CoreOS), Talos, K3OS, Bottlerocket-OS, Container Optimized OS, Qubes(?), OpenSUSE Kubic and probably there are more

Can you share your experience with running Kubernetes on top of Proxmox?

 

[davosian]

Hi @Marx,

can you share with us which path you took and how your experience was? I am facing the same challenge right now..

Best,
Dennis

 

[Marx]

I gave up lxc & VM approach and installed k3s alongside proxmox. In fact I'm considering removing proxmox and leaving k3s alone. The only things I use proxmox for are ceph and database storage on it (which is used as k3s backend - lxc HA) and VM with OMV with passed through disks (NAS).
Proxmox adds another layer of complexity, and while I like it as a whole, I don't use its power. Managing stateless containers is so much better then VMs or LXCs.

 

[davosian]

This is valuable insight, thanks Marx.
I was thinking that using Proxmox as a base, I have the flexibility to use both, VMs and Containers, but of course I should also consider the technical debt that comes with more complex systems.

Did you try running VMs inside Kubernetes with KubeVirt?

In Kubernetes, I could also restrict resource usage through limits.

Anyways, you got me thinking...

 

[sinkog]

I'm not just running a kubernet on the proxmox cluster, so the question wasn't whether the extra layer was needed.
i use qemu virtual machines Talos operating system (talos.dev) and it works quite well.
With a suitable script, it is ready in max 15p 3 master + 3 worker machine clusters set up and ready for use.
I haven't found an LXC implementation for Talos yet, but life is beautiful if you are pregnant with challenges ....
It works for the time being, although it is only used in development, testing and demo environments. In a product environment, it has not yet been decided whether we will run on a service provider or our own server