Proxmox + Kubernetes + Ceph

[landagen]

Can Proxmox servers have Kubernetes installed along side it to give you a Proxmox cluster plus a bare metal Kubernetes cluster. And then separately can you run a Ceph cluster on the same servers. In theory this would allow you to efficiently utilize resources while providing HA. I know it would take a lot of bandwidth especially with Ceph storage, but you would also have multiple HA clusters without having to have 3 sets of machines for each cluster type. You could in theory scale out each cluster separately by adding machines installed only with the components required. My questions are:

1. Would they place nice together or would there be conflicts in how things need to configured?
2. Would this be recommended for a small HA production setup if you want to be able to have a Kubernetes cluster and a small amount of VMs for things that don't run well on Kubernetes
3. What would be the minimum network speed for these to work? I know Ceph is quite greedy when it comes to bandwidth.
4. Would this be better or worse for performance/maintenance/scalability than having Proxmox + Kubernetes on VM

I would test this myself, but I don't currently have the hardware needed.

 

[dherzig]

Hi landagen,

# DISCLAIMER: I've never even considered running k8s on baremetal, except for desktop snap installs for quick evaluation of software that comes shipped ready for helm only.

1. Would they place nice together or would there be conflicts in how things need to configured?
2. Would this be recommended for a small HA production setup if you want to be able to have a Kubernetes cluster and a small amount of VMs for things that don't run well on Kubernetes
3. What would be the minimum network speed for these to work? I know Ceph is quite greedy when it comes to bandwidth.

While I guess that a setup of this kind is technically possible, I imagine maintenance to be wild at least. Just think of the network interfaces created by k8s. I'm mentioning this, as you'll not become happy with just one NIC on your host, if you plan to use ceph [0] [1].

If you run k8s on PVE virtual machines you'll be giving away minimal overhead, while keeping maintenance and scalability doable. You could quite reasonably run various k8s clusters alongside each other, and scaling k8s would be like adding a new vm with PVE-controlled hardware specs (or provisioning the existing vms with more CPU power or RAM via PVE, properly isolated on kernel-level). I'm not even talking about the niceness of snapshots during the initial k8s node creation.

HA will in the end be a matter of physical machine count (clearly one can run a multinode-k8s cluster on a single bare-metal server using virtualization techniques, but this is more a matter of configuration taste, as pulling the plug will still mean pulling the plug).

With careful planning (like placing a masterplane-node of each of your k8s clusters onto each PVE-host in a set of three, well adjusted ceph on highspeed network and server-grade drives, separate networks for PVE-inter-cluster communication), one could likely reach a system where sudden physical outtage of one server would not reach any of k8s-administered software services.

Best,
Daniel

[0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#networks
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_recommendations_for_a_healthy_ceph_cluster

 

[landagen]

Hi landagen,

# DISCLAIMER: I've never even considered running k8s on baremetal, except for desktop snap installs for quick evaluation of software that comes shipped ready for helm only.



While I guess that a setup of this kind is technically possible, I imagine maintenance to be wild at least. Just think of the network interfaces created by k8s. I'm mentioning this, as you'll not become happy with just one NIC on your host, if you plan to use ceph [0] [1].

If you run k8s on PVE virtual machines you'll be giving away minimal overhead, while keeping maintenance and scalability doable. You could quite reasonably run various k8s clusters alongside each other, and scaling k8s would be like adding a new vm with PVE-controlled hardware specs (or provisioning the existing vms with more CPU power or RAM via PVE, properly isolated on kernel-level). I'm not even talking about the niceness of snapshots during the initial k8s node creation.

HA will in the end be a matter of physical machine count (clearly one can run a multinode-k8s cluster on a single bare-metal server using virtualization techniques, but this is more a matter of configuration taste, as pulling the plug will still mean pulling the plug).

With careful planning (like placing a masterplane-node of each of your k8s clusters onto each PVE-host in a set of three, well adjusted ceph on highspeed network and server-grade drives, separate networks for PVE-inter-cluster communication), one could likely reach a system where sudden physical outtage of one server would not reach any of k8s-administered software services.

Best,
Daniel

[0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#networks
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_recommendations_for_a_healthy_ceph_cluster

Thanks for the information. Another thought I had to your point is that having it as a VM k8s cluster probably improves security since a hypervisor should give you better separation than docker containers. Having a mixture could compromise your Proxmox servers. Maybe a better way would be to run the master nodes as virtual and the worker nodes as bare metal (on their own machine). You could start the worker nodes as VM's and then later expand into bare metal worker nodes.