Proxmox cloud provider for kubernetes

[Pablo Alcaraz]

Hello

I want to use Proxmox hosts to provide nodes to a kubernetes cluster.

Using VMware, Kubernetes can create new nodes by using a vSphere cloud provider
https://kubernetes.io/docs/getting-started-guides/vsphere/#configuring-vsphere-cloud-provider

I believe Proxmox does not provide a kubernetes cloud provider. Does anybody know if there are a 3rd party equivalent?

Do anyone configured a kubernetes cluster using proxmox hosts?

 

[locusofself]

Which tool are you hoping to have integration with exactly?

Its true that Google Compute Engine 'gcloud' command allows automatic creation of kubenernets clusters, as does 'kops' for AWS, etc.

But you can make your own kubernetes cluster in a variety of ways: kubeadm, manual installation, using CoreOS/tectonic, using conjure-up/juju ..


There isn't a comparable tool to gcloud or kops to automatically create a kubernetes cluster for you. I agree it would be nice to have, but its not stopping you from using kubernetes on your VMs.

 

[eloekset]

I'm trying to use Ubuntu Server 18.04 and k3sup on a Proxmox server. Didn't go well on the first attempt, and for the second attempt, I was hoping to find some documentation on minimum requirement of disk allocation for the master and worker nodes. The default 32GB on each node is probably a waste of storage space for a simple dev/test cluster. https://github.com/alexellis/k3sup

 

[mbosma]

I've been using this cloud provider for a while now:
https://github.com/lnxbil/docker-machine-driver-proxmox-ve

Tested and working with RancherOS iso and debian 10 prepared image + cloudinit.

Because of the massive amount of docker images being pulled by customers on their dev/test environments I create at least 50gb disks.
My image has a 10GB disk which also works if I wanted to keep my vm's as small as possible.

 

[eloekset]

I’ve successfully used VMs on Ubuntu Server 20.04.1 with 8GB disks now. Could probably go even lower, but I’ve so far only installed one container with ASP.NET Core 3.1 runtime and a very simple app.

 

[cedvan]

Hello,

I've been using this cloud provider for a while now:
https://github.com/lnxbil/docker-machine-driver-proxmox-ve

mbosma, how did you add this custom cloud provider to rancher kubernetes engine please?
I readed this page "https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/custom/", but I can't find more information on how to configure the proxmox driver on the web

​

Waiting for your return,

Thanks in advance

 

[cedvan]

Up ?

 

[mbosma]

Hi Cedvan,
My response is still in draft, must have forgot to press post.

mbosma, how did you add this custom cloud provider to rancher kubernetes engine please?

I Added the driver using the Rancher webgui under Global > Tools > Drivers > Node Drivers using the Add Node Driver button.
Here you can paste the link to the binary.

 

[cedvan]

Yes, i found this today ^^. Thank you !!!

 

[Cookiefamily]

I think Proxmox would still benefit a lot if they provided a 1st-party rancher driver.
We would prefer getting proper commercial support for this.
Also automatically load-balancing (moving) VMs would be somewhat of a requirement if you constantly have new VMs popping up and shutting down again.

Right now we only use AWS for autoscaling deployments - and if we want to host them on-prem, we would need to go with vmware which I don't really like. Our PVE setup has a lot better performance and is cheaper to maintain than our VMware setup. But I don't know if there is a way around VMware in those use cases.

I think Proxmox is missing out on a big growth market here by not making PVE "cloud native" - Harvester HCI is also slowly beginning to take shape and would be the open-source Linux-based alternative for provisioning K8s VMs. I really hope that PVE doesn't loose its touch with whats evolving in the industry right now, I love PVE and I would love to continue using it in the future.

 

[lifeboy]

I think Proxmox is missing out on a big growth market here by not making PVE "cloud native" - Harvester HCI is also slowly beginning to take shape and would be the open-source Linux-based alternative for provisioning K8s VMs. I really hope that PVE doesn't loose its touch with whats evolving in the industry right now, I love PVE and I would love to continue using it in the future.

I share your sentiment and think PVE should be made "cloud native" as you call it.

A great additional, which I have only recently started playing with, is SDN integration into the GUI.

To have formal kubernetes support would be really great!

As far as auto-creation of clusters is concerned: I suppose it is possible to write a service to do this using LXC templates and the PVE API, not? Has anybody done something like this?

 

[dsieradzki]

@lifeboy I have created that kind of tool. It's open source but keep in mind it's in early stage of development. Maybe will help someone. Currently I'm working on big redesign of app, and big update will be dropped soon.
I have created this project due to lack of easy tool for creating Kubernetes cluster and I don't want to messing up with terraform and ansible. It's worth to mention, currently support only MicroK8S on Ubuntu Cloud image.
https://github.com/dsieradzki/k4prox