Proxmox cloud provider for kubernetes

P

Pablo Alcaraz

Member
Jul 6, 2017
53
8
13
55
  • Like
Reactions: Vladimir Bulgaru
Which tool are you hoping to have integration with exactly?

Its true that Google Compute Engine 'gcloud' command allows automatic creation of kubenernets clusters, as does 'kops' for AWS, etc.

But you can make your own kubernetes cluster in a variety of ways: kubeadm, manual installation, using CoreOS/tectonic, using conjure-up/juju ..


There isn't a comparable tool to gcloud or kops to automatically create a kubernetes cluster for you. I agree it would be nice to have, but its not stopping you from using kubernetes on your VMs.
 
  • Like
Reactions: Pablo Alcaraz
I'm trying to use Ubuntu Server 18.04 and k3sup on a Proxmox server. Didn't go well on the first attempt, and for the second attempt, I was hoping to find some documentation on minimum requirement of disk allocation for the master and worker nodes. The default 32GB on each node is probably a waste of storage space for a simple dev/test cluster. https://github.com/alexellis/k3sup
 
I've been using this cloud provider for a while now:
https://github.com/lnxbil/docker-machine-driver-proxmox-ve

Tested and working with RancherOS iso and debian 10 prepared image + cloudinit.

Because of the massive amount of docker images being pulled by customers on their dev/test environments I create at least 50gb disks.
My image has a 10GB disk which also works if I wanted to keep my vm's as small as possible.
 
I’ve successfully used VMs on Ubuntu Server 20.04.1 with 8GB disks now. Could probably go even lower, but I’ve so far only installed one container with ASP.NET Core 3.1 runtime and a very simple app.
 
Hi Cedvan,
My response is still in draft, must have forgot to press post.

cedvan said:
mbosma, how did you add this custom cloud provider to rancher kubernetes engine please?
Click to expand...
I Added the driver using the Rancher webgui under Global > Tools > Drivers > Node Drivers using the Add Node Driver button.
Here you can paste the link to the binary.
 
I think Proxmox would still benefit a lot if they provided a 1st-party rancher driver.
We would prefer getting proper commercial support for this.
Also automatically load-balancing (moving) VMs would be somewhat of a requirement if you constantly have new VMs popping up and shutting down again.

Right now we only use AWS for autoscaling deployments - and if we want to host them on-prem, we would need to go with vmware which I don't really like. Our PVE setup has a lot better performance and is cheaper to maintain than our VMware setup. But I don't know if there is a way around VMware in those use cases.

I think Proxmox is missing out on a big growth market here by not making PVE "cloud native" - Harvester HCI is also slowly beginning to take shape and would be the open-source Linux-based alternative for provisioning K8s VMs. I really hope that PVE doesn't loose its touch with whats evolving in the industry right now, I love PVE and I would love to continue using it in the future.
 
Cookiefamily said:
I think Proxmox is missing out on a big growth market here by not making PVE "cloud native" - Harvester HCI is also slowly beginning to take shape and would be the open-source Linux-based alternative for provisioning K8s VMs. I really hope that PVE doesn't loose its touch with whats evolving in the industry right now, I love PVE and I would love to continue using it in the future.
Click to expand...
I share your sentiment and think PVE should be made "cloud native" as you call it.

A great additional, which I have only recently started playing with, is SDN integration into the GUI.

To have formal kubernetes support would be really great!

As far as auto-creation of clusters is concerned: I suppose it is possible to write a service to do this using LXC templates and the PVE API, not? Has anybody done something like this?
 
@lifeboy I have created that kind of tool. It's open source but keep in mind it's in early stage of development. Maybe will help someone. Currently I'm working on big redesign of app, and big update will be dropped soon.
I have created this project due to lack of easy tool for creating Kubernetes cluster and I don't want to messing up with terraform and ansible. It's worth to mention, currently support only MicroK8S on Ubuntu Cloud image.
https://github.com/dsieradzki/k4prox
 
Last edited:
  • Like
Reactions: ams0
liorokman said:
I've created a Kubernetes cloud manager for Proxmox which allows creating Kubernetes clusters in an isolated SDN VNet.

The cloud manager also implements the load-balancer parts, so creating services of type LoadBalancer works correctly.

https://github.com/liorokman/proxmox-cloud-provider
Click to expand...
How do you manage multi site vlans for the load balancer? How do you integrated to separated phisical networks from each other in a single vlan ?

Thanks
 
navigator said:
How do you manage multi site vlans for the load balancer? How do you integrated to separated phisical networks from each other in a single vlan ?

Thanks
Click to expand...

All of the nodes that are part of the Kubernetes cluster are on the same SDN subnet. The Proxmox SDN implementation does the heavy lifting of isolating that subnet - the VNet needs to be a VXLAN or VLAN VNet so that it is isolated from the any other configured networks.

The loadbalancer is a Proxmox container with two interfaces - one on the K8S subnet and another on some externally reachable subnet. The actual loadbalancing is based on setting up IPVS in the loadbalancer container based on Kubernetes services that are configured as "type: LoadBalancer".
 
dsieradzki said:
@lifeboy I have created that kind of tool. It's open source but keep in mind it's in early stage of development. Maybe will help someone. Currently I'm working on big redesign of app, and big update will be dropped soon.
I have created this project due to lack of easy tool for creating Kubernetes cluster and I don't want to messing up with terraform and ansible. It's worth to mention, currently support only MicroK8S on Ubuntu Cloud image.
https://github.com/dsieradzki/k4prox
Click to expand...
Just came here to thank you, your app works great! Wishlist:

- cli
- spread vms over nodes
- let VMs use DHCP (probably you must rely on qemu-guest-agent then)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back