proxmox network setup for kubernetes

R

roninhockley

Member
Nov 17, 2013
16
0
21
Im sure this question has been asked a thousand times, but some of the pieces I have found still come up short.

I have 3 proxmox nodes in a cluster, sitting in a data center. I want to use these 3 nodes to run a k8s cluster, with the cluster spread among all three nodes. The idea is to have 3 masters, each running on one of the nodes. along with etcd as well, for hardware redundancy.

The challenge here is the networking. I would like to have all the vm's that make up the cluster assigned private ip's. so I am looking at a vmbr1 bridge on each node, with the private ip configured. Then put each k8s node on that bridge. Along with a dhcp server to assign them ip's.

The question is, how will the k8s nodes/pods communicate across the proxmox nodes with other pods that are stuck in isolated networks. How do I forward traffic coming into a node, to the appropriate pod?

If I had a public IP I would just set up an HAProxy vm on each node, but without a pub ip that seems pointless.

Im hoping this is not a big problem and that I am overthinking it.
 
Last edited:
Well, if you want the k8s VMs to communicate with each other, you can't place them in fully private networks without also having a fully private physical network. I.e. you have at least two options:
  • Second NIC on each physical host, assign it to vmbr1, connect them together via a physical switch
  • Connect the VMs to vmbr0 (which I assume is connected to a physical NIC that can communicate between the hosts), and simply assign IPs in a different subnet. Add VLANs for better seperation and put your DHCP in the same net.

PS: Think about your DHCP as well, that is a point of failure too - doesn't help to cluster your kubernetes setup but only have a single DHCP server, at least put it in HA or something.
 
The most likely solution after reading further is to use the inbuilt SDN feature of proxmox. There is also Openvswitch, and Im wondering if they are the same underneath. These 3 servers are not even in the same physical subnet, so SDN will be the only means to architect anything really.

I dont have control over anything beyond what is installed on these servers. The switches etc are managed by the data center.

Each server has 2 nics, and they are currently in a bond setup, but I cannot confirm how well this is supported in the physical switches that these servers connect to. However, given the capabilities of something like the EVPN controller, I dont see any reason to change the bond and split up the NICs when I can just use the SDN vlans to segment the vms.

The SDN setup seems pretty straightforward, but oddly I still am not sure how to handle the incoming traffic.

Some iptables rules applied to the bond0 I guess? I have not done iptables stuff in a while. This part of the equation is still unclear to me.
It seems like that inbound routing should be covered somewhere in the SDN section at the datacenter level of the proxmox GUI.
 
Hi,
I'm running k8s here in vms without any problem. (without the sdn feature, with simple bridge/vlan).

I'm using calico for k8s networking, with ip-ip encapsulation for inter pod networking + metallb for service type loadbalancer (peering with bgp with my routers).

I don't think you need something special at proxmox networking side, as k8s use his own overlay network. (until you don't have to do any overlay in k8s, in this case it'll be more complex)
 
Last edited:
spirit said:
Hi,
I'm running k8s here in vms without any problem. (without the sdn feature, with simple bridge/vlan).

I'm using calico for k8s networking, with ip-ip encapsulation for inter pod networking + metallb for service type loadbalancer (peering with bgp with my routers).

I don't think you need something special at proxmox networking side, as k8s use his own overlay network. (until you don't have to do any overlay in k8s, in this case it'll be more complex)
Click to expand...
My main challenge here is that I do not have control over any of the external routers/switches leading into my 3 proxmox hosts. I am having to use private IP's for all vm's. The only option I have is NAT'ing everything out and port forwarding it back in.
Far as I can tell, I need iptables rules on the eth0 adapters for each proxmox node to do the port forwarding.

When you say bgp for your routers, are you talking about physical routers that you have control over?
 
Stefan_R said:
Well, if you want the k8s VMs to communicate with each other, you can't place them in fully private networks without also having a fully private physical network. I.e. you have at least two options:
  • Second NIC on each physical host, assign it to vmbr1, connect them together via a physical switch
  • Connect the VMs to vmbr0 (which I assume is connected to a physical NIC that can communicate between the hosts), and simply assign IPs in a different subnet. Add VLANs for better seperation and put your DHCP in the same net.

PS: Think about your DHCP as well, that is a point of failure too - doesn't help to cluster your kubernetes setup but only have a single DHCP server, at least put it in HA or something.
Click to expand...
Hi Stefan, thanks for replying. I originally wanted to place a switch in the server cabinet and connect the secondary NICs directly to each other. Unfortunately, these 3 proxmox servers are not in the same cabinet, or even the same subnet. At that point I realized I would have to do tunneling via SDN.

I now have the SDN set up at the datacenter level, by following this: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvesdn_setup_example_evpn. from the pve docs website.

Now I just need to configure the port forwarding to allow incoming traffic back in. I assume the gateways I have configured in EVPN are already NAT'ing my traffic out. But I have to have port forwarding to get traffic in. In the proxmox docs it just says this:

Of course you need to add reverse routes for the 10.0.1.0/24 and 10.0.2.0/24 network to node1, node2 on your external gateway, so that the public network can reply back.
Click to expand...
This is where I am now.
 
Last edited:
roninhockley said:
When you say bgp for your routers, are you talking about physical routers that you have control over?
Click to expand...
yes indeed.
Another way is to use metallb with a layer2 vip for loadbalancer. (you just need to be able to send gratuitous arp)

My main challenge here is that I do not have control over any of the external routers/switches leading into my 3 proxmox hosts. I am having to use private IP's for all vm's. The only option I have is NAT'ing everything out and port forwarding it back in.
Click to expand...
for my vms, I'm using public ips inside vms. (if a container need to go outside, it'll be natted through the public ip of the vm)

If you only have port forwarding for inbound traffic, I don't see how you can implemented k8s. (or maybe only for http/https, with a couple of reverse proxy in front of your K8s)
 
roninhockley said:
I assume the gateways I have configured in EVPN are already NAT'ing my traffic out
Click to expand...
By default ,evpn is doing routing only (also for the exit gateway nodes). so you need to defined iptables masquerade rule on your exit gateway nodes if you want to snat.
(The traffic will go out from the evpn network through the physical interface where your default gw is defined, so you just need to add a iptables rule on this interface)
 
roninhockley said:
Im sure this question has been asked a thousand times, but some of the pieces I have found still come up short.

I have 3 proxmox nodes in a cluster, sitting in a data center. I want to use these 3 nodes to run a k8s cluster, with the cluster spread among all three nodes. The idea is to have 3 masters, each running on one of the nodes. along with etcd as well, for hardware redundancy.

The challenge here is the networking. I would like to have all the vm's that make up the cluster assigned private ip's. so I am looking at a vmbr1 bridge on each node, with the private ip configured. Then put each k8s node on that bridge. Along with a dhcp server to assign them ip's.

The question is, how will the k8s nodes/pods communicate across the proxmox nodes with other pods that are stuck in isolated networks. How do I forward traffic coming into a node, to the appropriate pod?

If I had a public IP I would just set up an HAProxy vm on each node, but without a pub ip that seems pointless.

Im hoping this is not a big problem and that I am overthinking it.
Click to expand...
Hi, i have Question: at the bottom level, do you include a hardware raid o shared storage, how is your config at this point. Thanks in advance. I want to go to that escenario, proxmox, kubernetes and terraform, but the selection of the storage drive me crazy.
 
jorgemop said:
Hi, i have Question: at the bottom level, do you include a hardware raid o shared storage, how is your config at this point. Thanks in advance. I want to go to that escenario, proxmox, kubernetes and terraform, but the selection of the storage drive me crazy.
Click to expand...
If you need persistant storage for your containers, you could use ceph inside your k8s with rook.io (https://rook.io/).

It's also possible to manage with root an external ceph cluster (if use ceph in your proxmox, you can dedicated a pool, and rook will create persistant rbd volume for your container)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom