Hello,
I’m starting to build an environment for my software agency. We plan to run a K8 cluster and deploy development environments. I’m familiar with deploying infrastructure in the cloud primarily so I’m excited about deploying infrastructure on-prem.
I suspect that a lot of the things that I’m going to ask are bazooka for what I need, but, I want to learn more about networking and Proxmox so I take it as a learning opportunity as well.
I’m trying to replicate an environment similar to what I’m used to in AWS.
Some context around infrastructure first. I’m experimenting with a two-host cluster. Hosts are connected via a switch between each other and I have a conventional home router with a single network.
This is what I have in my mind.
I would like to have a subnet for my k8 nodes. This subnet can spawn different hosts in my Proxmox cluster. Also, I would like to have a public subnet (which can also spawn multiple hosts in the Proxmox cluster). The k8 nodes would access the internet via the public subnet.
After doing some reading and playing around with Proxmox SDN, I was able to get the k8 nodes talking between each other across different hosts by using a subnet backed by a VXLAN zone. One problem that I have, is that k8 nodes can’t talk to the internet. After spending some time looking into the problem and reading posts, I concluded that I need some kind of router in my k8 subnet to route the traffic to the internet. Thinking of my desired state, I think the k8 nodes need to be able to route traffic to a node in the public subnet. That node in the public subnet can then have a vmbr0 interface and route traffic to the internet (I did some testing by attaching the vmbr0 interface to one of the k8 nodes and it worked).
After doing some extra reading, I understood that, for my k8 VXLAN subnet to talk with the public VXLAN subnet, I need to use EVPN. I tried doing some experimentation but I wasn’t able to get that working.
I’m trying to get some guidance here and understand if I’m on the right track. As I said at the beginning, this is my first time using Proxmox and doing on-prem infra so I can be making very dumb questions
.
I’m starting to build an environment for my software agency. We plan to run a K8 cluster and deploy development environments. I’m familiar with deploying infrastructure in the cloud primarily so I’m excited about deploying infrastructure on-prem.
I suspect that a lot of the things that I’m going to ask are bazooka for what I need, but, I want to learn more about networking and Proxmox so I take it as a learning opportunity as well.
I’m trying to replicate an environment similar to what I’m used to in AWS.
Some context around infrastructure first. I’m experimenting with a two-host cluster. Hosts are connected via a switch between each other and I have a conventional home router with a single network.
This is what I have in my mind.
I would like to have a subnet for my k8 nodes. This subnet can spawn different hosts in my Proxmox cluster. Also, I would like to have a public subnet (which can also spawn multiple hosts in the Proxmox cluster). The k8 nodes would access the internet via the public subnet.
After doing some reading and playing around with Proxmox SDN, I was able to get the k8 nodes talking between each other across different hosts by using a subnet backed by a VXLAN zone. One problem that I have, is that k8 nodes can’t talk to the internet. After spending some time looking into the problem and reading posts, I concluded that I need some kind of router in my k8 subnet to route the traffic to the internet. Thinking of my desired state, I think the k8 nodes need to be able to route traffic to a node in the public subnet. That node in the public subnet can then have a vmbr0 interface and route traffic to the internet (I did some testing by attaching the vmbr0 interface to one of the k8 nodes and it worked).
After doing some extra reading, I understood that, for my k8 VXLAN subnet to talk with the public VXLAN subnet, I need to use EVPN. I tried doing some experimentation but I wasn’t able to get that working.
I’m trying to get some guidance here and understand if I’m on the right track. As I said at the beginning, this is my first time using Proxmox and doing on-prem infra so I can be making very dumb questions
.