SDN VXLAN for private network in a cluster - how to configure properly?

stevops

New Member
Aug 7, 2022
2
0
1
Hi Proxmoxers out there ;)

here is an abstraction of what I finally want to achieve in general:
Untitled Diagram.drawio.png

That means:
  • There is a cluster of at least two PVEs behind a firewall managed by PVE
    • An actor should be able to access the PVEs via SSH
  • Within the cluster I want to have an internal network
    • In general instances in the internal network should be hidden to the outside world (incoming traffic) but should be able to communicate with the outside world (later on only to linux update servers etc. but no restrictions right now at the moment)
    • The instances of the internal network are spread all over the nodes and might be migrated from node to node
    • An admin and other people should be able to access a reverse proxy that would also serve content from "More Services"
    • An admin should be able to access a vpn server to ultimately connect to instances in the internal network via SSH or other protocol
Then:
  • Giving the admin access to the PVE instances is straight forward using the PVE datacenter and PVE node firewall.
  • Building the internal network crossing multiple PVE instances I came across SDNs and more specific VXLANs. Because seperate Linux Bridges on every node would overcomplicate the routing configuration.
    • Enabling SDN and creating a VXLAN without specific configuration was simple so far.
    • But how does the VXLAN/VNet/PVE nodes/VMs need to be configured that the above scenario applies?
      • I would be happy if there is a solution that is independent of an additional gateway/router. It would be the best in my eyes if all the routing can be enabled in the datacenter/node level.
      • How can incoming traffic from the internet be routet to the internal networks specific instances?
      • How can outgoing traffic from the internal network be routet to the internet?
      • Do I need a Subnet within the VNet?
      • ...

Please tell me if I need to provide more information. I think more details of what I tried might be misleading at the first place so I want to keep the more abstract view to my problem if that is ok.
 
Last edited:

guletz

Famous Member
Apr 19, 2017
1,581
255
103
Brasov, Romania
Hi,

Nice question. Even myself I was thinking for something like this, but not exactly in the same way.... I am only scratch this ideea...: My desired concept / ideea could be :

- I want to keep PMX nodes only with minimum additional non-Proxmox packages as it is possible, so the ideea of SDN on top of PMX, is not what I would like to do(not 100% excluded)
- I would like to do VXLAN on my network devices(routers/switches, in my case all of then can do it), because I consider it is easier, and more safe(and I think that a mistake on CDN, could affect all my PMX nodes, but will not be the case using VXLAN on routers/switches, maybe I am wrong, or I over-estimate my confidence ....)
- regarding routing I would like to use ospf, it is more simple, it is dynamic, and could have redundant routing path

I do not start to do anything(only some draft drawings on paper, reading some documentation about how VXLAN can be donne on linux), but in the near future, I would like to spend more time on this concept.



Good luck / Bafta !
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
But how does the VXLAN/VNet/PVE nodes/VMs need to be configured that the above scenario applies?
  • I would be happy if there is a solution that is independent of an additional gateway/router. It would be the best in my eyes if all the routing can be enabled in the datacenter/node level.
  • How can incoming traffic from the internet be routet to the internal networks specific instances?
  • How can outgoing traffic from the internal network be routet to the internet?
  • Do I need a Subnet within the VNet?

you need bgp-evpn vxlan if you want to do routing between differents vxlan. (and each proxmox node is an anycast gateway for each vnet, and you need to define subnets with gateway ip).

exit-nodes need to be cofigure in evpn zone, to forward traffic to real network.
 

stevops

New Member
Aug 7, 2022
2
0
1
Hi @spirit , thank you for answering. You wrote that bgp-evpn is neccessary if I want to do routing between different VxLANs. As far as I understand I would like to avoid multiple VxLANs. One VxLAN is at least enough to connect the 10.0.0.x instances (they can ping each other in the same VxLAN/VNet).
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
Hi @spirit , thank you for answering. You wrote that bgp-evpn is neccessary if I want to do routing between different VxLANs. As far as I understand I would like to avoid multiple VxLANs. One VxLAN is at least enough to connect the 10.0.0.x instances (they can ping each other in the same VxLAN/VNet).
I said that , because you said

I would be happy if there is a solution that is independent of an additional gateway/router. It would be the best in my eyes if all the routing can be enabled in the datacenter/node level.

I don't known where is your current gateway ? (the clusterfw on the schema ?).
if yes, you need to route real world && vxlan here with 2 nics + nat, no need to evpn, only a static vxlan.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!