[SOLVED] Cluster wide networking management with SDN

voriaz

Member
Mar 2, 2022
5
1
8
27
Hi,

I just want to know how do you guys manage the networking for big proxmox clusters ?

In my case I need to have multiple external networks for my VMs. For that, I create a dedicated vlan on my network switches and then configure a linux vlan interface on top of my linux bond and then add a linux bridge to connect my VMs to this network on PVE hosts. Attached is an example setup diagram.Example NetworkDiagram.png
The only problem is that I need to create all these interfaces manually on each host.
I'm wondering if there is a way to manage networks globally at cluster level instead of by host ?

I tried to play with the SDN feature of PVE, but it seams to be more relevant for internal networks between PVE hosts and cannot go outside on physical switches, but maybe I'm wrong.
Any thoughts ?


BR,

Antoine.
 
The sdn way
-------------------
on each server local network, keep only bond0 + vmbr0.

Then, on the sdn, create a vlan zone with "vmbr0" as bridge option.

Then create 1vnet (for example : vnet10) with tag=10 , and 1vnet "vnet20" with tag=20

Apply sdn config to push vnets on all nodes

configure vm to use vnet (without any vlan tag on the vm)

That's all :)



Another way:
-------------------
on each server local network, keep only bond0 + vmbr0.
on the vm nic : choose vmbr0 + tag = 10 (proxmox will create in background a bond0.10 + vmbr0v10 bridge)
 
Hi, and thanks for your reply.

This is what came to my mind after writing this post (SDN way). I tested this setup yesterday on my cluster and it's working perfectly.

Thanks.
 
  • Like
Reactions: spirit
Hi spirit
I've been looking at how best to setup a PVE 8.1 node (as a first PVE for a pending cluster) with SDN and have found this rather perplexing... until I read this post!

Can I please ask if this process reads correct for a slightly more complex hardware setup from voriaz's setup (sorry voriaz)

The PVE node has an onboard NIC eno1 and a PCIe quad NIC; en3s0f0, en3s0f1, en3s0f2, en3s0f3

Using the server view on the single PVE node (not Datacenter) should I create a bond0 from en3s0f0, en3s0f1, en3s0f2, en3s0f3
then add a vmbr1 connected to bond0?

1705912254274.png

Once the above is applied to the pve node, move to the Datacenter and setup SDN and setup VLAN zones using vmbr1
1705911119219.png

1705911264330.png

Is this typically where the configuration between the PVE node using "Network" and the Datacenter's SDN?

What should be configured on the PVE node versus what should be configured on SDN is where I've found this extremely confusing.
 

Attachments

  • 1705908581721.png
    1705908581721.png
    4.6 KB · Views: 50
  • 1705908596350.png
    1705908596350.png
    69.3 KB · Views: 50
  • 1705910891233.png
    1705910891233.png
    69.5 KB · Views: 64
sdn configuration is to create vm networks globally. (you can do it manually locally node by node if you want).

Local configuration (bond, interfaces, management ip,...) is done node by node.
in a cluster, sdn is configured at datacenter level, in /etc/pve/, you need network local first to be able to reach the cluster. Chicken/egg problem.

About the vlan zone "bridge", I have implemented like this because some users want to be able to configure it on both sdn && local. (migration path,...), and also management ip need to be done on the bridge, if the physical interfaces in slave of a bridge.
Also, this allow to have a common "alias" for all nodes, as if physical interface was used as option, it could be different between nodes.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!