[SOLVED] Cluster wide networking management with SDN

[voriaz]

Hi,

I just want to know how do you guys manage the networking for big proxmox clusters ?

In my case I need to have multiple external networks for my VMs. For that, I create a dedicated vlan on my network switches and then configure a linux vlan interface on top of my linux bond and then add a linux bridge to connect my VMs to this network on PVE hosts. Attached is an example setup diagram.
The only problem is that I need to create all these interfaces manually on each host.
I'm wondering if there is a way to manage networks globally at cluster level instead of by host ?

I tried to play with the SDN feature of PVE, but it seams to be more relevant for internal networks between PVE hosts and cannot go outside on physical switches, but maybe I'm wrong.
Any thoughts ?


BR,

Antoine.

 

[spirit]

The sdn way
-------------------
on each server local network, keep only bond0 + vmbr0.

Then, on the sdn, create a vlan zone with "vmbr0" as bridge option.

Then create 1vnet (for example : vnet10) with tag=10 , and 1vnet "vnet20" with tag=20

Apply sdn config to push vnets on all nodes

configure vm to use vnet (without any vlan tag on the vm)

That's all



Another way:
-------------------
on each server local network, keep only bond0 + vmbr0.
on the vm nic : choose vmbr0 + tag = 10 (proxmox will create in background a bond0.10 + vmbr0v10 bridge)

 

[voriaz]

Hi, and thanks for your reply.

This is what came to my mind after writing this post (SDN way). I tested this setup yesterday on my cluster and it's working perfectly.

Thanks.

 

[JoshoSAI]

Hi spirit
I've been looking at how best to setup a PVE 8.1 node (as a first PVE for a pending cluster) with SDN and have found this rather perplexing... until I read this post!

Can I please ask if this process reads correct for a slightly more complex hardware setup from voriaz's setup (sorry voriaz)

The PVE node has an onboard NIC eno1 and a PCIe quad NIC; en3s0f0, en3s0f1, en3s0f2, en3s0f3

Using the server view on the single PVE node (not Datacenter) should I create a bond0 from en3s0f0, en3s0f1, en3s0f2, en3s0f3
then add a vmbr1 connected to bond0?



Once the above is applied to the pve node, move to the Datacenter and setup SDN and setup VLAN zones using vmbr1




Is this typically where the configuration between the PVE node using "Network" and the Datacenter's SDN?

What should be configured on the PVE node versus what should be configured on SDN is where I've found this extremely confusing.

 

[spirit]

sdn configuration is to create vm networks globally. (you can do it manually locally node by node if you want).

Local configuration (bond, interfaces, management ip,...) is done node by node.
in a cluster, sdn is configured at datacenter level, in /etc/pve/, you need network local first to be able to reach the cluster. Chicken/egg problem.

About the vlan zone "bridge", I have implemented like this because some users want to be able to configure it on both sdn && local. (migration path,...), and also management ip need to be done on the bridge, if the physical interfaces in slave of a bridge.
Also, this allow to have a common "alias" for all nodes, as if physical interface was used as option, it could be different between nodes.