[SOLVED] Is there a cluster bridge or cluster-level networking?

[jptechnical]

Hi, first time post, long time reader. I have question that I can't quite figure out what to search for, as the terms seem pretty common. So here goes.

Is there any kind of cluster-level networking that would allow vms on various nodes to share a separate lan without a physical smart (vlan aware) switch?

I know this sounds like a vlan, but I don't have a spare smart switch to hand and would like to start testing with a dumb (but fast) inexpensive switch.

I have a lab cluster and I would like to test some things that would best be separated from my regular lan. It's a small office (home office where I run my IT biz), and I could just physically put it on it's own switch. But I would like to have the flexibility of just toggling a button here and there to move a VM to a different network segment. I can just set the vlan 999 on the VMs on a single node, but it breaks when I have vms on multiple nodes. Also, I would like to have a pfsense VM giving that private lan a wan.

 

[spirit]

you can use vxlan with the sdn feature.

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

 

[jptechnical]

I am happy to say this worked great... although I did go down a rabbit hole or two. Here is what worked for me, in case anyone else needs to do this.


VXNET on Proxmox for Cluster Private Network

1. Go to **Datacenter** > **SDN** > **Zones** and create a zone
- Add > **VXLAN**
- Give it a name for the **ID**, `vxzone`
- Provide it a list of IP Addresses of the Proxmox Nodes, `10.1.1.61, 10.1.1.62, 10.1.1.63, 10.1.1.64`
- Leave the rest blank
2. Go to **Datacenter** > **SDN** > **Vnets** and create a Vnet
- Name: `vxnet`
- Alias: blank
- Zone `vxzone`
- Tag: `99`
- Leave **VLAN Aware** box unchecked
3. Go to **Datacenter** > **SDN** > **Vnets** > `vxnet` and add a **Subnet**
- Subnet: `10.6.6.0/24`
- Gateway: `10.6.6.1`
- Leave the rest blank
4. Go to **Datacenter** > **SDN** > **Apply**
- Hit the Apply button to make it all happen
5. Create a pfSense (or your favorite router os)
- Put the WAN interface on `vmbr0`
- Put the LAN interface on `vxnet`
- Assign the LAN interface the IP `10.6.6.1` and enable DHCP
- Test that you get a DHCP address on a VM on a different node

 

[Vengance]

Is SDN already considered stable or still experimental?

 

[pille99]

Software Defined Network​

Proxmox Server Solutions GmbH
<support@proxmox.com>
version 7.4, Wed 22 Mar 2023 02:52:09 PM CET



The Software Defined Network (SDN) feature allows you to create virtual networks (VNets) at the datacenter level.

SDN is currently an experimental feature in Proxmox VE. This documentation for it is also still under development. Ask on our mailing lists or in the forum for questions and feedback.

 

[pille99]

i am working with it and works pretty well. as i understand, many here in the forum use it, even on productive proxmox

 

[spirit]

Hi,I'm the main author of the sdn feature. I'm running it in production (100 hosts, 4000 vms, with bgp-evpn).

The code is pretty stable, I don't have seen bug report since a long time. (only new request features)

and I don't plan any breaking changes. (as a lot of users already using it)

It's still experimental, because the proxmox team is not yet maintening it officially.


Oh, yes, and I need to write more examples in the doc ^_^

 

[le_top]

This is a great feature.

I setup a VXNET and to get it fully running on proxmox 8.1.3, I had to do a few things:

1. I added a firewall rule at the datacenter level to accept traffic related to the VXLAN. I probably should define an alias for the machines in the cluster and set that as the source for more security.




2. I could not ssh into one of the machines. After setting the MTU on OPNsense to 1450 manually, this worked (possibly VMs need the MTU set manually).

 

[Theo70]

Could anyone explain, how I put the WAN interface on vmbr0? I took the Mac address from the vmbr0 interface and chose DHCP, but that does not seem to be enough. I chose opnsense.

 

[le_top]

@Theo70 - This is not exactly the topic of this thread.

However, "bridge-ports <IFACE>" is the key.

​

On a server in a datacenter I have:

auto vmbr0
iface vmbr0 inet static
        address <STATIC_IP>/24
        gateway <STATIC_GW>
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0
        hwaddress FC:34:97:B0:58:43

And as this vmbr is bridged to enp3s0f0, OPNsense can be added to the same interface - the IP of OPNsense is defined in OPNsense.

For an offline home server where I used one physical interface for proxmox, and another for OPNsense, I documented:
```
auto enp4s0f1
iface enp4s0f1 inet dhcp

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
```
So proxmox is getting it's ip through DHCP and OPNsense connects on vmbr0 which bridges to the other port.

 

[Theo70]

Thank you! At the moment, it seems like due to security ip restrictions we will have to use another solution, but good to know, if the situation should change.

 

[bohtechprox]

I followed the exact steps below and I'm still having issues. I have a two node cluster. When using the VXLAN I can get an IP and can confirm it comes from Proxmox but cant connect to proxmox's admin. I can seem to get an IP address on one VM on node 1 and one VM on node 2 coming from my VM proxmox server on node 2 and I can look up Google.com BUT that seems to be where it stops. I cannot get to the firewall backend on the gateway address OR to other sites like amazon or forums.proxmox.com. BUT when I connect Proxmox VM and other VM both on node 2 using vmbr1 as the network device for both I can connect to proxmox backend and all traffic no problem. The VM on node 1 gets no connection at that point. Very weird. Any suggestions.

I am happy to say this worked great... although I did go down a rabbit hole or two. Here is what worked for me, in case anyone else needs to do this.


VXNET on Proxmox for Cluster Private Network

1. Go to **Datacenter** > **SDN** > **Zones** and create a zone
- Add > **VXLAN**
- Give it a name for the **ID**, `vxzone`
- Provide it a list of IP Addresses of the Proxmox Nodes, `10.1.1.61, 10.1.1.62, 10.1.1.63, 10.1.1.64`
- Leave the rest blank
2. Go to **Datacenter** > **SDN** > **Vnets** and create a Vnet
- Name: `vxnet`
- Alias: blank
- Zone `vxzone`
- Tag: `99`
- Leave **VLAN Aware** box unchecked
3. Go to **Datacenter** > **SDN** > **Vnets** > `vxnet` and add a **Subnet**
- Subnet: `10.6.6.0/24`
- Gateway: `10.6.6.1`
- Leave the rest blank
4. Go to **Datacenter** > **SDN** > **Apply**
- Hit the Apply button to make it all happen
5. Create a pfSense (or your favorite router os)
- Put the WAN interface on `vmbr0`
- Put the LAN interface on `vxnet`
- Assign the LAN interface the IP `10.6.6.1` and enable DHCP
- Test that you get a DHCP address on a VM on a different node

 

[arrozmio]

I followed the exact steps below and I'm still having issues. I have a two node cluster. When using the VXLAN I can get an IP and can confirm it comes from Proxmox but cant connect to proxmox's admin. I can seem to get an IP address on one VM on node 1 and one VM on node 2 coming from my VM proxmox server on node 2 and I can look up Google.com BUT that seems to be where it stops. I cannot get to the firewall backend on the gateway address OR to other sites like amazon or forums.proxmox.com. BUT when I connect Proxmox VM and other VM both on node 2 using vmbr1 as the network device for both I can connect to proxmox backend and all traffic no problem. The VM on node 1 gets no connection at that point. Very weird. Any suggestions.

i am in a similar boat but with a 3 node cluster on a mix of tiny/mff PCs with single NICs. They pick up DHCP and can ping my home gateway, other VMs in my internal and vxlan network. Except if i create a VM on another node configured for opnsense lan port. It can’t reach the admin gui for OPNsense. I made sure MTU was set to 1450 too.

 

[markc]

Hi, I'm the main author of the sdn feature. I'm running it in production (100 hosts, 4000 vms, with bgp-evpn).

Apologies for necrobumping but I noticed the magic words "bgp-evpn". I have an EVPN-VXLAN example working (thanks to your help) but I am still struggling to get any kind of BGP layer working to provide multiple exit nodes working in said example. Would you happen to know of any step by step guides for applying BGP, and also DHCP for the internal VXLAN network?

 

[proxdang]

Software Defined Network​

Proxmox Server Solutions GmbH
<support@proxmox.com>
version 7.4, Wed 22 Mar 2023 02:52:09 PM CET



The Software Defined Network (SDN) feature allows you to create virtual networks (VNets) at the datacenter level.

View attachment 48510

SDN is currently an experimental feature in Proxmox VE. This documentation for it is also still under development. Ask on our mailing lists or in the forum for questions and feedback.

As I found this post via a web search and the version is greater than 7.4, here is a link from the official documentation and to quote:

​

For Proxmox VE 8, a decision was made to lay the groundwork for full integration ofthe SDN feature by elevating the management of networks and interfaces to a corecomponent in the Proxmox VE access control stack.In Proxmox VE 8.1, two major milestones were achieved: firstly, DHCP integration was added to the IP address management (IPAM) feature, and secondly, the SDNintegration is now installed by default.​

​

I have been waiting for this feature to be improved. I also follow the steps defined by @jptechnical

-- updated
This is as simple as he documented. The one caveat is that simple zones are local to the proxmox node. If you have a proxmox cluster, you might want to use VLAN, QinQ, or VXLAN zone. I picked VXLAN for my cluster of 3 nodes. This will allow me to remove a physical layer that I previously "bridge" to my network based in 2020 (when I first created my Proxmox cluster).

Within your guests (that will be configured for the SDN), please be sure to manually set the MTU to 1450. While you can ping the other nodes, the MTU set to auto will not allow your Virtual guest to find the gateway, etc.


A few things I noticed with 8.2.2 as there are not many YouTube videos using VXLAN

• The ID field can be anything name you want to give.
• The Peer Address List fieeld should be contain the ip address of each of your Proxmox nodes. If the proxmox nodes are on more than network (such as 1GBe and/or 10 GBe network), then provide the preferred IP address for each node. For example, I manage my own proxmox node on 1GBe, but all the nodes are connected and communicate to each other over 10GBe (which is a backend network for servers only).
• The MTU was set to its default auto
• Nodes field was set its default all
• IPAM field was set to its default pve

When you configured the VNETs, you will need to add a tag.
When you configure the subnets,

Only the simple networks can use the DHCP process within Proxmox. As @jptechnical shared, you can use a pfSense instance within the SDN to manage DHCP.