[SOLVED] Is there a cluster bridge or cluster-level networking?

[jptechnical]

Hi, first time post, long time reader. I have question that I can't quite figure out what to search for, as the terms seem pretty common. So here goes.

Is there any kind of cluster-level networking that would allow vms on various nodes to share a separate lan without a physical smart (vlan aware) switch?

I know this sounds like a vlan, but I don't have a spare smart switch to hand and would like to start testing with a dumb (but fast) inexpensive switch.

I have a lab cluster and I would like to test some things that would best be separated from my regular lan. It's a small office (home office where I run my IT biz), and I could just physically put it on it's own switch. But I would like to have the flexibility of just toggling a button here and there to move a VM to a different network segment. I can just set the vlan 999 on the VMs on a single node, but it breaks when I have vms on multiple nodes. Also, I would like to have a pfsense VM giving that private lan a wan.

 

[spirit]

you can use vxlan with the sdn feature.

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

 

[jptechnical]

I am happy to say this worked great... although I did go down a rabbit hole or two. Here is what worked for me, in case anyone else needs to do this.


VXNET on Proxmox for Cluster Private Network

1. Go to **Datacenter** > **SDN** > **Zones** and create a zone
- Add > **VXLAN**
- Give it a name for the **ID**, `vxzone`
- Provide it a list of IP Addresses of the Proxmox Nodes, `10.1.1.61, 10.1.1.62, 10.1.1.63, 10.1.1.64`
- Leave the rest blank
2. Go to **Datacenter** > **SDN** > **Vnets** and create a Vnet
- Name: `vxnet`
- Alias: blank
- Zone `vxzone`
- Tag: `99`
- Leave **VLAN Aware** box unchecked
3. Go to **Datacenter** > **SDN** > **Vnets** > `vxnet` and add a **Subnet**
- Subnet: `10.6.6.0/24`
- Gateway: `10.6.6.1`
- Leave the rest blank
4. Go to **Datacenter** > **SDN** > **Apply**
- Hit the Apply button to make it all happen
5. Create a pfSense (or your favorite router os)
- Put the WAN interface on `vmbr0`
- Put the LAN interface on `vxnet`
- Assign the LAN interface the IP `10.6.6.1` and enable DHCP
- Test that you get a DHCP address on a VM on a different node

 

[Vengance]

Is SDN already considered stable or still experimental?

 

[pille99]

Software Defined Network​

Proxmox Server Solutions GmbH
<support@proxmox.com>
version 7.4, Wed 22 Mar 2023 02:52:09 PM CET



The Software Defined Network (SDN) feature allows you to create virtual networks (VNets) at the datacenter level.

SDN is currently an experimental feature in Proxmox VE. This documentation for it is also still under development. Ask on our mailing lists or in the forum for questions and feedback.

 

[pille99]

i am working with it and works pretty well. as i understand, many here in the forum use it, even on productive proxmox

 

[spirit]

Hi,I'm the main author of the sdn feature. I'm running it in production (100 hosts, 4000 vms, with bgp-evpn).

The code is pretty stable, I don't have seen bug report since a long time. (only new request features)

and I don't plan any breaking changes. (as a lot of users already using it)

It's still experimental, because the proxmox team is not yet maintening it officially.


Oh, yes, and I need to write more examples in the doc ^_^

 

[le_top]

This is a great feature.

I setup a VXNET and to get it fully running on proxmox 8.1.3, I had to do a few things:

1. I added a firewall rule at the datacenter level to accept traffic related to the VXLAN. I probably should define an alias for the machines in the cluster and set that as the source for more security.




2. I could not ssh into one of the machines. After setting the MTU on OPNsense to 1450 manually, this worked (possibly VMs need the MTU set manually).

 

[Theo70]

Could anyone explain, how I put the WAN interface on vmbr0? I took the Mac address from the vmbr0 interface and chose DHCP, but that does not seem to be enough. I chose opnsense.

 

[le_top]

@Theo70 - This is not exactly the topic of this thread.

However, "bridge-ports <IFACE>" is the key.

​

On a server in a datacenter I have:

auto vmbr0
iface vmbr0 inet static
        address <STATIC_IP>/24
        gateway <STATIC_GW>
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0
        hwaddress FC:34:97:B0:58:43

And as this vmbr is bridged to enp3s0f0, OPNsense can be added to the same interface - the IP of OPNsense is defined in OPNsense.

For an offline home server where I used one physical interface for proxmox, and another for OPNsense, I documented:
```
auto enp4s0f1
iface enp4s0f1 inet dhcp

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
```
So proxmox is getting it's ip through DHCP and OPNsense connects on vmbr0 which bridges to the other port.

 

[Theo70]

Thank you! At the moment, it seems like due to security ip restrictions we will have to use another solution, but good to know, if the situation should change.

 

[bohtechprox]

I followed the exact steps below and I'm still having issues. I have a two node cluster. When using the VXLAN I can get an IP and can confirm it comes from Proxmox but cant connect to proxmox's admin. I can seem to get an IP address on one VM on node 1 and one VM on node 2 coming from my VM proxmox server on node 2 and I can look up Google.com BUT that seems to be where it stops. I cannot get to the firewall backend on the gateway address OR to other sites like amazon or forums.proxmox.com. BUT when I connect Proxmox VM and other VM both on node 2 using vmbr1 as the network device for both I can connect to proxmox backend and all traffic no problem. The VM on node 1 gets no connection at that point. Very weird. Any suggestions.

I am happy to say this worked great... although I did go down a rabbit hole or two. Here is what worked for me, in case anyone else needs to do this.


VXNET on Proxmox for Cluster Private Network

1. Go to **Datacenter** > **SDN** > **Zones** and create a zone
- Add > **VXLAN**
- Give it a name for the **ID**, `vxzone`
- Provide it a list of IP Addresses of the Proxmox Nodes, `10.1.1.61, 10.1.1.62, 10.1.1.63, 10.1.1.64`
- Leave the rest blank
2. Go to **Datacenter** > **SDN** > **Vnets** and create a Vnet
- Name: `vxnet`
- Alias: blank
- Zone `vxzone`
- Tag: `99`
- Leave **VLAN Aware** box unchecked
3. Go to **Datacenter** > **SDN** > **Vnets** > `vxnet` and add a **Subnet**
- Subnet: `10.6.6.0/24`
- Gateway: `10.6.6.1`
- Leave the rest blank
4. Go to **Datacenter** > **SDN** > **Apply**
- Hit the Apply button to make it all happen
5. Create a pfSense (or your favorite router os)
- Put the WAN interface on `vmbr0`
- Put the LAN interface on `vxnet`
- Assign the LAN interface the IP `10.6.6.1` and enable DHCP
- Test that you get a DHCP address on a VM on a different node