VXLAN between standalone (non clustered) nodes.

[turbo777]

Hello,

I setup a VXLAN between two NON clustered nodes using PVE SDN wiki;
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_setup_example_vlan
As far as I can tell everything works.
The tutorial describes how to install VXLAN within a cluster.
Are there any differences between VXLAN for cluster setup vs VXLAN setup between NON clustered nodes ?

Thank you.

 

[guruevi]

A VXLAN is just the network layer, it can be used to transport network traffic like cluster traffic, or not, the underlying OS doesn’t really care what you do with it.

Just be careful if you have multiple cluster management/data interfaces on the same subnet (whether those are the Proxmox application, Ceph or others), this doesn’t always play nice together. I think the capability for multiple clusters on the same subnet is new to Proxmox 8, Ceph technically supports it but still not a good idea imho (there are configurations if you disable authentication where you could cause issues). Also cluster rebuild traffic, broadcasts, discovery etc may traverse that VXLAN, which may consume unnecessary bandwidth.

If there is no intention of using the VXLAN for cluster traffic, make sure your subnets/management interfaces aren’t (accidentally) included in that VXLAN (use a different IP subnet etc).

 

[turbo777]

A VXLAN is just the network layer, it can be used to transport network traffic like cluster traffic, or not, the underlying OS doesn’t really care what you do with it.

Just be careful if you have multiple cluster management/data interfaces on the same subnet (whether those are the Proxmox application, Ceph or others), this doesn’t always play nice together. I think the capability for multiple clusters on the same subnet is new to Proxmox 8, Ceph technically supports it but still not a good idea imho (there are configurations if you disable authentication where you could cause issues). Also cluster rebuild traffic, broadcasts, discovery etc may traverse that VXLAN, which may consume unnecessary bandwidth.

If there is no intention of using the VXLAN for cluster traffic, make sure your subnets/management interfaces aren’t (accidentally) included in that VXLAN (use a different IP subnet etc).

Thank you for the extended info!
We have two non clustered PVE servers on remote site with two IPs per server (management and production) , all on the same subnet, 192.1.1.0/24
Each server has one VM on production interface. Because production IPs will be changing from time to time,
I was thinking to setup VXLAN with 10.0.0.0/24 on the management interface (less traffic ) for Data Base replication between VMs.
So, I guess it is a bad idea, unless VXLAN is on dedicated subnet.

 

[guruevi]

I would set the VXLAN between the two and only connect the VMs to it (potentially on a new virtual network card). Just connect the VXLAN to a new bridge, not the Proxmox management/local network bridge.

 

[turbo777]

I would set the VXLAN between the two and only connect the VMs to it (potentially on a new virtual network card). Just connect the VXLAN to a new bridge, not the Proxmox management/local network bridge.

Each VM has dedicated virtual NIC for VXLAN. I have only two IPs per server for production and management and they used by vmbr0 and vmbr1. No more available physical NICs. In zone adding interface GUI for VXLAN the peer IP needs to be provided, which I only have two prod. and mngm. I can create a new vmbr2 without any physical interfaces attached ( not sure if this will help).
Now, how do I connect VXLAN to the new bridge? Do I understand it wrong?

Thank you.

 

[guruevi]

I thought from your description you had set up VXLAN you wanted to add the management networks (what you describe as 10.0.0.0/24) to the same vxlan10.

The Proxmox node IP are just the endpoints, they encapsulate the traffic and send it to the other end, at that point they take the traffic into the vxlan10 in your case and the host involvement should end.

Think of it this way, if you attempt to add the same subnet as your physical management interfaces or add vmbr0 to your vxlan10, how would the traffic flow?

 

[turbo777]

I thought from your description you had set up VXLAN you wanted to add the management networks (what you describe as 10.0.0.0/24) to the same vxlan10.

Sorry for the confusion. Here is my current setup:
Mngm. - 2xNICs > bond(LACP) > vmbr0 192.1.1.10 , second server 192.1.1.11
Prod. - 2xNICs > bond(LACP) > vmbr1 192.1.1.20 , second server 192.1.1.21
VXLAN10 peers: 192.1.1.10 and 192.1.1.11
VMs IPs: first server 10.0.0.10 on a second virt. NIC on vxlan10 bridge
second server 10.0.0.11 on a second virt. NIC on vxlan10 bridge

Does it look good?

 

[guruevi]

Yes, that looks good!

 

[turbo777]

Yes, that looks good!

Thank you for your help and patience!