VxLan configuration question

[Javier Vilarroig]

Hi.

I have a PVE 8.1.3 cluster with 3 nodes and I'm trying to build a VxLan based VNET. My nodes ares hosted at OVH, so VLAN is not an option.

My understanding is that this will create a single logical subnet where all the containers are connected using a local IP. That way, I can deploy or migrate the containers between nodes without having to worry about networking.

To do that I created first a vxlan zone:

vxlan: Internal
peers xxx.xxx.xxx.xxxx,yyy.yyy.yyy.yy,zzz.zzz.zzz.zzz
ipam pve
nodes node3,node2,node1

Here I have my first doubt. What is the right values for peers? I have entered the public IP address of the nodes separated by comma.

Then I defined the vnet:

vnet: IntNet
zone Internal
alias Internal Test
tag 100000

Then I created two containers in each node. I manually assigned to all of them IP addresses in the range 10.0.1.X

My expectation was that the containers will be able to reach each other without problems, but that was not true. They where only able to reach the other containar in the same node.

I have been reading documentation and trying to make it work to no avail.

Is there anything obvious that I have not taken into account? Any advise on how to make it work?

Thanks in advance!!
Javier Vilarroig

 

[spirit]

for peers, you need to use nodes1,2,3 ip addresses (where they are able to communicate between theses ips).

Maybe check that you don't have any firewall blocking port udp/4789

 

[Javier Vilarroig]

IP's are OK.
And no firewall. I checked with nmap and everything looks open.

 

[spirit]

can you send content of /etc/network/interfaces && /etc/network/interfaces.d/sdn of each node ?

 

[Javier Vilarroig]

Hi @spirit , thanks for your help

Here are the files:

/etc/network/interfaces:
(node1)

auto lo
iface lo inet loopback

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
address x.x.x.x/24
gateway x.x.x.x
bridge-ports enp5s0f0
bridge-stp off
bridge-fd 0
hwaddress A0:42:3F:42:AC:1E

source /etc/network/interfaces.d/*

(node 2)

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
address x.x.x.x/24
gateway x.x.x.x
bridge-ports eno1
bridge-stp off
bridge-fd 0
hwaddress A0:42:3F:4E:E3:68

source /etc/network/interfaces.d/*

(node 3)

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface enx966046fb0f94 inet manual

auto vmbr0
iface vmbr0 inet static
address x.x.x.x/24
gateway x.x.x.x
bridge-ports eno1
bridge-stp off
bridge-fd 0
hwaddress D0:50:99:F9:14:82

source /etc/network/interfaces.d/*

/etc/pve/sdn:
controllers.cfg is empty

zones.cfg

vxlan: Internal
peers X.X.X.X,X.X.X.X,X.X.X.X (These are the same addresses from the three nodes)
ipam pve
nodes node3,node2,node1

vnets.cfg

vnet: IntNet
zone Internal
alias Internal Test
tag 100000

 

[spirit]

I really need:
/etc/network/interfaces.d/sdn

too

(This is the where is the vxlan configuration generated by sdn)

 

[Javier Vilarroig]

Ouch! Our brain is very good at make us see what we know.

I was not even aware of the existence of this file. Now I understand better how the SDN is implemented from the definition, thanks for that

Here you have it

(node1)

auto IntNet
iface IntNet
bridge_ports vxlan_IntNet
bridge_stp off
bridge_fd 0
mtu 1450
alias Internal Test

auto vxlan_IntNet
iface vxlan_IntNet
vxlan-id 100000
vxlan_remoteip x.x.x.x (node3 address)
vxlan_remoteip x.x.x.x (node2 address)
mtu 1450

(node2)

auto IntNet
iface IntNet
bridge_ports vxlan_IntNet
bridge_stp off
bridge_fd 0
mtu 1450
alias Internal Test

auto vxlan_IntNet
iface vxlan_IntNet
vxlan-id 100000
vxlan_remoteip x.x.x.x (node1 address)
vxlan_remoteip x.x.x.x (node3 address)
mtu 1450

(node3)

auto IntNet
iface IntNet
bridge_ports vxlan_IntNet
bridge_stp off
bridge_fd 0
mtu 1450
alias Internal Test

auto vxlan_IntNet
iface vxlan_IntNet
vxlan-id 100000
vxlan_remoteip x.x.x.x (node1 address)
vxlan_remoteip x.x.x.x (node2 address)
mtu 1450