Internode Networking

[dpearceFL]

I have been trying to get this to work for about a year now with no success. I have a three PVE cluster with three nodes. all PVE 8.4.1. I have two VMs, on different nodes but the same cluster. I need to have the two VMs to communicate with each other using static IPs over a private network.

1. I have created a a Linux Bridge on each of the three nodes, named vmbr2. I list no Bridge Ports
2. Then on the Datacenter level
   1. Under SDN -> Zones, I create a zone of the vlan type called vlan2
   2. Then under SDN -> VNETS, I create an entry called vnet2 (VLAN Aware is checked).
   3. I do not create a subnet.
3. On each VM I add an interface connected to vnet2 and inside the OS assign two different IPs.

VM1 cannot talk to VM2. What am I doing wrong?

 

[shanreich]

I have created a a Linux Bridge on each of the three nodes, named vmbr2. I list no Bridge Ports

You need to add a bridge port to vmbr2 on every node and then connect those physical ports via a switch, otherwise there's no way for the traffic to flow from VM1 to VM2.

You could use VXLAN if you want to utilize an existing Layer 3 (IP) connectivity, but make sure that the network is trusted, since VXLAN has no built-in encryption.

 

[dpearceFL]

So, an SDN based off of a Linux Bridge requires an external switch? Which I don't have unfortunately. But VXLAN uses the existing switch I have for communications.

 

[dpearceFL]

You gave me a hint plus I found this Explainer (https://www.youtube.com/watch?v=YRweIgot6aA)

1. So I did nothing on the individual PVE nodes.
2. Then I went to Datacenter -> SDN -> Zones and added a VXLAN called sdn123
3. I added the IP addresses of each node and set an MTU of 1450 (to allow for overhead)
4. Then I went to VNets and created a VNet of vnet123 with a zone of sdn123
5. I then added a NIC to each VM using vnet123 and an MTU of 1 (special value)

And it worked! Does this sound right?

 

[weehooey-bh]

You gave me a hint plus I found this Explainer (https://www.youtube.com/watch?v=YRweIgot6aA)

1. So I did nothing on the individual PVE nodes.
2. Then I went to Datacenter -> SDN -> Zones and added a VXLAN called sdn123
3. I added the IP addresses of each node and set an MTU of 1450 (to allow for overhead)
4. Then I went to VNets and created a VNet of vnet123 with a zone of sdn123
5. I then added a NIC to each VM using vnet123 and an MTU of 1 (special value)

And it worked! Does this sound right?

Yes. That is the correct way to set up the VXLAN.

VXLANs work, while VLANs do not, because VXLANs encapsulate the traffic they carry (think VPN, but without the encryption).

VLANs tag each packet (actually, they are frames, but you get the idea), and all the network gear between the Proxmox VE hosts needs to have that VLAN defined so it recognizes the packet's VLAN tag.