Really simple networking across nodes

[dpearceFL]

I have a three node cluster. I am struggling to understand SDN. (This is the right way to go, right?) I want to create a private network contained within my cluster.

"Simple" is an isolated bridge that does not allow VMs on different nodes to talk to each other.

Given my simple needs, is VLAN the way to go?

 

[madalin]

Subscribing to the post too.

I'm evaluating Proxmox as a replacement of a 3 nodes simple VMware cluster we have currently.

I'm confused how to span private networks between nodes.

Hosts are like this:
- 1 public interface (I used it for connecting the hosts to main router) (eth0)
- 1 private interface with a dedicated simple switch for CEPH (works super fine) (eth1)
- 1 private interface with a separate simple switch intended for using it with virtual networks. (eth2)

Mostly it would match the VMware setup.

I created an OVS Bridge on each node, adding eth2 as bridge port (nothing else, did not touch any other field).
In SDN I created a VLAN zone (this is how I understood from documentation).
Created a VNET with tag 10.
Applied configuration on SDN.

On all virtual servers on nodes, I was now able to add a network with the new available bridge named from the VNET.

Between same host, instances can communicate.

Between different hosts, can't.

Am I missing something, or some special switches are mandatory and simple dumb ones can't be used?

Edit: I tried same scenario using Linux Bridges and SDN. Same result. Instances between nodes can't communicate.

 

[qiongzhu]

if you are using managed switches, you can enable trunk mode, set vmbr0 in PVE in 'vlan aware mode', and set vlan tag on vm/lxc nic. This is the most simple and flexible way.

otherwise, you need try vxlan in PVE SDN, and make all nodes IP in 'Peer Address List'

 

[madalin]

thank you @qiongzhu . with vxlan things started to work.

next I see another issue. although I have set a gateway and snat on, I can't get traffic between instances in separate vnets.

do I need to create a "router" instance and add all vnets to it and manage routing inside it? in VMware, this just worked, and I assume it was creating something virtual and just worked.