[SOLVED] share a private network between to nodes in a cluster

[landis]

Hi everyone,

I have 2 nodes on the same lan. i created a cluster between them. i have a private network on node1 vmbr1 is it possible for node2 to also connect to node1's private network.

 

[vesuvienne]

i think you need SDN to share network between node in a cluster

 

[landis]

I setup a SDN network and now i have the network side working. The vms cannot reach the WAN though.

Node1 -> VM -> can ping node2 vm
Node2 -> VM -> can ping node1 vm

Both vms are unable to reach the WAN.

My Setup:
SDN-Zones: type vxlan

 

[vesuvienne]

you should add:
1. vmbr0 to your VM & your VLAN
or
2. add simple SDN:
- zones->add->simple
- Vnets->create->choose name, and the simple zones you have created
- create subnet for the Vnets
-SNAT: give internet to your future VM

-> then DHCP range between 192.168.200.1 - 192.168.200.253

after add the 2 bridges to your VM vxlan and vnet snat, and u should have internet + communication between vm

 

[landis]

you should add:
1. vmbr0 to your VM & your VLAN
or
2. add simple SDN:
- zones->add->simple
- Vnets->create->choose name, and the simple zones you have created
- create subnet for the Vnets
-SNAT: give internet to your future VM
View attachment 70193
-> then DHCP range between 192.168.200.1 - 192.168.200.253

after add the 2 bridges to your VM vxlan and vnet snat, and u should have internet + communication between vm

After hours of trying the both the simple and vxlan.

The simple zone allows the vms within the same node 1 to talk to each other and the internet but it does not talk to other vms on the same network which is on another node.

NODE 1 : vm1 -> can talk to vm2
NODE 1 vm2 -> can talk to vm1
NODE 1: vm1 -> cannot talk to vm3(node2)
NODE 2: vm3 -> cannot talk to vm1 but can talk to vm4
NODE 2: vm4 -> can talk to vm3

The vxlan zone allows the vms to talk to each other regardless of the vm location (node1/node2). The issue is they cannot reach the internet for some reason. I cannot seem to figure out why.
The vxlan method lets vm's on the same network talk to each other, node1 vm can talk to node2 vm (vice verse). The problem is they both cannot talk to the WAN (internet)

 

[vesuvienne]

Did you attach vmbr0 & VLAN to your VM?
i never used vxlan but i suppose it work the same as vlan

So if you want u can forward connection of your VMBR0 to your vlan
u don't rly need SDN but it is more easy with, i show a classic linux bridge
a show u a exemple:
proxmox host: /etc/network/interfaces
create linux bridge on both of your node vmbr0.22 and forward WAN from vmbr0

auto lo
iface lo inet loopback

iface enp61s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.100.100/24
        gateway 192.168.100.254
        bridge-ports enp61s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.22.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.22.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

auto vmbr0.22 # vlan: 22 is the TAG
iface vmbr0.22 inet static
    address 192.168.22.254
    netmask 255.255.255.0

You have 2 choices: (i suppose you have the default network proxmox config with vmbr0)
add to your VM both network devices = vmbr0 + sdn vlan
or
- exemple above: add vmbr0 with tag22(with forwarding vmbr0 to your vlan, attach 1 network device to your VM)
INFO: i think u can't add WAN to SDN VLAN( so u need to add both network devices = vmbr0 + vlan)

 

[landis]

Did you attach vmbr0 & VLAN to your VM?
i never used vxlan but i suppose it work the same as vlan

I never added the vmbr0 only the network made by SDN. I added the vmbr0 after you mentioned it and still no internet.

I want my vm's to be able to talk to each other across nodes on the same net work and having access to WAN.

Took some time but this is how it looks/works.



With the above setup. VM 104 can talk to vm 106 on node2 and vice versa. Using the SDN (vxlan) 192.168.20.0/24 network. The issue is that on both vms there is no WAN access.

Does this perhaps clear up what i am trying to achieve.

 

[landis]

Solved!.

Thanks to @vesuvienne for the nudge. So since there is a possibility that vxlan cannot communicate with the WAN. So i though i could use both the simple and vxlan to accomplish what i need it to.

So for SDN.

Create 2 zones:

1. Simple Zone (handle the dhcp) - pnetdhcp
2. VXLAN Zone (communicate vms/lxcs across nodes) pnet

Create 2 Vnets:

1. vxnet -> zone pnetdhcp
2. vxlan -> zone pnet

When creating the VMs/LXCs add 2 network bridges. vxnet and set it to dhcp, the second one vxlan and you can set the ip for that.

Now the VMs can talk to each other across nodes and locally. The VMs/LXCs are also able to talk to eachother.

This could possibly be cleaned up using VLANs, Im still figuring out how exactly vlans work etc.