Cluster + VLAN + Unicast

Dockay

Member
Jul 9, 2013
3
0
21
Hello together,

right upfront I'm not an hyper expert on networking, so forgive me some mistakes.

I followed the basic setup of a HA Cluster (3 Nodes) on Proxmox VE 5.x and as I'm not on a private network (I rent 3 identical servers at hetzner.de), I used unicast to connect them to the same cluster. This is working well at the moment.

Unfortunately they are not connected on the same network, so I need to build a workaround to connect those in a way I can switch my VMs from one Node to another and that this VMs can reach each other across Nodes.

I followed this tutorial
Code:
icicimov.github.io/blog/virtualization/Cluster-Networking-for-Multi-tenant-isolation-in-Proxmox-with-OpenVSwitch/
which I found really good explained, but I think I'm just missing some basic setup, that was not clearly mentioned. Actually I dont need DHCP. I liked the approach of tagging the VLANs so I can setup some VMs that don't reach each other.

I created a VM on Node #1 and assigned it to vmbr2, but here I'm struggling on how this VM:
- can access to the internet?
- can access to other VMs?
- what network config to give them? (What IP?)


enp4s0 is eth0 (I couldn't name it to eth0 was giving me errors).

Node #1 /etc/network/interface
Code:
auto enp4s0
iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address 88.99.252.238
  netmask 255.255.255.192
  gateway 88.99.252.193
  # route 88.99.252.192/26 via 88.99.252.193
  up route add -net 88.99.252.192 netmask 255.255.255.192 gw 88.99.252.193 dev vmbr0
  bridge_ports enp4s0
  bridge_stp off
  bridge_fd 0
  metric 100

# GRE/VXLAN network
allow-vmbr1 eth3
iface eth3 inet manual
        ovs_bridge vmbr1
        ovs_type OVSPort
        mtu 1546
        up ip link set eth3 up
# GRE/VXLAN bridge
auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports eth3 tep0
        up ip link set vmbr1 up
# GRE/VXLAN interface
allow-vmbr1 tep0
iface tep0 inet static
        ovs_bridge vmbr1
        ovs_type OVSIntPort
        #ovs_options tag=11
        address 10.30.1.185
        netmask 255.255.255.0
# Integration bridge
auto vmbr2
allow-ovs vmbr2
iface vmbr2 inet manual
    ovs_type OVSBridge
    ovs_ports vx1 dhcptap0
    up ip link set vmbr2 up
# GRE/VXLAN tunnel
allow-vmbr2 vx1
iface vx1 inet manual
    ovs_type OVSTunnel
    ovs_bridge vmbr2
    ovs_tunnel_type vxlan
    ovs_options trunks=11,22,33
    ovs_tunnel_options options:remote_ip=10.30.1.186 options:key=flow options:dst_port=4789


Node #2 /etc/network/interface
Code:
auto enp4s0
iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address 136.243.57.19
  netmask 255.255.255.192
  gateway 136.243.57.1
  #route 136.243.57.0/26 via 136.243.57.1
  up route add -net 136.243.57.0 netmask 255.255.255.192 gw 136.243.57.1 dev vmbr0
  bridge_ports enp4s0
  bridge_stp off
  bridge_fd 0
  metric 100

# GRE/VXLAN network
allow-vmbr1 eth3
iface eth3 inet manual
        ovs_bridge vmbr1
        ovs_type OVSPort
        mtu 1546
        up ip link set eth3 up
 
# GRE/VXLAN bridge
auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports eth3 tep0
        up ip link set vmbr1 up
 
# GRE/VXLAN interface
allow-vmbr1 tep0
iface tep0 inet static
        ovs_bridge vmbr1
        ovs_type OVSIntPort
        #ovs_options tag=11
        address 10.30.1.186
        netmask 255.255.255.0
 
# Integration bridge
auto vmbr2
allow-ovs vmbr2
iface vmbr2 inet manual
    ovs_type OVSBridge
    ovs_ports vx1 dhcptap0
    up ip link set vmbr2 up
 
# GRE/VXLAN tunnel
allow-vmbr2 vx1
iface vx1 inet manual
    ovs_type OVSTunnel
    ovs_bridge vmbr2
    ovs_tunnel_type vxlan
    ovs_options trunks=11,22,33
    ovs_tunnel_options options:remote_ip=10.30.1.185 options:key=flow options:dst_port=4789
 
Hello together,

right upfront I'm not an hyper expert on networking, so forgive me some mistakes.

I followed the basic setup of a HA Cluster (3 Nodes) on Proxmox VE 5.x and as I'm not on a private network (I rent 3 identical servers at hetzner.de), I used unicast to connect them to the same cluster. This is working well at the moment.

Unfortunately they are not connected on the same network, so I need to build a workaround to connect those in a way I can switch my VMs from one Node to another and that this VMs can reach each other across Nodes.

I followed this tutorial
Code:
icicimov.github.io/blog/virtualization/Cluster-Networking-for-Multi-tenant-isolation-in-Proxmox-with-OpenVSwitch/
which I found really good explained, but I think I'm just missing some basic setup, that was not clearly mentioned. Actually I dont need DHCP. I liked the approach of tagging the VLANs so I can setup some VMs that don't reach each other.

I created a VM on Node #1 and assigned it to vmbr2, but here I'm struggling on how this VM:
- can access to the internet?
- can access to other VMs?
- what network config to give them? (What IP?)


enp4s0 is eth0 (I couldn't name it to eth0 was giving me errors).

Node #1 /etc/network/interface
Code:
auto enp4s0
iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address 88.99.252.238
  netmask 255.255.255.192
  gateway 88.99.252.193
  # route 88.99.252.192/26 via 88.99.252.193
  up route add -net 88.99.252.192 netmask 255.255.255.192 gw 88.99.252.193 dev vmbr0
  bridge_ports enp4s0
  bridge_stp off
  bridge_fd 0
  metric 100

# GRE/VXLAN network
allow-vmbr1 eth3
iface eth3 inet manual
        ovs_bridge vmbr1
        ovs_type OVSPort
        mtu 1546
        up ip link set eth3 up
# GRE/VXLAN bridge
auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports eth3 tep0
        up ip link set vmbr1 up
# GRE/VXLAN interface
allow-vmbr1 tep0
iface tep0 inet static
        ovs_bridge vmbr1
        ovs_type OVSIntPort
        #ovs_options tag=11
        address 10.30.1.185
        netmask 255.255.255.0
# Integration bridge
auto vmbr2
allow-ovs vmbr2
iface vmbr2 inet manual
    ovs_type OVSBridge
    ovs_ports vx1 dhcptap0
    up ip link set vmbr2 up
# GRE/VXLAN tunnel
allow-vmbr2 vx1
iface vx1 inet manual
    ovs_type OVSTunnel
    ovs_bridge vmbr2
    ovs_tunnel_type vxlan
    ovs_options trunks=11,22,33
    ovs_tunnel_options options:remote_ip=10.30.1.186 options:key=flow options:dst_port=4789


Node #2 /etc/network/interface
Code:
auto enp4s0
iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address 136.243.57.19
  netmask 255.255.255.192
  gateway 136.243.57.1
  #route 136.243.57.0/26 via 136.243.57.1
  up route add -net 136.243.57.0 netmask 255.255.255.192 gw 136.243.57.1 dev vmbr0
  bridge_ports enp4s0
  bridge_stp off
  bridge_fd 0
  metric 100

# GRE/VXLAN network
allow-vmbr1 eth3
iface eth3 inet manual
        ovs_bridge vmbr1
        ovs_type OVSPort
        mtu 1546
        up ip link set eth3 up
 
# GRE/VXLAN bridge
auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports eth3 tep0
        up ip link set vmbr1 up
 
# GRE/VXLAN interface
allow-vmbr1 tep0
iface tep0 inet static
        ovs_bridge vmbr1
        ovs_type OVSIntPort
        #ovs_options tag=11
        address 10.30.1.186
        netmask 255.255.255.0
 
# Integration bridge
auto vmbr2
allow-ovs vmbr2
iface vmbr2 inet manual
    ovs_type OVSBridge
    ovs_ports vx1 dhcptap0
    up ip link set vmbr2 up
 
# GRE/VXLAN tunnel
allow-vmbr2 vx1
iface vx1 inet manual
    ovs_type OVSTunnel
    ovs_bridge vmbr2
    ovs_tunnel_type vxlan
    ovs_options trunks=11,22,33
    ovs_tunnel_options options:remote_ip=10.30.1.185 options:key=flow options:dst_port=4789

Hetzner has vSwitches now. This might work way better and secure for an internal network.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!