Cluster-wide private network

[Deleted member 93625]

Hi all,

Hope I can get some help from here regarding what I want to achieve with Proxmox. I have setup a Proxmox cluster with two nodes (physical diagram attached). Cluster uses a shared storage (I've configured ZFS over iSCSI with FreeNAS 11.3). Currently, I am testing live migration and got stuck with an issue.

I have configured network on each node like below (interface name and order are exactly the same, just IP address difference).

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface enp0s29f0u2 inet manual

iface ens3f0 inet manual

auto ens4
iface ens4 inet static
        address 172.16.200.11/24
        mtu 9000
#storage and migration

iface ens3f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.128.255.1/16
        gateway 10.128.255.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#data and mgmt

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
#external

auto vmbr2
iface vmbr2 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#test subnet

I created two VMs that connect to vmbr0 and vmbr2 bridges. If they are on the same node, they can talk to each other (I am testing this with ping). However, if I migrate one VM to other node, they can't. Ping to external network via vmbr0 is okay, but they just can't talk to each other via vmbr2. I thought setting the same bridges on every cluster nodes can do this but maybe not? Or, is there anything that I am missing at the moment? I have to create more private networks but before doing that I have to make this work first. I'd appreciate it if I could get some help. Thanks.

Eoin

 

[Stoiko Ivanov]

vmbr2 has no bridge-port - i.e. no NIC where the traffic on that bridge would leave and reach the other Node.

a bridge is like a (layer 2) switch - it sends out all packets to the bridge-port where the mac-address is hosted

you need to connect the 2 node's vmbr2 somehow for this to work

if you don't have any more physical NICs you can consider configuring a vmbr2 on a VLAN on eno1 or eno2

I hope this explains it!

 

[Deleted member 93625]

vmbr2 has no bridge-port - i.e. no NIC where the traffic on that bridge would leave and reach the other Node.

a bridge is like a (layer 2) switch - it sends out all packets to the bridge-port where the mac-address is hosted

you need to connect the 2 node's vmbr2 somehow for this to work

if you don't have any more physical NICs you can consider configuring a vmbr2 on a VLAN on eno1 or eno2

I hope this explains it!

Hi Stoiko,

Thanks for your response, appreciate it. I can add additional NIC cards, no worries. This makes me ask further questions. Let me do my best to clearly ask questions with my poor English.

1. I need to create more private networks, at this stage minimum 10. I believe creating more bridges will not be a problem. I am wondering if multiple bridges can bind to the same physical interface. By reading your answer, I guess not? So, if I add additional NIC (say, eno3), can I create multiple bridges with the same NIC?
   
   
2. If VMs are connected to different bridges, they cannot talk to each other, correct? Because that's what I want. VMs on the same private network can talk but can't on different private network.

I believe the question #1 is quite important for my situation. Hope I didn't make any confusions. Thanks again.

Eoin

 

[Stoiko Ivanov]

if you need 10 networks consider using VLANs
you can also configure different IP-networks on the same bridge (i.e. don't make a separation)

please check the reference documentation on networking (and the internet) for further details:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_network_configuration

I hope this helps!

 

[Deleted member 93625]

if you need 10 networks consider using VLANs
you can also configure different IP-networks on the same bridge (i.e. don't make a separation)

please check the reference documentation on networking (and the internet) for further details:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_network_configuration

I hope this helps!

Hi Stoiko,

Thanks for that. I'll have a look, muck around a bit and let you know once I get something. Thanks again.

Eoin

 

[Deleted member 93625]

Hi Stokio,

I was thinking about your VLAN suggestion -

if you don't have any more physical NICs you can consider configuring a vmbr2 on a VLAN on eno1 or eno2

Based on your comment, I configured the setup as below.

iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto ens4
iface ens4 inet static
        address 172.16.200.11/24
        mtu 9000
#storage and migration

auto vmbr0
iface vmbr0 inet static
        address 10.128.255.1/16
        gateway 10.128.255.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#data and mgmt

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
#external

auto vmbr2
iface vmbr2 inet manual
        bridge-ports eno1.100
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#test private net 1

auto vmbr3
iface vmbr3 inet manual
        bridge-ports eno1.101
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#test private net 2

I configured this via GUI by creating a Linux bridge. Am I following you? I am wondering what creating Linux VLAN does instead of Linux bridge. The above configuration by the way works nicely. Thank you.

 

[Stoiko Ivanov]

A VLAN is a virtual layer 2 domain - https://en.wikipedia.org/wiki/Virtual_LAN, while a bridge (https://en.wikipedia.org/wiki/Bridging_(networking)) behaves like a switch - they are quite different concepts

As for your setup - why do you make the bridges on a tagged port vlan-aware? - I think this would result in a too complex setup (qinq) - remove the checkbox if you don't need it.

I hope this helps!

 

[Deleted member 93625]

A VLAN is a virtual layer 2 domain - https://en.wikipedia.org/wiki/Virtual_LAN, while a bridge (https://en.wikipedia.org/wiki/Bridging_(networking)) behaves like a switch - they are quite different concepts

As for your setup - why do you make the bridges on a tagged port vlan-aware? - I think this would result in a too complex setup (qinq) - remove the checkbox if you don't need it.

I hope this helps!

Hi Stoiko,

Thanks for that. I understand those concepts with networking gears but not virtualised networking. I'll have a read. Thanks again.

Eoin