[SOLVED] help understanding vlan in a proxmox cluster scenario

O

offerlam

Renowned Member
Dec 30, 2012
218
0
81
Denmark
Hi all,,

I have read it over and over but i just can't get my head around it....

http://pve.proxmox.com/wiki/Network_Model

Configuring VLAN in a cluster

For the simplest way to create VLAN follow the link: VLAN
Goal:

  • Have two separate network on the same NIC
  • Another host (firewall) manage the routing and rule to access to these VMs (out of this doc)
Suppose this scenario:

  • A cluster with two nodes
  • Each node have two NIC
  • We want bonding the NIC
  • We use two network: one untagged 192.168.1.0/24 and one tagged (VLanID=53) 192.168.2.0/24, we must configure the switch with port vlan.
  • We want separate these network at layer 2
Create bond0

First of all we create the bond0 (switch assisted 802.3ad) at the proxmox web interface, follow the video.
At the end we have a /etc/network/interface like this:
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
gateway 192.168.1.250
bridge_ports bond0
bridge_stp off
bridge_fd 0
Configure your switch appropriately. If you're using a bond of multiple links, you need to tell this to your switch and put the switch ports in a Link Aggregation Group or Trunk.
Create VLAN

We have two methods to follow:
First explicit method

auto vlan53
iface vlan53 inet manual
vlan_raw_device bond0
Second method

We can use direct the NIC dot VLAN ID, like bond0.53
I prefer the first one!
Create manually the bridge

Now we create manually the second bridge.
auto vmbr1
iface vmbr1 inet static
address 192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
bridge_ports vlan53
bridge_stp off
bridge_fd 0
post-up ip route add table vlan53 default via 192.168.2.250 dev vmbr1
post-up ip rule add from 192.168.2.0/24 table vlan53
post-down ip route del table vlan53 default via 192.168.2.250 dev vmbr1
post-down ip rule del from 192.168.2.0/24 table vlan53
NOTE:

  • We must not indicate the gateway, we must manually modify the routing table use ip route 2
  • The whole configuration must replicate on the other cluster's node, the only change is the IP of the node.
Create the table in ip route 2

We must change the file /etc/iproute2/rt_tables, add the following line:
# Table for vlan53
53 vlan53
use these commands to add:
echo "# Table for vlan53" >> /etc/iproute2/rt_tables
echo "53 vlan53" >> /etc/iproute2/rt_tables
Create the vlan on switch

For example on a HP Procurve 52 ports we use the following instructions to create the vlan.
Suppose:

  • Ports 47-48 trunk (switch assisted 802.3ad) for gateway
  • Ports 1-2 trunk (switch assisted 802.3ad) for the first node of cluster proxmox
  • Ports 3-4 trunk (switch assisted 802.3ad) for the second node
Enter in configuration mode and type:
trunk 1-2 Trk1 LACP
trunk 3-4 Trk2 LACP
trunk 47-48 Trk3 LACP
vlan 2
name "Vlan2"
untagged Trk1-Trk3
ip address 192.168.1.254 255.255.255.0
exit
vlan 53
name "Vlan53"
tagged Trk1-Trk3
exit
Test the configuration

Reboot the cluster node one by one for testing this configuration.
Unsupported Routing

Physical NIC (eg., eth1) cannot currently be made available exclusively for a particular KVM / Container , ie., without bridge and/or bond.
Naming Conventions


  • Ethernet devices: eth0 - eth99
  • Allowable bridge names: vmbrn, where 0 ≤ n ≤ 4094
  • Bonds: bond0 - bond9
  • VLANs: Simply add the VLAN number to the ethernet device name, separated by a period. For example "eth0.50"
Click to expand...


I have a 3 node cluster...

I would like the administration network to not be a in VLAN. The administration network is where i connect to the proxmox nodes using either web or ssh for setups..

I will be using this 3 node cluster for hosting and it could be that i will be hosting other clients infrastructure like virtual file, dc and mail server and so on. Therefore i need to know how to use vlans for virtual machines since each client will have to be on his or hers own VLAN...

My problem is i just can't get my head around the network model documentation..

My plan is to create a sub interface for each client in my firewall. Each subinterface will be assigned a vlan tag. from here I will have two switches in a LACP configuration that will carry the VLAN tag to the 3 proxmox nodes.

My assumption is that the only thing i need to configure is to have say VLAN 10 tagged from the firewall through the switches and tagged to each cluster node NOT configuring anything in any of the cluster nodes interface files. and than configure VLAN 10 on the different VMs virtual network cards...

Is this assumption right? and will it work in a failover scenario? Or do i have to set something up in the cluster nodes interface file?

THANKS

Casper
 
Last edited:
Why not create a bridge for each vlan and then assign veths to VM's on the bridge which the VM should belong to?

E.g.
Create a bridge on vlan 30:
Code: 
auto vmbr30
iface vmbr30 inet manual
    bridge_ports bond0.30
    bridge_stp off
    bridge_fd 0
Each VM will then be instructed to use the following:
DHCP
Code: 
auto eth0
iface eth0 inet dhcp
Your switch or something else will then assign addresses to clients on vlan 30

Static
Code: 
auto eth0
iface eth0 inet static
    address 10.0.30.1
    netmask 255.255.255.0
    gateway 10.0.30.254
Traffic is then routed either by your switch or a router assigned 10.0.30.254 on vlan 30.
 
Hi Mir,

Yet again you come to my rescure :)

I love you suggestion .. so simple..

I suppose the

Code: 
[COLOR=#3E3E3E]auto vmbr30
[/COLOR]iface vmbr30 inet manual
    bridge_ports bond0.30
    bridge_stp off [COLOR=#3E3E3E]    
    bridge_fd 0
[/COLOR]

should be in written in the interface file of each node..

so if i would add a second customer it would maybe look like this

Code: 
auto vmbr31
iface vmbr31 inet manuel
     bridge_ports bond0.31
     bridge_stp off
     [COLOR=#3E3E3E]bridge_fd 0[/COLOR]

would i have to tag the physical switch port where the nodes connect to vlan 31 or untagged?

also my serverse would have two interfaces in LACP bond connected to each switch. As fare as i understand it i would then have to make a bond and not a bridge? or do you make a bond and than bridge the bond?

Thanks for answering and marry christmas!

Casper
 
Last edited:
offerlam said:
Hi Mir,

Yet again you come to my rescure :)

I love you suggestion .. so simple..

I suppose the

Code: 
[COLOR=#3E3E3E]auto vmbr30
[/COLOR]iface vmbr30 inet manual
    bridge_ports bond0.30
    bridge_stp off [COLOR=#3E3E3E]    
    bridge_fd 0
[/COLOR]

should be in written in the interface file of each node..

so if i would add a second customer it would maybe look like this

Code: 
auto vmbr31
iface vmbr31 inet manuel
     bridge_ports bond0.31
     bridge_stp off
     [COLOR=#3E3E3E]bridge_fd 0[/COLOR]

would i have to tag the physical switch port where the nodes connect to vlan 31 or untagged?

also my serverse would have two interfaces in LACP bond connected to each switch. As fare as i understand it i would then have to make a bond and not a bridge? or do you make a bond and than bridge the bond?

Thanks for answering and marry christmas!

Casper
Click to expand...

I do pretty much exactly like this. Create the bond, then create a bridge on the bond. Packets on the switch need to be tagged, and the bond0.30 is that tag. The VM/container use the nic like normal and do not need any tagging at all... To the VM/container, it is just a standard nic.

Sent from my Nexus 5
 
The bond/bridging part has already been answered by pirateghost.

For the connection to the physical switch your bond needs to be connected to switch ports which are in trunk mode to be able to carry the vlan tags. If your switch is capable of doing routing you are home free otherwise you will need a router to take care of routing your vlans.
 
offerlam said:
thanks guys.. this really made my day..a nice and simple solution.. :)
Click to expand...
Hi,
there is one easier way (but don't know if its work with bonds, but why not):
Simply create an bridge like this:
Code: 
auto vmbr1
iface vmbr1 inet manual
     bridge_ports bond0
     bridge_stp off
     bridge_fd 0
and use in the gui (Add: Network Device) Bridge mode "vmbr1" and vlan tag "44" (or which vlan you like).

This has the effort that you easy use new vlans (after tagging on the switch-side). With bond0.xx you need an reboot - or create by hand like this:
Code: 
vconfig add bond0 44
ifconfig bond0.44 up
brctl addbr vmbr44
brctl addif vmbr44 bond0.44
ifconfig vmbr44 up

I use also multible bridges, because I use the maintenance network also via vlan-tagging.

Udo
 
Last edited:
udo said:
This has the effort that you easy use new vlans (after tagging on the switch-side). With bond0.xx you need an reboot - or create by hand like this:
Code: 
vconfig add bond0 44
ifconfig bond0.44 up
brctl addbr vmbr44
brctl addif vmbr44 bond0.44
ifconfig vmbr44 up
Click to expand...
This is not entirely true. After creating a new interface in /etc/network/interfaces an 'ifup vmbrxx' should do the magic.
 
Than came the day where i needed to set this up.. and more questions came...

so I choose what i call "The Mir Approach"

Why not create a bridge for each vlan and then assign veths to VM's on the bridge which the VM should belong to?

E.g.
Create a bridge on vlan 30:
Code:
auto vmbr30
iface vmbr30 inet manual
bridge_ports bond0.30
bridge_stp off
bridge_fd 0
Each VM will then be instructed to use the following:
DHCP
Code:
auto eth0
iface eth0 inet dhcp
Your switch or something else will then assign addresses to clients on vlan 30

Static
Code:
auto eth0
iface eth0 inet static
address 10.0.30.1
netmask 255.255.255.0
gateway 10.0.30.254
Traffic is then routed either by your switch or a router assigned 10.0.30.254 on vlan 30.
Click to expand...

Now I need to bondle 3 network cards for redundancy and performance..

The only way to do this is using a bond so i create bond0 with port eth1 eth2 and eth3 in a LACP configuration AKA 802.3ad

Since only bridges can be chose for a network interface i figure I somehow have to make bond0 part of a bridge.. so would that be done by creating a new bridge with interface bond0 as the bridgeport and than use the mir approach to assign that bridg a VLAN?

or would that have to be done another way?

PS: I configured no IP for the bond0.. i just want it to act as LACP and nothing else.. will that work?

THANKS

Casper
 
"Since only bridges can be chose for a network interface i figure I somehow have to make bond0 part of a bridge.. so would that be done by creating a new bridge with interface bond0 as the bridgeport and than use the mir approach to assign that bridg a VLAN?"

This will work nicely. This is how I do it.
Code: 
auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto bond0
iface bond0 inet manual
    slaves eth1 eth2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
    bond_lacp_rate fast
And the create a bridge for each vlan involved.
 
Hi again,

So i made vlan 2 and wanted to make a bridge on bond0 for that..

So i went to /etc/network/interfaces/ and added this:

Code: 
auto vmbr1
         iface vmbr1 inet static
          bridge_ports bond0.2
          bridge_stp off
          bridge_fd 0

I than saved it and in terminal i wrote ifup vmbr1 and got this:

Code: 
root@proxmox02:~# ifup vmbr1Missing required variable: address
Missing required configuration variables for interface vmbr1/inet.
Failed to bring up vmbr1.

can't i create a bridge without and IP address?

THANKS

Casper
 
Last edited:
"can't i create a bridge without and IP address?". You can depending on choosing the proper mode. static means a static IP so a bridge without IP will need to be in mode manual. Or static 0.0.0.0/0. But choose manual as this is nicer;-)
Code: 
auto vmbr1
         iface vmbr1 inet manual
          bridge_ports bond0.2
          bridge_stp off
          bridge_fd 0
 
Last edited:
Dear all, I am new to proxmox and I would like to achieve a 3 nodes HA cluster with the following abilities: 1- Cisco Router with 3 subinterfaces (30-40-50)--->Cisco Switch with Etherchannel group1 to allow trunked vlans 30 -40-50 --> 3 physical servers with Bonded interfaces. Router config: ============= interface GigabitEthernet0/1.30 description Vlan30 encapsulation dot1Q 30 ip address 192.168.30.1 255.255.255.0 interface GigabitEthernet0/1.40 description Vlan40 encapsulation dot1Q 40 ip address 192.168.40.1 255.255.255.0 interface GigabitEthernet0/1.50 description Vlan50 encapsulation dot1Q 50 ip address 192.168.50.1 255.255.255.0 Switch Config: =============== interface Port-channel1 description =>Proxmox HA Cluster switchport trunk encapsulation dot1q switchport trunk allowed vlan 30,40,50 switchport mode trunk Int range G1-6 Description =>Proxmox Cluster Channel-Group1 mode on interface GigabitEthernet 8 description Trunk->Router switchport trunk encapsulation dot1q switchport trunk allowed vlan 30,40,50 switchport mode trunk Is it possible to create the Vlans on the Proxmox nodes via Webgui especially that in PVE3.2 we have vswitch feature, then lets say i want to create per ex: 2 openvz VMs on the cluster: Openvz VM1: Distro Ubuntu 12.04 Eth0: 192.168.30.2/24 ( This subnet is reserved to Vlan 30) Eth1: 192.168.50.4/24 ( This subnet is reserved to Vlan 50) Openvz VM2: Distro Debian 7.5 Eth0: 192.168.30.5/24 ( This subnet is reserved to Vlan 30) Eth1: 192.168.40.3/24 ( This subnet is reserved to Vlan 40) Eth2: 192.168.50.6/24 ( This subnet is reserved to Vlan 50) Is it possible to create this scenario that allows us to add VM's that belong to many Vlans at the same time with possibility to add new vlans just by allowing the Vlan in the channel-group,create it on the Proxmox nodes via web gui and then create inside the VM itself a new interface (Same as Vmware and Citrix-Xen...) Like this when adding a new Vlan we only require to restart the network service inside the VM only instead of restarting the whole node which is not appropriate. Can anyone help me with a step by step detailed config on the proxmox side? I can add an attached network diagram to make the picture more clear but somehow i am not able to attach a picture here. Thank you in advance Berny
 
Last edited:
mir said:
Your text formatting is foobar making it almost impossible to read. Please proper format the text.
Click to expand...

Hi Mir,

Sorry for this i tried to fix it many times but dunno why the site is putting the words in sequence.

I sent the text attached with a network diagram , i hope now it is clear :D

Regards
Berny
 

Attachments

  • Proxmox Cluster.jpg
    Proxmox Cluster.jpg
    78.5 KB · Views: 52
  • Proxmox cluster.txt
    2.3 KB · Views: 28
berny said:
Hi Mir, Sorry for this i tried to fix it many times but dunno why the site is putting the words in sequence. I sent the text attached with a network diagram , i hope now it is clear :D Regards Berny
Click to expand...
Hi Mir, Any update on the below ?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back