[SOLVED] VLAN setup for Cluster and Storage

[boopzz]

I have 2 Proxmox hosts. One is at my network edge with a pfSense VM doing VLAN routing. Another sits on a couple of VLANs, I have a VLAN for Client to Server connection, a VLAN for Storage (NFS) and a VLAN for the Clustering (advised in the cluster wiki).

I have tried a few different config but I can figure out what Im doing wrong. I have a LAN bridge for all my VLANs which I then attach VLANs with a tagged interface, this all works fine.

The problem is when I try and have a new bridge for my Storage and Cluster VLANs. I create a bridge and assign an IP, I can then talk between all my VMs on that bridge but not out to the other host. How should I be creating the link to the switch?

vmbr0 is attached to eth0, has my IP for mgmt of PVE and has "VLAN Aware" tagged.
vmbr255 is my Storage bridge and has an IP configured

How do I setup the ports/slaves part?
I've tried using eth0.255
I've tried adding an interface called vlan254 and used "vlan-raw-device eth0.254" and attaching the bridge to this but it doesnt seem to like it either.

Whats the correct way of setting up clustering and storage networks for clustered nodes?

 

[boopzz]

I've been thinking and I think the correct method is to have an interface for VMs tagged as VLAN 255 and passed to the LAN bridge (vmbr0). This works fine. But then I dont know how to give the Proxmox box an interface address on this VLAN?

I suppose I need to add a VLAN interface but dont know how

 

[manu]

Actually although it is possible to do everything with VLANS, maybe you could consider adding a second physical NIC to your pve host, so the storage traffic on vmbr255 does not impact the VMs own traffic on vmbr0 ?

 

[boopzz]

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

iface eth1 inet manual

auto eth0.254
vlan_raw_device eth0

auto eth0.255
vlan_raw_device eth0

auto vmbr0
iface vmbr0 inet static
address 192.168.58.241
netmask 255.255.255.0
gateway 192.168.58.254
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
#LAN

auto vmbr254
iface vmbr254 inet static
address 192.168.254.241
netmask 255.255.255.0
bridge_ports eth0.254
bridge_stp off
bridge_fd 0
#Cluster VLAN

auto vmbr255
iface vmbr255 inet static
address 192.168.255.241
netmask 255.255.255.0
bridge_ports eth0.255
bridge_stp off
bridge_fd 0
#Storage VLAN

This is my config, VMs attached to vmbr255 I can see on the switch (MAC addresses) but the Proxmox is incorrectly dropping onto the untagged VLAN

 

[boopzz]

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

iface eth1 inet manual

auto eth0.255
iface eth0.255 inet manual
  vlan_raw_device eth0

auto vmbr0
iface vmbr0 inet static
  address  192.168.58.241
  netmask  255.255.255.0
  gateway  192.168.58.254
  bridge_ports eth0
  bridge_stp off
  bridge_fd 0
  bridge_vlan_aware yes
#LAN

auto vmbr255
iface vmbr255 inet static
  address  192.168.255.241
  netmask  255.255.255.0
  bridge_ports eth0.255
  bridge_stp off
  bridge_fd 0
#Storage VLAN

I'm just using my generic server VLAN for the cluster VLAN, it was too much of a pain otherwise. The config above is what Im using, able to add a NIC for any VMs on the 255 VLAN and attach to vmbr255, everything else goes through vmbr0.