[SOLVED] VLAN bridge problem with gateways

[lifeboy]

I have a situation where I need different networks for different application groups on a single pmx cluster. To achieve that I have added VLAN's to the config as below.



I have repeated the config on each node of the cluster.

I have various virtual machines that are configured on the various bridges, so, for example, a Windows 2016 server has a NIC attached to vmbr4 with ip address 192.168.151.100.

I have a pfSense firewall running with interfaces on all the bridges configured and ip addresses in the relevant ipv4 range. So vmbr4 on the pfSense firewall is set to 192.168.151.254 which should act as a gateway for this LAN.

The problem is that I'm not able to ping this address although I have a rule to allow pings on the firewall. I can for example ping the bridge ip address 192.168.151.4 from the VM on 192.168.151.100, but I can't ping 192.168.141.3 which is on another node.

Why is this? Surely if these nodes all have a bridge on the same VLAN, then I should be able to "see/ping" all the addresses on that VLAN regardless of the nodes. What should I do that get this to work the way I expect it to?

 

[lifeboy]

As a followup question: Is it enough that I create the VLAN on the enf7f0 nic or should I also tag the bridge interface with vlan 35 when I assign the bridge vmbr4 to a VM NIC?

 

[lifeboy]

Another issue that I note it that Proxmox doesn't seem to be aware that there are multiple networks on these nodes. I see that when I try to add a gateway to the bridge for VLAN 35 for example, I get an error saying there's a gateway already. But the gateway it displays is for a different network.



Surely each VLAN should be allowed to have it's own gateway? This indicates to me that my nodes are not as VLAN aware as I would like them to be.

 

[spirit]

Another issue that I note it that Proxmox doesn't seem to be aware that there are multiple networks on these nodes. I see that when I try to add a gateway to the bridge for VLAN 35 for example, I get an error saying there's a gateway already. But the gateway it displays is for a different network.

View attachment 31677

Surely each VLAN should be allowed to have it's own gateway? This indicates to me that my nodes are not as VLAN aware as I would like them to be.

you don't need to define network or gateway on the vmbr, to get them works inside the vms.
it's only to setup ip for the proxmox hypervisor in this vlan.
(and you can't have 2 default gw)

 

[lifeboy]

you don't need to define network or gateway on the vmbr, to get them works inside the vms.
it's only to setup ip for the proxmox hypervisor in this vlan.
(and you can't have 2 default gw)

Ok, that is what I was thinking when I started out on this effort. But even inside the vms I can't get a network connection or ping the other ip adresses on the VLAN (35 in this case) which doesn't make sense to me.

 

[lifeboy]

However, your answer does not solve my problem in the first instance. Any assistance you can offer there?

 

[spirit]

However, your answer does not solve my problem in the first instance. Any assistance you can offer there?

I really don't known, it should work out of the box.
Are you sure that physical switch allow theses vlans ?

 

[lifeboy]

I really don't known, it should work out of the box.
Are you sure that physical switch allow theses vlans ?

It's a Mellanox switch on which none of the VLAN are set up because I'm not limiting VLANs to specific physical ports. Do you think that I have to do something special on the switch to get VLAN traffic to be allowed? Simply tagging ip packets with a VALN tag should be enough, not?

 

[spirit]

It's a Mellanox switch on which none of the VLAN are set up because I'm not limiting VLANs to specific physical ports. Do you think that I have to do something special on the switch to get VLAN traffic to be allowed? Simply tagging ip packets with a VALN tag should be enough, not?

if you use mellanox onyx os:

at minimum, you need to declare the allowed vlans globally

#conf t
# vlan 10
# vlan 11
# vlan 12
..
# exit

and configure switch port as trunk

  interface ethernet xxxx switchport mode trunk

 

[lifeboy]

if you use mellanox onyx os:

at minimum, you need to declare the allowed vlans globally

#conf t
# vlan 10
# vlan 11
# vlan 12
..
# exit

and configure switch port as trunk

  interface ethernet xxxx switchport mode trunk

I have added the VLAN's and made the ports hybrid. For some reason "trunk" disconnected things (probably because we are using VLAN 1).

The problem has been solved now.