Cannot make VLANS work after hours of searching

[tgwaku]

Hi,

normally i dont make post, im use to spending a long time researching and getting the satisfaction of finding the answer myself.
Im sure im probably already come across the answer but im unable to make it work in my lab environment.
after hours and hours (probably over 20) of looking for the answer and testing things. i've decided to make a post (sorry).

im trying to ping my ubuntu server VM from my desktop pc
the ubuntu server is on 172.16.100.3/24 - vlan 100 (default gw 172.16.1.1) and the desktop pc is 172.16.1.6/24 - vlan 1 [native] (default gw 172.168.1.1)

The Details:
I have a proxmox server running proxmox version 8.2.2
the proxmox server has two NIC's 1g and 2.5g. (installed the driver for the 2.5g to get it working)
1x ubiquiti flex mini switch which connects my proxmox server, main desktop and router (mini pc with pfsense)
i have configured vlans on my pfsense mini pc (router) to tag the lan port with 3 vlans (20,50 and 100)
pfsense firewall has rules setup as any source/port etc to any destination/port etc to be allowed.

IP Configuration:
pfsense has a subnet of 172.16.1.1/24 as the routers ip address.
the flex mini switch is 172.16.1.2
my main computer (which is plugged into the flex mini switch) is configured with 172.16.1.6 (from dhcp)

proxmox is configured with 172.16.1.3/24 with 172.16.1.1 as the gateway (on the vmbr0 bridge which i believe is the management ip of proxmox)
as with the above line, i've also tried with 172.16.1.3/24 as the NIC ip enp4s0.

the second NIC (enp42s0) has a bridge, vmbr1 which is connected to the ubuntu VM. and i've configured in the network settings of the proxmox host for vmbr1 to have an ip of 172.16.100.3 (and i've set vlan 100 on the virto network adapter in the ubuntu vm hardware/network adapter settings)

also a vlan.100 was configured in the proxmox /etc/network/interfaces file from shell. (see below conf file)

the ubuntu server has vlans configured in the netplan file (id 100) proper indentation for yaml was used (i read the documentation)
firewall is off for the ubuntu vm in proxmox (did this as a test and left it off to eliminate as an issue)

now i will paste my configuration files.

Proxmox (/etc/network/interfaces)

auto lo
iface lo inet loopback

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.16.1.3/24
        gateway 172.16.1.1
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Proxmox Management

auto vmbr1
iface vmbr1 inet static
        address 172.16.100.3/24
        bridge-ports enp42s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1.100
iface vmbr1.100 inet manual
        address 172.16.100.3
        gateway 172.16.1.1


source /etc/network/interfaces.d/*

Ubuntu Server VM Netplan yaml:


pfsense vlans:



as far as im aware that all the info.

cannot ping between 172.16.1.6 (main pc) and ubuntu server (172.16.100.3)
also, ubuntu server doesn't even have internet access. (cant ping 8.8.8.8)

if i set ubuntu server (and related network settings) to 172.16.1.x subnet (and remove vlan settings) it gets internet and pingable.

sorry for the long post, just trying to give as much info as possible.

Thanks for any help or suggestions!

 

[uberarchiver]

I myself am using openvswitch after some wiggling and stuff. Got stuff to work and I am satisfied.

Like so:

auto mgmtvlan10
iface mgmtvlan10 inet static
        address 192.168.187.5/24
        gateway 192.168.187.254
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        ovs_options tag=10

And:

auto bond0
iface bond0 inet manual
        ovs_bonds enp1s0f0np0 eno1
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_options bond_mode=active-backup other_config:bond-primary=enp1s0f0np0

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports mgmtvlan10 bond0

the vlan number 10 is for managing the box.

Am now running passed through NIC but used multiple virtio net-adapters on the pfsense. All of them had no tag assigned and similar setup to yours with vlans configured in pfsense.

Dunno if this helps any.

Cheers..

 

[weehooey-bh]

You indicated your PVE server has two NICs, but the interfaces file only shows one. Your vmbr1 shows an interface that is not in your interfaces file. I would expect to see a stanza that looks like this:

iface enp42s0 inet manual

You have an IP address specified twice for your PVE server. You only need it once. It should be removed from vmbr1. Although, it is unclear if you actually want your PVE server to have an IP address on VLAN 100. You have given your PVE server and your Ubuntu VM the IP address of 172.16.100.3. If you did not intend to give your PVE server an IP address on VLAN 100, remove the address lines from vmbr1 and vmbr1.100.

Even if you did intend to give your PVE server an IP address on VLAN 100, you should remove the gateway line. You should only have one gateway per server in most cases.

Your Ubuntu VM has the wrong gateway. It should be the IP address of the SERVER (VLAN 100) interface on the pfSense device, and it should be in the 172.16.100.0/24 subnet.

If this does not fix your issue, please post the contents of your Ubuntu VM's configuration file:

/etc/pve/nodes/<server_name/qemu-server/<VMID>.conf

 

[louie1961]

Here's how I do it. I have a pfSense firewall, a managed switch (mokerlink, generic Chinese stuff), and my Proxmox host with more than one NIC:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr1
iface vmbr1 inet static
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr0.10
iface vmbr0.10 inet static
address 10.10.10.3/24
gateway 10.10.10.1

A couple of points to note: Bridges inside of Proxmox do not need an IP address. As I understand it at least, you only need one IP address in your /etc/network/interfaces file, and that is the IP address of the web interface. You also only need one gateway if all the VLANs point to the same router or L3 switch. Your specific VMs or CTs can be placed on either bridge in their configuration (hardware) settings, and likewise, you can choose any VLAN that is valid for that bridge, inside of the VM hardware settings. At least that is how I successfully do this.

Also, you need to make sure the ports in your managed switch are properly configured. Once you change a vmbr to be VLAN aware, it will no longer work on a regular access/untagged port. You need to switch to a trunked/tagged port. In my set up I have my pfsense box plugged into a trunked/tagged port on the switch, and my VLAN aware wireless access point, and my VLAN aware proxmox hosts are also connected via trunked ports. All the other ports on my switch (30 in total) are set up as untagged access ports, meaning they only pass traffic for one VLAN, which ever is the PID for that port. I set this in my switch interface. Because all my VLANs have DHCP servers associated with them in pfSense, any device I plug into an untagged port get an IP address from DHCP on that specific VLAN. So my Synology NAS and Terramaster NAS, as well as things like my Ring alarm, raspberry pi devices, PCs, etc. all get assigned to a VLAN based on which port they are plugged into on the switch, and all IPs are assigned by DHCP. Same for Wifi. Each SSID is assigned to a specific VLAN by my WAP. If you log into the guest SSID, you will be on the Guest VLAN no matter what. I control inter VLAN traffic with specific firewall rules inside of pfSense.

 

[tgwaku]

Hey all! thanks so much for all the replies!

i've got a solution to my problem (you'll hate me for it).

turns out my pfSense firewall rules that i was creating for allow any to any as a test etc was created with only TCP traffic selected (seems to select this by default unless i've somehow configured it that way :S).
once i changed this to "any" protocal all the ping/icmp request started flowing as expected (DOH!)

all this was being done on a L2 switch (which was preventing me from doing dhcp relay/ip helpers to setup a multi subnet dhcp super scope to assign dhcp leases to devices plugged into an access port on e.g. vlan 20 to receive a dhcp address of 172.16.20.2,3,4,5 etc. or at least this is what i current believe i need to do to get that to work which requires a L3 switch [please correct me if im wrong ] )

so now i've now pulled my cisco 3560G switch out from under my bed. (funny story, had proxmox plugged into port 3 and couldn't ping or access webgui and tried for hours with trunk, vlans ip configs etc etc. turns out the port gi0/3 was faulty or maybe not faulty but just caked in dog hair and dust).

so now im able to access my proxmox server and i'll implement the ip recommendations above to the interfaces file.

one question i have is do i need to have an IP assigned on the vmbr1.100 interface or even the vmbr1 interface if i have it configured statically on the virtual machine OS.

and hopefully someone can clear up the having multiple subnets on pfsense and using windows server vm on proxmox to configure dhcp superscope and dhcp lease the different subnets to their respective/appropraite clients. e.g. i want my computer in port gi0/2 on the 3560G (trunk vlan 20,10,100,50) to receive a dhcp lease in the network 172.16.20.0/24 and my WAP connected clients on gi0/5 to receive dhcp in the network of 172.16.50.0/24.... and then the plan is to firewall connections between them and servers etc and other clients or networks.

THANK you, even if none of the above gets answered, you've imparted knowledge on my that has grown my understanding of this stuff immensely! and i feel privliged to have even be worthy of such clear and consice responses from incredibly learnered gentlemen!

 

[weehooey-bh]

one question i have is do i need to have an IP assigned on the vmbr1.100 interface or even the vmbr1 interface if i have it configured statically on the virtual machine OS.

No. Only IP addresses for your PVE host should be on those interfaces. Any IP addresses on those interfaces will be used by the PVE host.