[SOLVED] Problem changing proxmox ip - further info about VLAN puzzle

[Nachvee]

​

Heya all, and thanks for reading.

Started my proxmox learning journey about 6 months ago and have a fully working setup as follows:

ISP -> Asus 58u router (wireless off) -> Proxmox -> Opnsense -> 2 Omada Access Points (meshed)

Asus router: Lan 192.168.13.1
Proxmox: bridges vmbr0 192.168.13.50 GW 192.168.13.1, vmbr1 192.168.13.51
Opnsense VM: WAN 192.168.13.63 GW 192.168.13.1, LAN 192.168.63.1 (Vlan on .10)
VM for omada controller on .63 network.

The .13 network is Proxmox management, the .63 is my main LAN and the .10 vlan is for IoT devices.
So far so good, throughput maxed to my internet contract, firewall and zenarmor logs show firewall is working well, nmap and nessus scans show network comms are where I want them ...... however, the Asus 58u has just gone end of life so no more updates!!

As a result I tried an experiment to eventually remove the asus and use OpnSense VM as primary router:
Stage 1 was to change proxmox IP to use opnsense LAN.

Here`s where the problem is .... I changed proxmox bridges to vmbr0: 192.168.63.50 GW 192.168.63.1 and vmbr1: 192.168.63.51, /etc/hosts changed as well. Rebooted and network all functions great except proxmox management. No access to gui, host unreachable on ping, no SSH. All VM`s are functional including internet access, so opnsense is working fine (it`s gui is accessible too). Proxmox accessible locally, /etc/network/interfaces and /etc/hosts look like they should but dead on the network.

Am I missing something very basic here? Being self taught and not in the `biz`, my knowledge is sketchy at best. Is there some flaw in my understanding of Linux bridges perhaps?

I realise the easiest option is to buy a new gateway router, or build an OpnSense box, or even flash OpenWrt onto the asus, and I may well end up doing one of these, but I love problem solving and this one has me stumped.

Thanks again for your consideration.

 

[Dunuin]

Proxmox: bridges vmbr0 192.168.13.50 GW 192.168.13.1, vmbr1 192.168.13.51

You shouldn't give a host 2 IPs of the same subnet or this might cause problems.

Did you also change the /etc/resolv.conf?
What does your /etc/network/interfaces exactly look like?

 

[Nachvee]

Thanks for reply

Wasn`t aware of your first point, I`ll remedy as soon as.

/etc/network/interfaces as it is now (I`ve reverted back to original to get gui back, this works):

auto lo
iface lo inet loopback

iface enx00e04c68a17c inet manual

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.13.50/24
        gateway 192.168.13.1
        bridge-ports enx00e04c68a17c
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.13.51/24
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

When I changed this I simply changed the .13 to .63 on all entries. Same in /etc/hosts.

I did not change /etc/resolv.conf. I didn`t come across this in my googling.

 

[Nachvee]

So if I take all your points and changed /etc/network/interfaces to:

auto lo iface lo inet loopback iface enx00e04c68a17c inet manual iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.63.50/24 gateway 192.168.63.1 bridge-ports enx00e04c68a17c bridge-stp off bridge-fd 0 auto vmbr1 iface vmbr1 inet static address 192.168.25.1/24 bridge-ports eno1 bridge-stp off bridge-fd 0

And /etc/hosts to:

127.0.0.1 localhost.localdomain localhost 192.168.63.50 pve.nuc.home pve

And /etc/resolv.conf to:

search nuc.home nameserver 192.168.63.1

I should be good to go?

 

[Dunuin]

Which bridge is your OPNsenses virtual NIC for LAN attached to?

Why "192.168.25.1/24"? You got two physical NICs and you are virtualizing your router. So in case you are not using tagged VLANs you should use one physical NIC attached to vmbr0 for your LAN side with "address 192.168.63.50/24" + "gateway 192.168.63.1" and one physical NIC attached to vmbr1 for your WAN side and there you usually don't want to set any IP because management that isn't behind a firewall is a bad idea. But even if you give your PVE node a IP on the WAN side, shouldn't that then be something in the 192.168.13.0/24 subnet?

Your bridges are also not vlan aware, so they can't work with tagged packets. So how do you use your IoT VLAN? Your OPNsense shouldn't be able to use any VLANs without it.

 

[Nachvee]

Which bridge is your OPNsenses virtual NIC for LAN attached to?

Why "192.168.25.1/24"? You got two physical NICs and you are virtualizing your router. So in case you are not using tagged VLANs you should use one physical NIC attached to vmbr0 for your LAN side with "address 192.168.63.50/24" + "gateway 192.168.63.1" and one physical NIC attached to vmbr1 for your WAN side and there you usually don't want to set any IP because management that isn't behind a firewall is a bad idea. But even if you give your PVE node a IP on the WAN side, shouldn't that then be something in the 192.168.13.0/24 subnet?

Your bridges are also not vlan aware, so they can't work with tagged packets. So how do you use your IoT VLAN? Your OPNsense shouldn't be able to use any VLANs without it.

OpnSense LAN is attached to vmbr1.

It is becoming apparent that my original setup, while functional, is non ideal! Thank you for your patience. The bridges are not VLAN aware, but the Omada AP`s are. Is this the reason VLAN10 is fully functional and isolated from the LAN?

Looking at your questions/explanations, have I got the bridges attached to OpnSense the wrong way around!? At the moment vmbr0 = WAN,
vmbr1 = LAN. With the asus router in the loop I do have an IP in the 192.168.13.0/24 subnet, but eventually I would like to change this to pppoe to ISP.

 

[Dunuin]

It is becoming apparent that my original setup, while functional, is non ideal! Thank you for your patience. The bridges are not VLAN aware, but the Omada AP`s are. Is this the reason VLAN10 is fully functional and isolated from the LAN?

Jup, no VLAN tagged packets could pass the bridges, so no data from your APs should reach the OPNsense VM.

OpnSense LAN is attached to vmbr1.


Looking at your questions/explanations, have I got the bridges attached to OpnSense the wrong way around!? At the moment vmbr0 = WAN,
vmbr1 = LAN. With the asus router in the loop I do have an IP in the 192.168.13.0/24 subnet, but eventually I would like to change this to pppoe to ISP.

Then you should have "address 192.168.63.50/24" + "gateway 192.168.63.1" at vmbr1 and no IPs/gateways (beside the virtual WAN NIC of your OPNsense VM) at vmbr0 and use that vmbr0 as your WAN.

 

[Nachvee]

Jup, no VLAN tagged packets could pass the bridges, so no data from your APs should reach the OPNsense VM.

This is strange then because all the devices on VLAN10 can connect to the internet and each other. (OpnSense firewall rules prevent them from reaching LAN). I can join vlan10 by laptop through omada AP and access internet and other devices.

Then you should have "address 192.168.63.50/24" + "gateway 192.168.63.1" at vmbr1 and no IPs/gateways (beside the virtual WAN NIC of your OPNsense VM) at vmbr0 and use that vmbr0 as your WAN.

Thankyou, I shall give it a try.
I do appreciate your patience on this. Early on I used tutorials/guides to set up and tried to build my understanding. It is relatively easy to build some quite complex set ups from web guides etc., but I fear I am still not at a point that I fully understand what is happening at the network layers, hence troubleshooting can be difficult. I am in awe of all the people like your good self that have this knowledge, and take the time to pass it on.

 

[Dunuin]

This is strange then because all the devices on VLAN10 can connect to the internet and each other. (OpnSense firewall rules prevent them from reaching LAN). I can join vlan10 by laptop through omada AP and access internet and other devices.

Maybe your managed switch is stripping the VLAN tags? So AP send the packets tagged, switch will strip the VLAN tag and send it to PVE Node as untaggd traffic.

 

[Nachvee]

Maybe your managed switch is stripping the VLAN tags? So AP send the packets tagged, switch will strip the VLAN tag and send it to PVE Node as untaggd traffic.

Curiouser and curiouser, I don`t have a managed switch in this topology. Apart from a wired connection from base AP to proxmox, and one from proxmox to gateway router I solely use wifi.

 

[Dunuin]

Thats a strange setup. So its only router directly to PVE vmbr0 and from vmbr1 directly to the AP?
Then maybe your AP is only using VLAN internally to separate the different wifi networks but stripping the tags when talking to the gateway?

 

[Nachvee]

Maybe your managed switch is stripping the VLAN tags? So AP send the packets tagged, switch will strip the VLAN tag and send it to PVE Node as untaggd traffic.

Thats a strange setup. So its only router directly to PVE vmbr0 and from vmbr1 directly to the AP?
Then maybe your AP is only using VLAN internally to separate the different wifi networks but stripping the tags when talking to the gateway?

Very strange and you are correct: asus router -> vmbr0, base AP ->vmbr1. Could be tag stripping going on, but how then does OpnSense differentiate between the LAN and VLAN10?

Based on all your advice, how does this now look for config:

vmbro - no ip, no gateway
vmbr1 - 192.168.63.50/24 gateway 192.168.63.1
/etc/hosts - 192.168.63.50
/etc/resolv.conf - 192.168.63.50

vmbr0 set as WAN for OpnSense
vmbr1 set as LAN for OpnSense

Thanks.

 

[Dunuin]

/etc/resolv.conf - 192.168.63.50

This should point to your DNS server, so probably your OPNsense IP (192.168.63.1) on the LAN subnet.

For VLAN you should check the "vlan aware" checkbox of the vmbr1, empty "VLAN tag" for the virtio NIC of your OPNsenses LAN and you then can work with VLAN directly with your OPNsense VM (where you will have to create some VLAN interfaces and so on).

 

[Nachvee]

This should point to your DNS server, so probably your OPNsense IP (192.168.63.1) on the LAN subnet.

Sorry, I didn`t include this in my reply, but I did change /etc/resolv.conf to 192.168.63.1.

Had to make a change or 2 in opnsense DNS general settings, but .... ta da ... we have it up and running on new config! Many, many thanks for your help, you made me see things a lot clearer. I`ve also changed opnsense WAN to pppoe and connected direct to ont, so my goal of retiring the asus 58u is complete.

For VLAN you should check the "vlan aware" checkbox of the vmbr1, empty "VLAN tag" for the virtio NIC of your OPNsenses LAN and you then can work with VLAN directly with your OPNsense VM (where you will have to create some VLAN interfaces and so on).

It`s interesting that my vlans work fine as is, even with the new config, but I am going to follow this advice too, thank you. Who knows what may happen further down the line!

Am marking solved, Thanks again!

 

[Nachvee]

Thats a strange setup. So its only router directly to PVE vmbr0 and from vmbr1 directly to the AP?
Then maybe your AP is only using VLAN internally to separate the different wifi networks but stripping the tags when talking to the gateway?

Hi again, running 24 hours rock solid Thanks again, couldn`t have done it without your help.

Just for information, together with another contributor on another forum, I think we have cracked the VLAN puzzle:

In the proxmox gui -> node -> network page there is a help button that I forgot about (the other person found it). In the help page under the section VLAN`s it says this:

"VLAN for Guest Networks

Proxmox VE supports this setup out of the box. You can specify the VLAN tag when you create a VM. The VLAN tag is part of the guest network configuration. The networking layer supports different modes to implement VLANs, depending on the bridge configuration:

1. VLAN awareness on the Linux bridge: In this case, each guest’s virtual network card is assigned to a VLAN tag, which is transparently supported by the Linux bridge. Trunk mode is also possible, but that makes configuration in the guest necessary.

2. "traditional" VLAN on the Linux bridge: In contrast to the VLAN awareness method, this method is not transparent and creates a VLAN device with associated bridge for each VLAN. That is, creating a guest on VLAN 5 for example, would create two interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.

3. Open vSwitch VLAN: This mode uses the OVS VLAN feature.

4. Guest configured VLAN: VLANs are assigned inside the guest. In this case, the setup is completely done inside the guest and can not be influenced from the outside. The benefit is that you can use more than one VLAN on a single virtual NIC."

So there are 4 methods of creating vlans in proxmox and you don`t always need the bridge to be vlan aware. This last example (number 4) I think explains my scenario. None of my guest VM`s are on a vlan hence no sign of them in /etc/network/interfaces. The vlan for my network is configured inside the OpnSense VM and mirrored by the Omada access points. The bridge must be able to natively pass the vlan altered packets.

What do you think?

 

[Dunuin]

What do you think?

According to your /etc/network/interfaces you are definitely not using method 1 to 3. And I thought for method 4 you still have to enable a vlan aware feature of the bridge. But maybe I'm wrong there.

 

[Nachvee]

According to your /etc/network/interfaces you are definitely not using method 1 to 3. And I thought for method 4 you still have to enable a vlan aware feature of the bridge. But maybe I'm wrong there.

Still working fine, without VLAN aware and I created a new vlan20 for guests on opnsense and Omada AP with no problems. Thanks for all your help.