[SOLVED] Failed to reach Proxmox on VLAN

[saxc]

Dear community

I am pretty new to proxmox but like it so far except when it comes to VLAN's. I know there is ton of articles out there dealing with this topic and I tried quite some of them with no help so far.

I have a baremetal machine that connects with two sfp+ ports to my uniFi switch, where I have setup some VLAN's. Idea behind it is, to have one sfp for filesharing (vlan 55) and management (vlan 77) and the other for container / vm trafic (vlan 144) and some lab boxes to play with (vlan111).

On the UDM Pro I have configured the native vlan on port 10 (sfp+ 1) to be on 144, where the other, port 11 (sfp+ 2) is set to native vlan of 77. Both ports allow all available VLAN's

On proxmox I have configure the following in the interfaces file:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 144, 111, 107, 88, 55, 44

auto vmbr1
iface vmbr1 inet manual
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 55, 77

auto Management
iface Management inet static
   address 192.168.77.20/24
   gateway 192.168.77.1
   vlan-id 22
   vlan-raw-device vmbr1

auto Containers
iface Containers inet static
   address 192.168.144.10/24
   gateway 192.168.144.1
   vlan-id 144
   vlan-raw-device vmbr0

I have tried a lot of different variations but as soon as I change something in the inital interfaces config I can't reach proxmox anymore. Copying back the original also does not work anymore. To get it back to work I have to reinstall but then the process starts all over again.

Thus I was wondering, if I'm doing something wrong here?
I also added some explanatory prtnScrn of my unify config. SFP10 is where eno1 is connected to and eno2 is on SFP11

Any help is greatly appreciated, I would hate to go back to windows, just because I failed to get the vlans working....

 

[Undergrid]

Here's how I've set up VLAN's on my servers, and I use Ubiquiti switches too, so this works with them. This gives you VLAN's 10 and 666 ( a VLAN for naughty devices that misbehave). Just change the numbers after the "vmbr." to change the VLAN number.

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.224/22
        gateway 192.168.1.254
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        post-up /usr/sbin/ethtool -s enp2s0 wol g

auto vmbr0.10
iface vmbr0.10 inet static
        address 192.168.4.224/24

auto vmbr0.666
iface vmbr0.666 inet static
        address 192.168.200.224/24

Edit: one thing I note, the "22" VLAN ID for your Management interface isn't listed on vmbr1's bridge-vids list

 

[saxc]

Here's how I've set up VLAN's on my servers, and I use Ubiquiti switches too, so this works with them. This gives you VLAN's 10 and 666 ( a VLAN for naughty devices that misbehave). Just change the numbers after the "vmbr." to change the VLAN number.

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.224/22
        gateway 192.168.1.254
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        post-up /usr/sbin/ethtool -s enp2s0 wol g

auto vmbr0.10
iface vmbr0.10 inet static
        address 192.168.4.224/24

auto vmbr0.666
iface vmbr0.666 inet static
        address 192.168.200.224/24

Edit: one thing I note, the "22" VLAN ID for your Management interface isn't listed on vmbr1's bridge-vids list

Thanks a lot undergrid! I'll give it a try later today.
I have also noticed the "post up...." command. Care to explain what it does? I have also noticed, that you as well as another system of mine denotes "enp2..." as the 10gig interface, whereas on the system i am having issues with the interface is called eno1. I got a feeling, that the driver that is loaded for the 10gig sfp interface is some kind of generic driver (on the system i am having troubles with) and that might contribute to the issue i am seeing. Just a guess though...
I'll give it a shot and let you know.
Thanks up front though!

 

[Undergrid]

Thanks a lot undergrid! I'll give it a try later today.
I have also noticed the "post up...." command. Care to explain what it does? I have also noticed, that you as well as another system of mine denotes "enp2..." as the 10gig interface, whereas on the system i am having issues with the interface is called eno1. I got a feeling, that the driver that is loaded for the 10gig sfp interface is some kind of generic driver (on the system i am having troubles with) and that might contribute to the issue i am seeing. Just a guess though...
I'll give it a shot and let you know.
Thanks up front though!

The post-up occurs after link is established on vmbr0 (via enp2s0). In this case it's re-enabling wake-on-lan on the hardware port (which for some reason gets disabled on my hardware after you use WOL to wake the server. (the local power sub-station has a bad habit of blowing a fuse so the servers will shut down when they end up on UPS power. They don't start automatically because the engineers have a habit of flicking power on and off a few times while fixing things).

As for interface names, there's a standard for predictable names in linux (here's a reference), but basically "enp" means the name is based on the physical location of the connector. "eno" means its using a firmware/BIOS provided index for the port (usually the case for on-board devices). The device names themselves have got nothing to do with the drivers however.

 

[saxc]

Hi Undergrid

I finally found the issue why vLAN's were not working. It was indeed the NIC's (eno1 and eno2).
Usually I install the System, then create the vLAN's and finally update the system followed by a reboot. So I managed to check the interfaces this time at the very beginning. They were named eno1 and eno2 due to the driver that was selected for the devices.

However, AFTER the update when the system had rebooted, I looged into the system using IPMI and checked the interfaces that were listed... and guess what.... this time the interfaces were not called eno1 and eno2 as before but f****ing eno1np0 and eno2np1 with updated drivers!!!

So, of course the vLAN would not work, because the interfaces they were bound to did not exist anymore. After changing the interface names to the new and updated ones and a reboot all worked perfectly.

Thanks very much Undergrid, as it was your comment that made me think of checking the interfaces again.... who would have thought that an interface would change its name without changing hardware or renaming it manually yourself... at least in the Windows world that has never happened to me

So for anyone that has the same issue, check the interface names after the update using:

ip link show

and if necessary update your

/etc/network/interfaces

config accordingly.