[SOLVED] Network issue when using only IP on VLANs

[kifeo]

Hi,

I wanted to report an issue I got when using only vlan interfaces.

The gateway would not get up due to : "failed: returned 2 (Error: Nexthop device is not up.)"

This error was due to the underlying interface not up because it does not have an IP.

The workaround is to modify the physical interface with :
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down


Full output of the /etc/network/interface :

auto enp2s0

iface enp2s0 inet manual

    pre-up ifconfig $IFACE up

    post-down ifconfig $IFACE down


auto enp2s0.5

iface enp2s0.5 inet static

    address 10.0.5.12/24

    gateway 10.0.5.1

#proxmox vlan


auto enp2s0.1

iface enp2s0.1 inet static

    address 192.168.1.10/24


auto vmbr0

iface vmbr0 inet manual

    bridge-ports enp2s0

    bridge-stp off

    bridge-fd 0

    bridge-vlan-aware yes

    bridge-vids 2-4094

 

[spirit]

as you use a vlan-aware bridge, a better setup could be to tag vmbr0.x instead enp2s0.X

auto enp2s0
iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.1
iface vmbr0.1 inet static
    address 192.168.1.10/24

auto vmbr0.5
iface vmbr0.5 inet static
    address 10.0.5.12/24
    gateway 10.0.5.1

 

[kifeo]

Hi,
Thanks for the advice, I use these only for the host access and communication, not for the VMs, and migrated proxmox cluster IPs from VLAN ID1 to 5, so I maybe will revert the native vlan to ID 5.

How is it better ? just so that I can remove the added workaround in the interface itself?
Thanks !

 

[spirit]

Hi,
Thanks for the advice, I use these only for the host access and communication, not for the VMs

yes, it's not a problem here vmbr.X are only internal interfaces on specifc vlan to add an ip address.
with vlan-aware bridge, each vm port is tagged directly.


I don't known why your config is not working, because it should works too. you could also try to install "ifupdown2" package to manage /etc/network/interfaces, it could work

 

[kifeo]

it seems to be documented here : https://wiki.debian.org/NetworkConfiguration#Bringing_up_an_interface_without_an_IP_address
if there is no ip address, then the interfaces does not come up and generates the error at 'gateway' instruction

ifupdown2 was installed

 

[spirit]

if there is no ip address, then the interfaces does not come up and generates the error at 'gateway' instruction

yes, of course, you need to have an ip configured in same subnet, to be able to define a gateway.

but you have an ip here

    address 10.0.5.12/24
    gateway 10.0.5.1

that's why I don't understand why it's not working

ifupdown2 was installed

it was already installed with your config with enp2s0.5 && enp2s0.1 ?
if yes, that's remember me an bug I have reported some months ago:
https://github.com/CumulusNetworks/ifupdown2/issues/124

can you try to write enp2s0.5 && enp2s0.1 after vmbr0 ?
(if you use proxmox gui for network configuration, the order should already be correct)

auto enp2s0
iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto enp2s0.5
iface enp2s0.5 inet static
    address 10.0.5.12/24
    gateway 10.0.5.1

auto enp2s0.1
iface enp2s0.1 inet static
    address 192.168.1.10/24

 

[kifeo]

sorry I did not try, I reverted to the native vlan to be the proxmox vlan 10.0.5.0/24 as the suggested workaround does not work at the end, sometime the enp2s interface does not go up even with the pre-up ifconfig $IFACE up.

However, as documented on debian Wiki network page, using a bridge will automatically activate the interface, so your suggestion might work.

 

[castes]

I've been running into the same issue with the gateway not set after reboot with error "Nexthop device is not up".
As a workaround I'm using the "pre-up" command like: 'pre-up ip link set dev enp1s0f1 up'

Previous config which renders missing gateway after reboot:

iface enp1s0f1 inet manual

iface vmbr0 inet manual
        bridge_ports enp1s0f1
        bridge_stp off
        bridge_vids 100
        bridge_vlan_aware 1

auto vlan100
iface vlan100 inet static
        address 10.10.10.1/24
        gateway 10.10.10.250
        vlan-raw-device enp1s0f1

Using "pre-up" statement:

iface enp1s0f1 inet manual

iface vmbr0 inet manual
        bridge_ports enp1s0f1
        bridge_stp off
        bridge_vids 100
        bridge_vlan_aware 1

auto vlan100
iface vlan100 inet static
        # For gateway to be applicable, nexthop device has to be up
        pre-up ip link set dev enp1s0f1 up
        address 10.10.10.1/24
        gateway 10.10.10.250
        vlan-raw-device enp1s0f1

Alternative working solution, attach the vlan to the bridge:

iface enp1s0f1 inet manual

iface vmbr0 inet manual
        bridge_ports enp1s0f1
        bridge_stp off
        bridge_vids 100
        bridge_vlan_aware 1

auto vlan100
iface vlan100 inet static
        address 10.10.10.1/24
        gateway 10.10.10.250
        vlan-raw-device vmbr0

 

[spirit]

@castes

if you use vlan-aware bridge, you should tag bridge instead the physical interface

iface vmbr0 inet manual
        bridge_ports enp1s0f1
        bridge_stp off
        bridge_vids 100
        bridge_vlan_aware 1

auto vmbr0.100
iface vmbr0.100 inet static
        address 10.10.10.1/24
        gateway 10.10.10.250

 

[castes]

@spirit

Although I'd prefer to tag the bridge instead of the physical interface, this leads to a conflict when using SDN with the same VLAN ID (i.e. zone using bridge vmbr0 and vnet using VLAN ID 100). The interface "vmbr0.100" will be used by the SDN-bridge.

SDN config in: /etc/network/interfaces.d/sdn

auto vnet1
iface vnet1
        bridge_ports vmbr0.100
        bridge_stp off
        bridge_fd 0
        alias sdn bridge

 

[spirit]

@spirit

Although I'd prefer to tag the bridge instead of the physical interface, this leads to a conflict when using SDN with the same VLAN ID (i.e. zone using bridge vmbr0 and vnet using VLAN ID 100). The interface "vmbr0.100" will be used by the SDN-bridge.

SDN config in: /etc/network/interfaces.d/sdn

auto vnet1
iface vnet1
        bridge_ports vmbr0.100
        bridge_stp off
        bridge_fd 0
        alias sdn bridge

you can add ip on vnet1 directly in /etc/network/interfaces in this case

iface vnet1
       address 10.10.10.1/24
        gateway 10.10.10.250

It'll be merged with the sdn configuration, only for this node

 

[castes]

@spirit

Yes, adding the ip on vnet1 in /etc/network/interfaces solves the conflict.

In order for the interface vnet1 to be shown on the network panel via the web interface, I've added the protocol and method statement:

iface vnet1 inet static
       address 10.10.10.1/24
       gateway 10.10.10.250

Do you know of any technical reason why it is best to tag the vlan-interface to the bridge instead of the physical interface?

 

[spirit]

Do you know of any technical reason why it is best to tag the vlan-interface to the bridge instead of the physical interface?

because with vlan-aware bridge, the vlan is done at bridge layer, and sometime linux kernel is routing wronly the traffic if you defined vlan also the the interface.
(the traffic could be redirect to the interface.vlan in priority, instead the bridge)

 

[castes]

because with vlan-aware bridge, the vlan is done at bridge layer, and sometime linux kernel is routing wronly the traffic if you defined vlan also the the interface.
(the traffic could be redirect to the interface.vlan in priority, instead the bridge)

Thanks for the explanation!

 

[grin]

Sidenote: the error can be prevented by reordering the interfaces in the config file:

1. physical interfaces
2. bridges
3. vlan interfaces