vlan (not) routing, possible self inflicted issue..

mystique

New Member
May 26, 2013
9
0
1
tl;dr - does vmbr0 need to exist for routing to work in proxmox?

vlan lab at work just to toss some machines and test with..

Not sure what the issue is..

(history)
Normally we down vlan1 but we have received a grant and the equipment can not down vlan1.

So trying to make a lab to figure out how we are going to make these devices work..

Code:
cat /etc/network/interfaces

auto lo
iface lo inet loopback


auto vlan1
iface vlan1 inet manual
        vlan-raw-device eth0


auto vlan10
iface vlan10 inet manual
        vlan-raw-device eth0


auto vlan20
iface vlan20 inet manual
        vlan-raw-device eth0


auto vlan30
iface vlan30 inet manual
        vlan-raw-device eth0


auto vmbr1
iface vmbr1 inet static
        address 172.16.1.99
        netmask 255.255.255.0
        gateway 172.16.1.1
        bridge_ports vlan1
        bridge_stp off
        bridge_fd 0


auto vmbr10
iface vmbr10 inet static
        address 192.168.10.99
        netmask 255.255.255.0
        gateway 192.168.10.1
        bridge_ports vlan10
        bridge_stp off
        bridge_fd 0


auto vmbr20
iface vmbr20 inet static
        address 192.168.20.99
        netmask 255.255.255.0
        gateway 192.168.20.1
        bridge_ports vlan20
        bridge_stp off
        bridge_fd 0


auto vmbr30
iface vmbr30 inet static
        address 192.168.30.99
        netmask 255.255.255.0
        gateway 192.168.30.1
        bridge_ports vlan30
        bridge_stp off
        bridge_fd 0

Code:
root@proxmox:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr1           8000.0023ae8a408e       no              vlan1
vmbr10          8000.0023ae8a408e       no              vlan10
vmbr20          8000.0023ae8a408e       no              vlan20
vmbr30          8000.0023ae8a408e       no              vlan30
Code:
root@proxmox:~# vzlist      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       142          9 running   172.16.1.42     vm142.tcentral.lan
      1042          9 running   192.168.10.42   vm1042.tcentral.lan
      2042          9 running   192.168.20.42   vm2042.tcentral.lan
      3042          9 running   192.168.30.42   vm3042.tcentral.lan

From a physical host on vlan10

I am able to ping each host which is powered on. Downing the CT's obviously the pings fail.

Code:
 for i in 192.168.10.42 192.168.20.42 192.168.30.42 172.16.1.42; do ping -c 1 $i; done

PING 192.168.10.42 (192.168.10.42) 56(84) bytes of data.
64 bytes from 192.168.10.42: icmp_seq=1 ttl=64 time=0.077 ms


--- 192.168.10.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.077/0.077/0.077/0.000 ms
PING 192.168.20.42 (192.168.20.42) 56(84) bytes of data.
64 bytes from 192.168.20.42: icmp_seq=1 ttl=64 time=0.289 ms


--- 192.168.20.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.289/0.289/0.289/0.000 ms
PING 192.168.30.42 (192.168.30.42) 56(84) bytes of data.
64 bytes from 192.168.30.42: icmp_seq=1 ttl=64 time=0.315 ms


--- 192.168.30.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.315/0.315/0.315/0.000 ms
PING 172.16.1.42 (172.16.1.42) 56(84) bytes of data.
64 bytes from 172.16.1.42: icmp_seq=1 ttl=64 time=0.304 ms


--- 172.16.1.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.304/0.304/0.304/0.000 ms

Code:
root@proxmox:~# ip -s route

192.168.10.42 dev venet0  scope link
192.168.30.42 dev venet0  scope link
172.16.1.42 dev venet0  scope link
192.168.20.42 dev venet0  scope link
192.168.20.0/24 dev vmbr20  proto kernel  scope link  src 192.168.20.99
172.16.1.0/24 dev vmbr1  proto kernel  scope link  src 172.16.1.99
192.168.30.0/24 dev vmbr30  proto kernel  scope link  src 192.168.30.99
192.168.10.0/24 dev vmbr10  proto kernel  scope link  src 192.168.10.99
default via 172.16.1.1 dev vmbr1

Management should be vlan10 but I see default route as vlan1 (172.16.1.1) - because vmbr1 is first interface listed in /etc/network/interfaces?

Router (in this lab) is PfSense and is properly (afaict) setup to route packets and is NAT'ing..

10.20.0.1 gets me where I want to be..

Which is the default gw of the pfsense device..

Code:
root@proxmox:/etc/network# netstat -rnKernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.10.42   0.0.0.0         255.255.255.255 UH        0 0          0 venet0
192.168.30.42   0.0.0.0         255.255.255.255 UH        0 0          0 venet0
172.16.1.42     0.0.0.0         255.255.255.255 UH        0 0          0 venet0
192.168.20.42   0.0.0.0         255.255.255.255 UH        0 0          0 venet0
192.168.20.0    0.0.0.0         255.255.255.0   U         0 0          0 vmbr20
172.16.1.0      0.0.0.0         255.255.255.0   U         0 0          0 vmbr1
192.168.30.0    0.0.0.0         255.255.255.0   U         0 0          0 vmbr30
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 vmbr10
0.0.0.0         172.16.1.1      0.0.0.0         UG        0 0          0 vmbr1


root@proxmox:/etc/network# ping 10.20.0.1
PING 10.20.0.1 (10.20.0.1) 56(84) bytes of data.
64 bytes from 10.20.0.1: icmp_req=1 ttl=254 time=0.810 ms
64 bytes from 10.20.0.1: icmp_req=2 ttl=254 time=0.938 ms
^C
--- 10.20.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.810/0.874/0.938/0.064 ms

Code:
vzctl enter 142
entered into CT 142
[root@vm142 /]#

[root@vm142 /]# for i in 192.168.10.42 192.168.20.42 192.168.30.42 172.16.1.42; do ping -c 1 $i; done
PING 192.168.10.42 (192.168.10.42) 56(84) bytes of data.
64 bytes from 192.168.10.42: icmp_seq=1 ttl=64 time=0.030 ms


--- 192.168.10.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.030/0.030/0.030/0.000 ms
PING 192.168.20.42 (192.168.20.42) 56(84) bytes of data.
64 bytes from 192.168.20.42: icmp_seq=1 ttl=64 time=0.020 ms


--- 192.168.20.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.020/0.020/0.020/0.000 ms
PING 192.168.30.42 (192.168.30.42) 56(84) bytes of data.
64 bytes from 192.168.30.42: icmp_seq=1 ttl=64 time=0.028 ms


--- 192.168.30.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.028/0.028/0.028/0.000 ms
PING 172.16.1.42 (172.16.1.42) 56(84) bytes of data.
64 bytes from 172.16.1.42: icmp_seq=1 ttl=64 time=0.012 ms


--- 172.16.1.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.012/0.012/0.012/0.000 ms
From a CT I am able to ping the other CT's (this is good)

Code:
[root@vm142 /]# for i in 192.168.10.1 192.168.20.1 192.168.30.1 172.16.1.1; do ping -c 1 $i; done
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.


^C
--- 192.168.10.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 1903ms


PING 192.168.20.1 (192.168.20.1) 56(84) bytes of data.
^C
--- 192.168.20.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 975ms


PING 192.168.30.1 (192.168.30.1) 56(84) bytes of data.
^C
--- 192.168.30.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 2687ms


PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=63 time=0.188 ms


--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.188/0.188/0.188/0.000 ms
From the same CT I can not ping any PfSense interface..

Simply looking for the CT's to route.

Not sure where I broke something..

Suggestions or Questions are welcomed.

Thank you for taking the time to read this.
 
[SOLVED] Re: vlan (not) routing, possible self inflicted issue..

so long story short..
https://pve.proxmox.com/wiki/Vlans

Incomplete.

Not sure if what follows is the most efficient/minimal config; but it works.

Code:
root@proxmox:~# egrep -v \# /etc/network/interfaces

auto lo
iface lo inet loopback


iface eth0 inet manual


auto vmbr0
iface vmbr0 inet static
        address  192.168.10.99
        netmask  255.255.255.0
        gateway  192.168.10.1
        bridge_ports eth0.10
        bridge_stp off
        bridge_fd 0


auto vmbr1
iface vmbr1 inet manual
        bridge_ports eth0.1
        bridge_stp off
        bridge_fd 0


auto vmbr20
iface vmbr20 inet manual
        bridge_ports eth0.20
        bridge_stp off
        bridge_fd 0


auto vmbr30
iface vmbr30 inet manual
        bridge_ports eth0.30
        bridge_stp off
        bridge_fd 0

vzctl set 142 --netif_add eth0 --save
vzctl set 2042 --netif_add eth0 --save
vzctl set 3042 --netif_add eth0 --save

Apparently the CT needs to be shutdown.

Remove the IP address from each CT in the Gui/WebIf

Add a veth device to each CT, named eth0 attached to the bridge/vlan interface. (vlan1 == vmbr1, vlan20 == vmbr20..)

Then start the CT and configure the networking for whatever the OS is, CentOS/Debian/Arch/etc..

Code:
[root@vm142 /]# cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
TYPE="Ethernet"
ONBOOT=yes
BOOTPROTO=static
IPADDR=172.16.1.42
NETMASK=255.255.255.0

Code:
[root@vm142 /]# cat /etc/sysconfig/network

NETWORKING="yes"
HOSTNAME="vm142.tcentral.lan"
GATEWAYDEV="eth0"
NETWORKING_IPV6="no"
IPV6_DEFAULTDEV="eth0"
GATEWAY=172.16.1.1

Code:
[root@vm2042 /]# cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
TYPE="Ethernet"
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.20.42
NETMASK=255.255.255.0

Code:
[root@vm2042 /]# cat /etc/sysconfig/network

NETWORKING="yes"
GATEWAYDEV="eth0"
NETWORKING_IPV6="yes"
IPV6_DEFAULTDEV="eth0"
HOSTNAME="vm2042.tcentral.lan"
GATEWAY=192.168.20.1

Code:
[root@vm2042 /]# ping -c 1 10.20.0.1

PING 10.20.0.1 (10.20.0.1) 56(84) bytes of data.
64 bytes from 10.20.0.1: icmp_seq=1 ttl=254 time=0.891 ms


--- 10.20.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.891/0.891/0.891/0.000 ms

Code:
[root@vm142 /]# ping -c1 10.20.0.1

PING 10.20.0.1 (10.20.0.1) 56(84) bytes of data.
64 bytes from 10.20.0.1: icmp_seq=1 ttl=254 time=0.861 ms


--- 10.20.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.861/0.861/0.861/0.000 ms
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!