[SOLVED] Routing logic question/help

shocker

Renowned Member
Jun 21, 2016
26
0
66
38
Hello,
I'm trying to migrate an old router from a physical machine to a KVM on a proxmox. There will be two proxmox machines in a cluster-based for high availability.

For the moment I have strange behaviour and I think it's something from the logic that I'm missing from.

I have lots of VLAN that there were added to the proxmox machine on an LACP port, the VLAN's are working great and the virtual machine can route the traffic. The only problem is that the traffic cannot be routed back to the host machine (proxmox) that is hosting the VM and I cannot access it anymore.

Setup:
LACP for Proxmox1 - VLAN 502 for management and others (trunk mode)
LACP for Proxmox2 - VLAN 502 for management and others (trunk mode)
The virtual machine is on Proxmox 2 and the network configuration on the host machine is:
auto lo iface lo inet loopback iface eno1 inet manual iface eno2 inet manual iface eno3 inet manual iface eno4 inet manual auto bond0 iface bond0 inet manual slaves eno1 eno2 bond_miimon 100 bond_mode 802.3ad mtu 9000 auto bond0.502 iface bond0.502 inet manual vlan_raw_device bond0 mtu 9000 auto vmbr0 iface vmbr0 inet static address proxmox2_ip/27 gateway the_gateway bridge_ports bond0.502 bridge_stp off bridge-vlan-aware yes bridge_fd 0 mtu 9000 auto vmbr1 iface vmbr1 inet manual bridge_ports bond0 bridge_stp off bridge_fd 0 mtu 9000

The interface assigned to the router VM is vmbr1.
On the switch, I can see all the 3 mac addresses for vlan502 (proxmox1, proxmox2, router machine).
Nothing is reachable on vlan502, I cannot reach proxmox1, proxmox2. Everything else works fine.
Tcpdump on vmbr0 shows nothing except the arp requests and I can see the arp's from proxmox1-proxmox2, nothing from the VM is visible.
Disabling the VM network interface and activating the old router with vlan502 restores the issue.

Definitely, it's something that the VM from vmbr1 is not communicating back with the network on vlan502 but I cannot understand the logic as also proxmox1 is not responding back.

Is there something that I need to add to vmbr1? Should I add to vmbr0 bridge_ports vmbr1 (not sure if bridge in bridge will work :) ).

Thanks!
 
Thanks for the feedback! Tried that, not working. The initial config was without vlan aware option.
 
did you restart networking after changing the vlan aware option? is the VM the gateway IP for vlan 502?
 
did you restart networking after changing the vlan aware option? is the VM the gateway IP for vlan 502?
Yes, I have restarted the network interface after changing the options.
The VM is the gw for the vmbr0 vlan502.
 
From vmbr1 I have other VLANs that work ok. The problem is only with vlan502 that is assigned to the proxmox host for management.
Also, the mac from VM, mac from the host machine is visible on the switch, no issue on L2. I think I'm missing something from the logic !? :)
 
Updated configuration

auto lo iface lo inet loopback auto eno1 iface eno1 inet manual mtu 9000 auto eno2 iface eno2 inet manual mtu 9000 iface eno3 inet manual iface eno4 inet manual auto bond0 iface bond0 inet manual bond-slaves eno1 eno2 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 mtu 9000 auto bond0.502 iface bond0.502 inet manual mtu 9000 vlan_raw_device bond0 auto vmbr0 iface vmbr0 inet static address ip.36/27 gateway ip.33 bridge-ports bond0.502 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 502 mtu 9000 auto vmbr1 iface vmbr1 inet manual bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 mtu 9000

I have a test machine now on VMBR1 with two VLAN's:
VLAN100 - IP assigned, fully accessible
VLAN502 assigned an IP address from /27 subnet with gw to the physical router. Not working as well.

auto lo iface lo inet loopback auto eth0 iface eth0 inet manual mtu 9000 #Vlan Domestic auto eth0.100 iface eth0.100 inet static vlan_raw_device eth0 address ip.4/27 gateway ip.1 mtu 9000 auto eth0.502 iface eth0.502 inet static vlan_raw_device eth0 alias Domestic traffic address ip.37/27 mtu 9000

eth0.100 works fine
from eth0.502 I cannot ping the IP assigned on the host on vmbr0

~# ping -I eth0.502 -c 2 ip.36 PING ip.36 (ip.36) from ip.37 eth0.502: 56(84) bytes of data. --- ip.36 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1026ms

From the .100 VLAN via the physical router, I can reach ip.36
~# ping -I eth0.100 -c 2 ip.36 PING ip.36 (ip.36) from ip.4 eth0.100: 56(84) bytes of data. 64 bytes from ip.36: icmp_seq=1 ttl=63 time=0.443 ms --- ip.36 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.443/0.443/0.443/0.000 ms
 
I have removed from the networking bond0.502 and vmbr0. Now from the VM I can ping and see the traffic from 502 VLAN.
Adding again bond0.502 on the host machine breaks the traffic. Seems that if I'm tagging the VLAN locally it will not send it over via the bridge? :rolleyes:

Did anyone play with this scenario before?

1615841158246.png

From VM:

~# ping -I eth0.502 ip.33 PING ip.33 (ip.33) from ip.37 eth0.502: 56(84) bytes of data. 64 bytes from ip.33: icmp_seq=1 ttl=64 time=0.261 ms 64 bytes from ip.33: icmp_seq=2 ttl=64 time=0.263 ms --- ip.33 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1018ms rtt min/avg/max/mdev = 0.261/0.262/0.263/0.001 ms


Adding on the host machine:
1615841481023.png
and applying the configuration ifup -a stops the traffic on vlan 502 on the VM.
 
Last edited:
Managed to make it work :)

instead of adding the VLAN on the bond interface I have added it at the vmbr0 interface and created the vmbr0.502. Now everything is working as intended :)

Not sure if this is a feature or a bug, but in this setup is working, maybe it will help others in the future.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!