Problem with Proxmox 3.0 with VLAN and public & private IP routing

[MartinX]

Hello,

I've reached a weird situation where we have multiple servers that run different versions of Proxmox - Proxmox 2.3 and 3.0 for testing. They have the same kind of configuration but networking doesn't behave quite the same way unfortunately. For background information, the machines are hosted with OVH (if that's any help).

For this example, the setup includes two Host Machines (HM #1 and HM #2) connected to the VLAN + Public IP range routed through VLAN + Private IP range routed through VLAN.

When I reboot the HM #1 and start OpenVZ containers then the OpenVZ container can ping Internet, but can't ping HM #1, HM #2 or any other container inside VLAN Public IP (Private IP works). The OpenVZ container can be reached from the Internet and HM #1, but can't be pinged from HM #2 or any other container inside the same VLAN.

After few hours, the same OpenVZ container no longer replies to ping from Internet or anywhere else.

Based on what I know this far, it seems like a the router forgets the OpenVZ container and for some reason the HM #1 doesn't properly deliver the ARP requests to the network any more. Why it happens is quite unknown to me at this point.

Hopefully I'm not the only one facing this problem. Anything that helps me to get to the right way is much appreciated.

This is /etc/network/interface

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
        bridge_ports dummy0
        bridge_stp off
        bridge_fd 0


# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
        address 5.135.XXX.XXX
        netmask 255.255.255.0
        network 5.135.XXX.0
        broadcast 5.135.XXX.XXX
        gateway 5.135.XXX.XXX
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto eth0.ZZZZ:0
iface eth0.ZZZZ:0 inet static
        address 192.168.0.10
        netmask 255.255.0.0
        up /sbin/ip route add 192.168.0.0/16 via 192.168.255.254 dev eth0.ZZZZ:0
        post-down /sbin/ip route del 192.168.0.0/16 via 192.168.255.254 dev eth0.ZZZZ:0
        pre-up iptables -t nat -A POSTROUTING -s 192.168.0.0/16 ! -d 192.168.0.0/16 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s 192.168.0.0/16 ! -d 192.168.0.0/16 -j MASQUERADE

auto eth0.ZZZZ:1
iface eth0.ZZZZ:1 inet static
        address 5.39.YYY.YYY
        network 5.39.YYY.YYY
        broadcast 5.39.YYY.YYY
        netmask 255.255.255.192
        up /sbin/ip route add default via 5.39.YYY.YYY dev eth0.ZZZZ:1 table 125
        up /sbin/ip rule add from 5.39.YYY.YYY/26 table 125
        post-down /sbin/ip route del default via 5.39.YYY.YYY dev eth0.ZZZZ:1 table 125
        post-down /sbin/ip rule del from 5.39.YYY.YYY/26 table 125

This is /etc/sysctl.conf

net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.ip_forward=1

net.ipv4.conf.default.proxy_arp = 1
net.ipv4.conf.all.proxy_arp = 1
net.ipv4.conf.vmbr0.proxy_arp = 1

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.vmbr0.rp_filter = 0

kernel.ve_allow_kthreads = 1

This is /etc/sysctl.d/vzctl.conf

# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled

net.ipv4.ip_forward = 1
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.proxy_arp = 1

# Enables source route verification
net.ipv4.conf.all.rp_filter = 0

# Enables the magic-sysrq key
kernel.sysrq = 1

# TCP Explict Congestion Notification
#net.ipv4.tcp_ecn = 0

# we do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.default.rp_filter =0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0

If there's anything else that I could provide and that could help with this riddle then please let me know.

Thank you very much in advance!

 

[mir]

Are you sure you mean VLAN? This is not the proper way to create a VLAN 'auto eth0.ZZZZ:0'. What this exactly is doing is creating an alias 'eth0:0' for tagged for VLAN ZZZZ. I cannot entirely say that this is causing undefined behavior but I have never seen a setup like this before. Why do you not simply create more bridges instead? A bridge can also be tagged.

 

[MartinX]

The previously uploaded configuration was based on mainly on this post: http://www.chemeo.com/doc/technology/openvz
There are also these articles that I've tried to follow: http://help.ovh.ie/Vrack & http://guides.ovh.com/vrackproxmox

The first post was followed for Proxmox 2.3 installation and somehow it worked perfectly there.

I'll try to read a bit further on tagging bridges and see if this could be a solution. Thank you.

 

[mir]

This is somewhat what you are looking fore:

auto eth0
iface eth0 inet manual

# VLAN0 is default so no need to tag this
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
gateway 192.168.0.1
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr10
iface vmbr10 inet static
address 192.168.1.1
bridge_ports eth0.10
bridge_stp off
bridge_fd 0

auto vmbr20
iface vmbr20 inet static
address 192.168.2.1
bridge_ports eth0.20
bridge_stp off
bridge_fd 0

 

[MartinX]

Thank you. Unfortunately it looks like the configuration in previous post didn't help on my case.

It seems like vmbr1 is needed as well by OVH for internal routing.

 

[mir]

The configuration shown was not support to be a copy/past for your specific requirements. It was merely an example for you to use for a starting point. You should of course craft your own configuration tailoring you specific needs.