[SOLVED] Network routing in cluster environment

[2cadz]

Hi to all ,
I have a cluster proxmox 4.1 with 3 nodes configured and functional. On one of the nodes I have a LXC container using a private routed network. The container can access the Internet and everything works fine.
To access to this container from the other nodes I added the necessary routes.
However, I have a problem when moving the container on one of the other nodes: the routes are not updated.
Is there a way to update the routes automaticaly or possibility to execute a script that update the routes during migration from one node to another ?

Best regard.

 

[LnxBil]

Why do you need to route on the pve hosts itself?

I created cluster wide vLANs for this purpose and added the hosts there. I created also a router for each vLAN, which are virtual machines itself. Therefore the route is always the same because the router has always the same IP.

 

[2cadz]

Hi LnxBil,

Thank for your answer.
Can you put your network config here, because i dont understand why vLANs solve the problem ?

Regard

 

[LnxBil]

Because you have a dedicated, virtualized router with a static IP. Therefore you can migrate the VMs as you like and routing still works perfectly! The router endpoint is fixed therefore no change is required on VM migration.

The VLAN does only ensure the private routed network is secured, you do not need it for a working environment, only for a secure one.

 

[2cadz]

I think I expressed myself badly. I have a container (CT100) at node1. This container is on a private network. From node2 and node3 (and containers on these nodes) I can reach the container through specific routes. If I move CT100 on node2, I'm unable to reach it from node1 and 3 because the routes are false.
Hence my question about the possibility update the routes during migration.

 

[LnxBil]

That is exactly what I understand in the first place and I still do not get why you need a route in the first place. Could you please show how this route looks like?

I have multiple nested network environments and do not need to set one route on any of my pve nodes. Everything working perfectly with vLANs. This setup is common in any virtualization environment I ever saw.

 

[2cadz]

Below my network configuration :

Node 1
vmbr2 is the interface used for the private network.

auto vmbr2
iface vmbr2 inet static
        address  172.16.61.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '172.16.61.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '172.16.61.0/24' -o vmbr0 -j MASQUERADE

CT100

auto eth0
iface eth0 inet static
        address 172.16.61.101
        netmask 255.255.255.255
        post-up ip route add 172.16.61.1 dev eth0
        post-up ip route add default via 172.16.61.1
        pre-down ip route del default via 172.16.61.1
        pre-down ip route del 172.16.61.1 dev eth0

Node 2
Route to reach CT100.
vmbr1 is the interface used for the cluster communication.

172.16.61.100 via 172.16.100.1 dev vmbr1

 

[LnxBil]

I still think that a dedicated router VM will solve your problem. The VM will be masquerading and you enter the route to the this network directly on your default gateway, so you don't have to change route on nodes. Everything is handled automatically via your default gateway.

 

[2cadz]

sorry for the delay, but i try some config and i come back with the result.

Regard

 

[2cadz]

After some experimentation, below you find how I solved my problem:
I added in the configuration of the lxc container the following lines:

lxc.network.script.up: /etc/lxc-manage-routes.sh
lxc.network.script.down: /etc/lxc-manage-routes.sh

the latter adds or deletes a route to the container when it start, and then I have used 'Quagga' (on each supervisor) to transfer these new routes to other supervisors.

If anyone is interested in the complete config, it can leave a post here.

Best regard.

 

[dbayer]

Hi LnxBil,

Can you give specifics on your network setup? Like what are you using for the router? What are your interfaces file listing? I'm just going down the road of setting up separate subnets and vlans for our environment. It seems like having proxmox do the layer 3 / routing makes more sense then getting a separate router or layer 3 switch.

Thanks,
Daniel

 

[LnxBil]

Hi Daniel,

I have a router VM (KVM, because of HA) which has two interfaces in Proxmox, one for each VLAN (or normal-LAN and VLAN in your case) and the IPs are in different subnets. The VLAN settings are applied on VM level on Proxmox, so there is no vlan configuration inside the VM. This router for the to-be-routed-network has always .1 as IP in the private network and will act as a router (or firewall if needed). It has obviously IP forward enabled (and if applicable masquerades all private-network-packets to the rest/internet). For direct routing you set a route for the private network on each PVE host and use the router VM IP as Gateway. This is the same for all nodes and for all time. Then you can move around your LXC container in your clustered environment and everything works as expected. I hope it helps.

Best,
LnxBil

 

[dbayer]

Hi LnxBil,

Thanks for getting back to me. Just a couple more questions.

What software do you use for your VM router?

For VM to VM networking on the same node, do you still have it go through the router? If you do, does that keep the network traffic from leaving the Node? What I mean is, is it efficient?

Thanks,
Daniel

 

[LnxBil]

I use Debian for everything. Traffic has to pass the router, but this is very efficient because it uses memory (if both are on the same host) but uses network if they are distributed across all nodes. The later one should be the default - at least for me it is.

 

[dbayer]

Perfect! That is exactly what I wanted to know.

Thank you for your help!