Asymmetric routing with Promox 5 / LXC

M

michaelj

Renowned Member
Jun 30, 2016
57
0
71
37
Hi Proxmox community,

I'm facing an asymmetric routing issue.

Each host have public and private network.

Public network : vmbr0
Private network : vmbr2

Postrouting rule on HOST A :
iptables -t nat -A PREROUTING -i vmbr0 -s xxxx -p tcp --dport 10636 -j DNAT --to 172.25.2.19:636

172.25.2.19 = virtual ip master

HOST A / VM A with master IP HOST B / VM B slave

virtual ip : 172.25.2.19 ip : 172.25.2.2
ip : 172.25.2.1

When i switch the Master IP on VM B, i can't connect to 636.

Under Proxmox / Openvz, i was using "VE_ROUTE_SRC_DEV=vmbr2" parameter in vz.conf in order to work.

VE_ROUTE_SRC_DEV="device"

This parameter specifies the network device name which IP address will be used as the source IP. This is helpful in case more than one network interface is configured on HN and there is a need to specify the source IP address. Default is the first device in the network device list.

Do you know how to configure this under Proxmox 5 / LXC ?


Regards.
 
michaelj said:
I'm facing an asymmetric routing issue.

Each host have public and private network.

Public network : vmbr0
Private network : vmbr2

Postrouting rule on HOST A :
iptables -t nat -A PREROUTING -i vmbr0 -s xxxx -p tcp --dport 10636 -j DNAT --to 172.25.2.19:636
Click to expand...
That's prerouting rule of course
michaelj said:
172.25.2.19 = virtual ip master

HOST A / VM A with master IP HOST B / VM B slave

virtual ip : 172.25.2.19 ip : 172.25.2.2
ip : 172.25.2.1

When i switch the Master IP on VM B, i can't connect to 636.
Click to expand...
AFAIU the used terminology ("Master", "Slave") means in Proxmox terms:
* There is a container with IP 172.25.2.19 which may run alternatively at Host A or B
* Port forwarding is always performed from Host A's public IP to the container (wherever it currently runs)
* When running at A the route back uses 172.25.2.1 (i.e. A) as gateway - everything OK
* When running at B the route back uses 172.25.2.2 (i.e. B) as gateway - asymmetric routing, some applications do not allow this.

A solution would be to use always 172.25.2.1 as gateway, but there may be some reasons not to do that - therefore the request for source based routing.

michaelj said:
Under Proxmox / Openvz, i was using "VE_ROUTE_SRC_DEV=vmbr2" parameter in vz.conf in order to work.

VE_ROUTE_SRC_DEV="device"

This parameter specifies the network device name which IP address will be used as the source IP. This is helpful in case more than one network interface is configured on HN and there is a need to specify the source IP address. Default is the first device in the network device list.

Do you know how to configure this under Proxmox 5 / LXC ?
Click to expand...

OpenVZ was a special solution which bypassed some basic Linux rules (by using venet0 device). LXC is more like a normal Linux and therefore the usual Linux methods are requested for source based routing. The solution is to use more routing tables, details you can find here: https://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/
 
Hi Richard,

Thanks for your feedback.

You right, it's PREROUTING rule, my bad.

* There is a container with IP 172.25.2.19 which may run alternatively at Host A or B

Yes when my container A on HOST A is master, VM A have two ips 172.25.2.1 & 172.25.2.19.

Then when i switch to the VM B on HOST B, VM B have ip : 172.25.2.2 & 172.25.2.19.

For information, i want this configuration because i have a DNS CNAME entry in a lot of configuration, and this CNAME is redirecting to the HOST A, that's why i would like to use ip 172.25.2.19 in my prerouting rule.


HOST A route :

default 51.255.x.x 0.0.0.0 UG 0 0 0 vmbr0
51.255.x.x 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vmbr2
224.0.0.0 0.0.0.0 240.0.0.0 U 0 0 0 vmbr2

HOST B route :

default 149.202.x.x 0.0.0.0 UG 0 0 0 vmbr0
149.202.x.x 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vmbr2
224.0.0.0 0.0.0.0 240.0.0.0 U 0 0 0 vmbr2


VM A route :

default 172.25.0.11 (private ip host A) 0.0.0.0 UG 0 0 0 eth0
172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

VM B route :

default 172.25.0.17 (private ip host B) 0.0.0.0 UG 0 0 0 eth0
172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

Can you help me to configure my routing table if it's possible please ?

Regards.
 
In order to be precise and avoid misunderstanding I suggest you post:
- /etc/network/interfaces from the hosts
- /etc/pve/lxc/*.conf for both containers
 
HOST A

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0


# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
address 51.255.x.x
netmask 255.255.255.0
network 51.255.x.x
broadcast 51.255.x.x
gateway 51.255.x.x
bridge_ports eth2
bridge_stp off
bridge_fd 0

iface vmbr0 inet6 static
address 2001:41d0:1008:0dd6::
netmask 64
post-up /sbin/ip -f inet6 route add 2001:41d0:1008:0dff:ff:ff:ff:ff dev vmbr0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:1008:0dff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del default via 2001:41d0:1008:0dff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del 2001:41d0:1008:0dff:ff:ff:ff:ff dev vmbr0



# bridge vrack 1.5
auto vmbr2
iface vmbr2 inet static
address 172.25.0.11
netmask 255.255.0.0
broadcast 172.25.255.255
bridge_ports rename5
bridge_stp off
bridge_fd 0
post-up /sbin/route add -net 224.0.0.0/4 vmbr2
pre-down /sbin/route del -net 224.0.0.0/4 vmbr2



HOST B

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0


# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
address 149.202.x.x
netmask 255.255.255.0
network 149.202.x.x
broadcast 149.202.x.x
gateway 149.202.x.x
bridge_ports eth2
bridge_stp off
bridge_fd 0

iface vmbr0 inet6 static
address 2001:41d0:1000:207b::
netmask 64
post-up /sbin/ip -f inet6 route add 2001:41d0:1000:20ff:ff:ff:ff:ff dev vmbr0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:1000:20ff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del default via 2001:41d0:1000:20ff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del 2001:41d0:1000:20ff:ff:ff:ff:ff dev vmbr0

# bridge vrack 1.5
auto vmbr2
iface vmbr2 inet static
address 172.25.0.17
netmask 255.255.0.0
broadcast 172.25.255.255
bridge_ports eth3
bridge_stp off
bridge_fd 0
post-up /sbin/route add -net 224.0.0.0/4 vmbr2
pre-down /sbin/route del -net 224.0.0.0/4 vmbr2


VM A on HOST A :

arch: amd64
cores: 1
hostname: axxxx
memory: 8192
mp0: /apps/scripts,mp=/apps/scripts,ro=1
mp1: /share,mp=/share
net0: name=eth0,bridge=vmbr2,gw=172.25.0.11,hwaddr=C6:48:F7:E1:41:65,ip=172.25.2.17/16,type=veth
onboot: 1
ostype: debian
rootfs: zfs-storage:subvol-204-disk-0,size=50G
searchdomain: xxx
startup: order=3
swap: 1024


VM B on HOST B :

arch: amd64
cores: 2
cpulimit: 0.5
hostname: xxxx
memory: 8192
mp0: /apps/scripts,mp=/apps/scripts,ro=1
mp1: /share,mp=/share
net0: name=eth0,bridge=vmbr2,gw=172.25.0.17,hwaddr=AA:A3:DC:EA:13:0A,ip=172.25.2.18/16,type=veth
onboot: 1
ostype: debian
rootfs: zfs-storage:subvol-205-disk-0,size=50G
searchdomain: xxxxx
startup: order=4
swap: 1024
 
michaelj said:
HOST A

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0


# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
address 51.255.x.x
netmask 255.255.255.0
network 51.255.x.x
broadcast 51.255.x.x
gateway 51.255.x.x
bridge_ports eth2
bridge_stp off
bridge_fd 0

iface vmbr0 inet6 static
address 2001:41d0:1008:0dd6::
netmask 64
post-up /sbin/ip -f inet6 route add 2001:41d0:1008:0dff:ff:ff:ff:ff dev vmbr0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:1008:0dff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del default via 2001:41d0:1008:0dff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del 2001:41d0:1008:0dff:ff:ff:ff:ff dev vmbr0



# bridge vrack 1.5
auto vmbr2
iface vmbr2 inet static
address 172.25.0.11
netmask 255.255.0.0
broadcast 172.25.255.255
bridge_ports rename5
bridge_stp off
bridge_fd 0
post-up /sbin/route add -net 224.0.0.0/4 vmbr2
pre-down /sbin/route del -net 224.0.0.0/4 vmbr2



HOST B

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# for Routing
auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0


# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
address 149.202.x.x
netmask 255.255.255.0
network 149.202.x.x
broadcast 149.202.x.x
gateway 149.202.x.x
bridge_ports eth2
bridge_stp off
bridge_fd 0

iface vmbr0 inet6 static
address 2001:41d0:1000:207b::
netmask 64
post-up /sbin/ip -f inet6 route add 2001:41d0:1000:20ff:ff:ff:ff:ff dev vmbr0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:1000:20ff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del default via 2001:41d0:1000:20ff:ff:ff:ff:ff
pre-down /sbin/ip -f inet6 route del 2001:41d0:1000:20ff:ff:ff:ff:ff dev vmbr0

# bridge vrack 1.5
auto vmbr2
iface vmbr2 inet static
address 172.25.0.17
netmask 255.255.0.0
broadcast 172.25.255.255
bridge_ports eth3
bridge_stp off
bridge_fd 0
post-up /sbin/route add -net 224.0.0.0/4 vmbr2
pre-down /sbin/route del -net 224.0.0.0/4 vmbr2


VM A on HOST A :

arch: amd64
cores: 1
hostname: axxxx
memory: 8192
mp0: /apps/scripts,mp=/apps/scripts,ro=1
mp1: /share,mp=/share
net0: name=eth0,bridge=vmbr2,gw=172.25.0.11,hwaddr=C6:48:F7:E1:41:65,ip=172.25.2.17/16,type=veth
onboot: 1
ostype: debian
rootfs: zfs-storage:subvol-204-disk-0,size=50G
searchdomain: xxx
startup: order=3
swap: 1024


VM B on HOST B :

arch: amd64
cores: 2
cpulimit: 0.5
hostname: xxxx
memory: 8192
mp0: /apps/scripts,mp=/apps/scripts,ro=1
mp1: /share,mp=/share
net0: name=eth0,bridge=vmbr2,gw=172.25.0.17,hwaddr=AA:A3:DC:EA:13:0A,ip=172.25.2.18/16,type=veth
onboot: 1
ostype: debian
rootfs: zfs-storage:subvol-205-disk-0,size=50G
searchdomain: xxxxx
startup: order=4
swap: 1024
Click to expand...


As already mentioned - the easiest way would be to as gateway 172.25.0.11 . The only one reason to not do that would be if there are other applications too at VM B which sould use for outgoing connections the other public IP. In that case you need at first a second IP address in VM B, let's say 172.25.2.118. Then host A port 10636 should be forwarded to 172.25.2.118:636


For proper route back add in the container VM B the a source dependent default route for 172.25.2.118 to 172.25.0.11 e.g. as follows:

Code: 
echo 200 custom >> /etc/iproute2/rt_tables
ip rule add from 172.25.2.118 lookup custom
ip route add default via 172.25.0.11 dev eth0 table custom

For more details again the reference to https://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom