container uses internal proxmox host ip to route external

seechiller · Jan 5, 2014

Hello

I've added an internal backup LAN to my proxmox hosts. Today I noticed that some console output are very very slow if the backup LAN is present (NIC eth1 UP) on the proxmox host. So I did some tests with traceroute and found out that the traffic flows over the internal backup LAN IP if the sec. NIC is up.

I've no idea why, containers works but I would like to understand why the traffic hit the hosts internal IP.

Proxmox Host IP Setup:
eth0 - 176.xx.xx.0/24 = own public class C for VM's - routed over Cisco HSRP
eth1 - 192.168.80.0/24 = internal Backup LAN, no www access or router present, dedicated Cisco GB Switch with own VLAN on it

Proxmox Hosts /etc/network/interfaces:

Code:

# network interface settings

auto lo vmbr0 eth1

iface lo inet loopback

iface vmbr0 inet static
        address  176.xx.xx.23
        netmask  255.255.255.0
        gateway  176.xx.xx.3
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

iface eth0 inet manual

iface eth1 inet static
        address  192.168.80.23
        netmask  255.255.255.0

iface eth2 inet manual

iface eth3 inet manual

Routing on Proxmox Host:

Code:

root@pm3:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
176.xx.xx.105  0.0.0.0         255.255.255.255 UH    0      0        0 venet0
176.xx.xx.188  0.0.0.0         255.255.255.255 UH    0      0        0 venet0
....
....
176.xx.xx.100  0.0.0.0         255.255.255.255 UH    0      0        0 venet0
192.168.80.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
176.xx.xx.0    0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
0.0.0.0         176.xx.xx.3    0.0.0.0         UG    0      0        0 vmbr0

Note: eth1 is NOT the default gw....

ifconfig:

Code:

root@pm3:~# ifconfig
eth0      Link encap:Ethernet  HWaddr e4:1f:13:b3:2c:dc
          inet6 addr: fe80::e61f:13ff:feb3:2cdc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22560305 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17482196 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19646761034 (18.2 GiB)  TX bytes:5943876483 (5.5 GiB)

eth1      Link encap:Ethernet  HWaddr e4:1f:13:b3:2c:de
          inet addr:192.168.80.23  Bcast:192.168.80.255  Mask:255.255.255.0
          inet6 addr: fe80::e61f:13ff:feb3:2cde/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12361 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8638 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1281000 (1.2 MiB)  TX bytes:1808034 (1.7 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2315452 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2315452 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1348195645 (1.2 GiB)  TX bytes:1348195645 (1.2 GiB)

tap1396i0 Link encap:Ethernet  HWaddr fe:cc:39:0e:5b:5f
          inet6 addr: fe80::fccc:39ff:fe0e:5b5f/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:79514 errors:0 dropped:0 overruns:0 frame:0
          TX packets:498223 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:12709895 (12.1 MiB)  TX bytes:43213507 (41.2 MiB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet6 addr: fe80::1/128 Scope:Link
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:24102780 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28451814 errors:0 dropped:13 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:9261256001 (8.6 GiB)  TX bytes:22521191708 (20.9 GiB)

vmbr0     Link encap:Ethernet  HWaddr e4:1f:13:b3:2c:dc
          inet addr:176.xx.xx.23  Bcast:176.xx.xx.255  Mask:255.255.255.0
          inet6 addr: fe80::e61f:13ff:feb3:2cdc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21955668 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17320461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:19201405289 (17.8 GiB)  TX bytes:5846372083 (5.4 GiB)

Traceroute on the Proxmox Host to google (eth0 & eth1 up AND with eth1 down):

Code:

root@pm3:~# traceroute google.ch
traceroute to google.ch (173.194.116.56), 30 hops max, 60 byte packets
 1  host-xx-3.xxxxx.ch (176.xx.xx.3)  0.227 ms  0.349 ms  0.523 ms
 2  swissix.google.com (91.206.52.74)  1.408 ms  1.549 ms  1.521 ms
 3  209.85.244.125 (209.85.244.125)  1.550 ms  1.951 ms  1.746 ms
 4  zrh04s07-in-f24.1e100.net (173.194.116.56)  1.103 ms  1.110 ms  1.113 ms

root@pm3:~# ifdown eth1
root@pm3:~# traceroute google.ch
traceroute to google.ch (173.194.35.55), 30 hops max, 60 byte packets
 1  host-xx-3.xxxxx.ch (176.xx.xx.3)  0.238 ms  0.347 ms  0.460 ms
 2  swissix.google.com (91.206.52.74)  1.269 ms  1.282 ms  1.268 ms
 3  72.14.233.44 (72.14.233.44)  4.901 ms  4.997 ms  4.996 ms
 4  209.85.241.67 (209.85.241.67)  6.325 ms  5.923 ms  6.110 ms
 5  mil01s17-in-f23.1e100.net (173.194.35.55)  4.870 ms  4.871 ms  4.880 ms

root@pm3:~# ifup eth1
root@pm3:~# traceroute google.ch
traceroute to google.ch (173.194.116.55), 30 hops max, 60 byte packets
 1  host-xx-3.xxxxx.ch (176.xx.xx.3)  0.310 ms  1.268 ms  1.409 ms
 2  swissix.google.com (91.206.52.74)  1.235 ms  27.144 ms  27.149 ms
 3  209.85.244.125 (209.85.244.125)  1.739 ms  1.930 ms  2.092 ms
 4  zrh04s07-in-f23.1e100.net (173.194.116.55)  1.044 ms  0.962 ms  0.931 ms

Strange is that if eth1 is up traceroute needs 4 hops to google, if it is down it needs 5 hops, no idea why

----------------

Now the "problem": Example with one container VM

Traceroute on VM with eth1 DOWN on the proxmox Host:

Code:

[root@setup /]# traceroute google.ch
traceroute to google.ch (173.194.35.56), 30 hops max, 60 byte packets
 1  host-xxx-23.xxxxxx.ch (176.xx.xx.23)  0.022 ms  0.007 ms  0.006 ms
 2  swissix.google.com (91.206.52.74)  1.001 ms  1.001 ms  1.012 ms
 3  72.14.233.44 (72.14.233.44)  4.749 ms  4.965 ms  4.978 ms
 4  209.85.241.67 (209.85.241.67)  9.756 ms  10.189 ms  9.972 ms
 5  mil01s17-in-f24.1e100.net (173.194.35.56)  4.775 ms  4.821 ms  5.160 ms

Traceroute on VM with eth1 UP on the proxmox Host (now uses internal IP):

Code:

[root@setup /]# traceroute google.ch
traceroute to google.ch (173.194.35.63), 30 hops max, 60 byte packets
 1  192.168.80.23 (192.168.80.23)  0.024 ms  0.007 ms  0.006 ms
 2  swissix.google.com (91.206.52.74)  1.160 ms  1.162 ms  1.157 ms
 3  72.14.233.44 (72.14.233.44)  5.197 ms  5.043 ms  5.023 ms
 4  209.85.241.67 (209.85.241.67)  5.050 ms  5.424 ms  5.754 ms
 5  mil01s17-in-f31.1e100.net (173.194.35.63)  4.812 ms  4.916 ms  4.913 ms

ifconfig & route on VM

Code:

[root@setup /]# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3177 (3.1 KiB)  TX bytes:3177 (3.1 KiB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:127.0.0.1  P-t-P:127.0.0.1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:14523 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7510 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:21288779 (20.3 MiB)  TX bytes:399363 (390.0 KiB)

venet0:0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:176.xx.xx.250  P-t-P:176.xx.xx.250  Bcast:176.xx.xx.250  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1


[root@setup /]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 venet0
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 venet0

I've absolutely no idea why the vm traffic goes over the internal IP of the proxmox Host out, any idea?

BTW: I'll add later NIC bonding over eth2 & eth3, so I just use 1 NIC (eth0) right now

Thanks!

Jorge · Mar 6, 2014

I have got here because a similar problem found in proxmox both 2 and 3. I took me a long time but at the end found a solution that worked as a miracle:

http://blog.droidzone.in/2013/10/20/setting-up-a-dedicated-server-on-online-net/

I would like to share this here with those of you who are stucked at the same point. Please just check this before starting to mess with interfaces and the other network stuff.

I hope it helps.

Search

Search

container uses internal proxmox host ip to route external

seechiller

Renowned Member

Jorge

New Member

We value your privacy