Networking issues with LXC on Proxmox 4.4 running on VMware ESXi

[bonki]

We recently deployed Proxmox 4.4 as a (single) VM in a vSphere cluster and have serious issues with traffic between LXC containers and virtual machines directly running on ESXi which are residing on the same VLAN (see the picture attached).

Pinging from ct05 (10.251.85.16) -> ct04 (10.251.85.15), two LXC containers on the same VLAN/subnet on the same Proxmox host:

--- [snip] ---

[root@ct05 ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
10.251.85.7 (incomplete) eth0
10.251.85.15 (incomplete) eth0
10.251.85.2 (incomplete) eth0
10.251.85.1 (incomplete) eth0
[root@ct05 ~]# hostname -i
10.251.85.16
[root@ct05 ~]# ping -c 1 10.251.85.15
PING 10.251.85.15 (10.251.85.15) 56(84) bytes of data.
64 bytes from 10.251.85.15: icmp_seq=1 ttl=64 time=0.114 ms

--- 10.251.85.15 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.114/0.114/0.114/0.000 ms

[root@ct05 ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
10.251.85.7 (incomplete) eth0
10.251.85.15 ether 62:33:ed:c5:7f:bc C eth0
10.251.85.2 (incomplete) eth0
10.251.85.1 (incomplete) eth0

--- [snip] ---

root@proxmox:~# tcpdump -nnn -i vmbr1040 -e "(arp or icmp) and (host 10.251.85.2 or host 10.251.85.15)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr1040, link-type EN10MB (Ethernet), capture size 262144 bytes
13:39:11.028576 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.15 tell 10.251.85.16, length 28
13:39:11.028628 62:33:ed:c5:7f:bc > a6:dd:b3:91:f5:a8, ethertype ARP (0x0806), length 42: Reply 10.251.85.15 is-at 62:33:ed:c5:7f:bc, length 28
13:39:11.028637 a6:dd:b3:91:f5:a8 > 62:33:ed:c5:7f:bc, ethertype IPv4 (0x0800), length 98: 10.251.85.16 > 10.251.85.15: ICMP echo request, id 397, seq 1, length 64
13:39:11.028658 62:33:ed:c5:7f:bc > a6:dd:b3:91:f5:a8, ethertype IPv4 (0x0800), length 98: 10.251.85.15 > 10.251.85.16: ICMP echo reply, id 397, seq 1, length 64

--- [snip] ---

[root@ct05 ~]# tcpdump -nnn -e "arp or icmp"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:39:11.028567 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.15 tell 10.251.85.16, length 28
11:39:11.028631 62:33:ed:c5:7f:bc > a6:dd:b3:91:f5:a8, ethertype ARP (0x0806), length 42: Reply 10.251.85.15 is-at 62:33:ed:c5:7f:bc, length 28
11:39:11.028636 a6:dd:b3:91:f5:a8 > 62:33:ed:c5:7f:bc, ethertype IPv4 (0x0800), length 98: 10.251.85.16 > 10.251.85.15: ICMP echo request, id 397, seq 1, length 64
11:39:11.028659 62:33:ed:c5:7f:bc > a6:dd:b3:91:f5:a8, ethertype IPv4 (0x0800), length 98: 10.251.85.15 > 10.251.85.16: ICMP echo reply, id 397, seq 1, length 64


As you can see this works, we see both ARP request and reply both on the bridge on the Proxmox host (vmbr1040) as well as inside the container's eth0 interface.

Now let's ping an "external" VM on the same VLAN/subnet:

--- [snip] ---

[root@ct05 ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
10.251.85.7 (incomplete) eth0
10.251.85.15 (incomplete) eth0
10.251.85.2 (incomplete) eth0
10.251.85.1 (incomplete) eth0
[root@ct05 ~]# hostname -i
10.251.85.16
[root@ct05 ~]# ping -c 2 10.251.85.2
PING 10.251.85.2 (10.251.85.2) 56(84) bytes of data.
From 10.251.85.16 icmp_seq=1 Destination Host Unreachable
From 10.251.85.16 icmp_seq=2 Destination Host Unreachable

--- 10.251.85.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1007ms

--- [snip] ---

root@proxmox:~# tcpdump -nnn -i vmbr1040 -e "(arp or icmp) and (host 10.251.85.2 or host 10.251.85.15)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr1040, link-type EN10MB (Ethernet), capture size 262144 bytes
13:49:00.248320 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:00.248439 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:00.248596 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype ARP (0x0806), length 60: Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46
13:49:01.245920 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:01.246041 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:01.246189 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype ARP (0x0806), length 60: Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46
13:49:02.245894 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:02.246017 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:02.246156 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype ARP (0x0806), length 60: Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46

--- [snip] ---

[root@ct05 ~]# tcpdump -nnn -e "arp or icmp"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:49:00.248306 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
11:49:00.248452 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
11:49:01.245908 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
11:49:01.246057 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
11:49:02.245884 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
11:49:02.246031 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46

--- [snip] ---

root@proxmox:~# tcpdump -nnn -i eth1 -e "(arp or icmp) and (host 10.251.85.2 or host 10.251.85.15)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:49:00.248339 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1040, p 0, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:00.248439 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:00.248596 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46
13:49:01.245963 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1040, p 0, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:01.246041 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:01.246189 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46
13:49:02.245939 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1040, p 0, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 28
13:49:02.246017 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Request who-has 10.251.85.2 tell 10.251.85.16, length 46
13:49:02.246156 00:50:56:87:cc:e7 > a6:dd:b3:91:f5:a8, ethertype 802.1Q (0x8100), length 64: vlan 1040, p 1, ethertype ARP, Reply 10.251.85.2 is-at 00:50:56:87:cc:e7, length 46

--- [snip] ---

We still see the ARP requests leaving the container (eth0) and on the VLAN bridge interface (vmbr1040) as well as the VLAN trunk interface (eth1) as seen by ESXi.
Replies come in on eth1 and even on vmbr1040 but they never reach the inside of the container (eth0).
That means networking on the hypervisor-level works and we do have all three of Allow promiscuous mode, Allow forged transmits and Allow MAC changes enabled.

There is a blog post out there titled "Network connectivity problems when running LXC (with veth) in VMware VM" (as a new user I don't seem to be allowed to directly link to it) which seems to exactly describe the problems we are facing.
The workaround described - manually switching a container to use MACVLAN instead of Veth - works for us. Manually populating the arp table on the containers works as well, as soon as the other side's MAC address is made known to the container traffic seems to flow as expected.
All this means that there must be some problem on the Proxmox host.

What's happening here?
Where and why is traffic being filtered and can we get this working with the default Veth?
Should we really need to switch to the MACVLAN type (why so, in this case?), can this be done without breaking management via the GUI?

The network configuration as created via the GUI:

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.251.85.4
netmask 255.255.255.0
gateway 10.251.85.1

iface eth1 inet manual

auto vmbr1040
iface vmbr1040 inet manual
bridge_ports eth1.1040
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes

auto vmbr1041
iface vmbr1041 inet manual
bridge_ports eth1.1041
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes

auto vmbr1043
iface vmbr1043 inet manual
bridge_ports eth1.1043
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes

 

[manu]

Are you seeing the ARP replies when inspecting the veth device from the Proxmox host ? (the veth device as see from the host should have a name like veth0<CTID>i<nic index>
normally you should

for instance here I am inspecting the output of a veth device assigned to CT100( 192.168.31.134), running in a Proxmox VM, which is pinging an external machine located in the same subnet

tcpdump -nnn -i veth100i0 -e "(arp or icmp) and (host 192.168.2.122 or host 192.168.31.134)"
15:19:17.267879 9a:5a:f7:e3:9a:0d > 0c:c4:7a:c5:26:d4, ethertype ARP (0x0806), length 42: Request who-has 192.168.16.1 tell 192.168.31.134, length 28
15:19:17.268207 0c:c4:7a:c5:26:d4 > 9a:5a:f7:e3:9a:0d, ethertype ARP (0x0806), length 60: Reply 192.168.16.1 is-at 0c:c4:7a:c5:26:d4, length 46

 

[bonki]

The ARP replies don't hit the veth device either:

[root@ct05 ~]# ping -c 2 10.251.85.2
PING 10.251.85.2 (10.251.85.2) 56(84) bytes of data.
From 10.251.85.16 icmp_seq=1 Destination Host Unreachable
From 10.251.85.16 icmp_seq=2 Destination Host Unreachable

--- 10.251.85.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1007ms

root@proxmox:~# tcpdump -nnn -i veth104i0 -e "(arp or icmp) and (host 10.251.85.2 or host 10.251.85.16)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth104i0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:43:41.306302 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
08:43:41.306451 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
08:43:42.302578 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
08:43:42.302785 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
08:43:43.302571 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.251.85.2 tell 10.251.85.16, length 28
08:43:43.302773 a6:dd:b3:91:f5:a8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.251.85.2 tell 10.251.85.16, length 46
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
root@proxmox:~#

 

[bonki]

In a nutshell, the following works:

ct@proxmox -> internet (obvious because no ARP)
ct@proxmox -> vm@esx (different VLANs/subnets; no ARP)
ct@proxmox -> ct@proxmox (different VLANs/subnets; no ARP)
ct@proxmox -> ct@proxmox (same VLAN/subnet)

This doesn't:

ct@proxmox -> vm@esx (same VLAN/subnet)

 

[bonki]

Ping! Any idea?

This is a real show-stopper for us as we can't go into production which means we might have to ditch Proxmox VE altogether.

 

[tom]

Ping! Any idea?

This is a real show-stopper for us as we can't go into production which means we might have to ditch Proxmox VE altogether.

Using Proxmox VE inside VMware is not suited for production anyways, use Proxmox VE on bare-metal.

Maybe your issues are caused by the VMware network.?

 

[Leo David]

Hi,
Have you enabled "promiscuous mode" on the vSwitch ?

 

[Leo Winter]

Hi,
Have you enabled "promiscuous mode" on the vSwitch ?

Thanks, this works.
Take a look at the vmware Knowledge Base Article: 1004099