Routing/Arp problem accessing host or other guest vm on vmbr through OpenVPN

nulldeluxe

Renowned Member
Nov 7, 2011
2
0
66
I am using Proxmox VE to run a few KVM virtual machines behind a firewall/gateway. The Proxmox Server has one NIC, assigned to vmbr0. The bridge is assigned a local IP address (192.168.185.254) from the local net (192.168.185.0/8).

One of the VMs is acting as a CentOS based OpenVPN Gateway (static IP 192.168.185.200 assigned). Port 1194 UDP is forwarded from the Gateway connecting the LAN to the Internet to the Proxmox host IP (192.168.185.254) and OpnVPN clients can succesfully connect to the OpenVPN Server on the VM (192.168.185.200). Traffic from the VMs tun+ adapters is masqueraded/NATed on the VM to allow access to machines on the LAN through OpenVPN. For testing purposes, no other firewall rules are in effect on the VMs or the Proxmox host.

I can access the OpenVPN gateway VM from the ouside through ssh once the OpenVPN connection has been established. I can also access a webbased frontend on the same VM (both over http and https). I can also ping other machines on the LAN through OpenVPN (both physical and virtual), DNS resolution is also working. Finally, I am able to connect to other _physical_ machines on the LAN through ssh, http, smb, etc. So far, so good.

The weird thing is, while I am able to ping other VMs connected to the same vmbr device on the Proxmox server, I cannot connect to them or to the Proxmox host web interface through neither ssh, http or https. Running tcpdump on the Proxmox host while trying to access it's web interface over the OpenVPN channel, I could see packages being sent from the OpenVPN VM (properly masqueraded) to the destinantion, as well as the answer from the http host. But after the initial handshake no further packages seem to reach the OpenVPN client. I can see some ARP requests and answers from the http host and then, suddenly, the http server tries to send packages to the unmasqueraded address of the OpenVPN client (10.8.0.6)...

I tried instead setting up ssh port forwarding of the Proxmox host port 443 through the OpenVPN VM and it works without delays (e.g. ssh -L 8080:192.168.185.254:443 192.168.185.200) - I can connect to the Proxmox host's Web interface and manage VMs. If I establish a separate ssh tunnel to forward port 5900 to me, I can even access the VNC console.

I do not quite understand what is happening here. As I said, I can access other physical hosts on the LAN through the OpenVPN gateway without problems, so the OpenVPN setup seems to work as intended. To me it looks as if the different networking layers on the bridge are somehow confused, maybe on the ARP side of things?

I looked around quite a bit and read through a lot of different setup ideas. However I choose this setup to keep it simple - after all, it's a small LAN and none of the switches involved are having difficulties with multiple MACs on the same port, so why not just plug all VMs onto the same vmbr device...

I'm probably not seeing the obvious solution here, but maybe somebody could point me in the right direction? I'd be happy to provide more information if needed.

Thx in advance! Proxmox VE is a fantastic product and I'm very impressed with it!
 

Hi Dietmar,

I followed the discussion you mentioned and tried switching rp_filter off on all involved interfaces, but it dind't chnage the behaviour. Meanwhile I followed the packets back and forth and to my surprise I can see replies from the contacted server on the OpenVPN Clients tun-interface using tcpdump - but the http-client does not "see" or recognise them... So I'm confused as this might be related to OpenVPN, but then again, is it is, why can I conatct services (ssh, http, https,...) physically "outside" the vmbr0-interface without problems over OpenVPN? I' like to apologize in advance if this is something related to OpenVPN that i have overlooked.

Again, to clarify:
I have a box with a single NIC that ist connected to vmbr0 (the brigde is assigned IP 192.168.185.254 from the local 192.168.185.0/8 net) in Proxmox 1.9.
I have a KVM guest (CentOS based ClearOS 5.2, kernel 2.6.18-194.8.1.v5) acting as my OpenVPN gateway, connected to vmbr0 and assigned IP 192.168.185.200. There are also two other KVM guests (Ubuntu and Windows 2003) connected to vmbr0 and assigned IPs from the local net.
I can connect to the OpenVPN gateway from outside and successfully ping both the gateway, the other KVM guests and other physical machines on the local network. I can also establish TCP connections to the OpenVPN gateway server on 192.168.185.200 as well as other _physical_ hosts on the network (e.g. 192.168.185.1) over http, https and ssh. But whenever I try to access other hosts on vmbr0 or the Proxmox box itself on 192.168.185.254 over TCP, my OpenVPN Client machine does only "see" the initial connection packets, even if I can see other packets in the connection arrive on the OpenVPN clients tun-interface. So something must be "happening" to the packets being sent over the vmbr0 interface. Or maybe vital information is lost in one of the tunneling interfaces...

I appreciate any ideas on how to diagnose this further. If this question does not belong here, please let me know!