[SOLVED] VM cannot respond to port forwarded traffic

[mymindstorm]

Hi,

I have a bit of a confusing issue. My guest vms are currently connected to my LAN using the default bridged mode. I have a VM serving http requests on vmbr0, and I can access its http server from another device in the same subnet. I have turned off all firewalls on the vm and proxmox except for ebtables. If I access the server from a public IP via a port forwarded request, no response is given. When I look at tcpdump, it is clear that the VM receives the request from a public ip, but it does not respond. I'm fairly certain that this is a networking problem as other guest VMs have the exact same issue. Any ideas on how to fix this would be appreciated.

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet dhcp
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet dhcp

auto vmbr0.20
iface vmbr0.20 inet dhcp

auto vmbr0.50
iface vmbr0.50 inet dhcp

 

[Lorenz.S]

Do the VMs get a default gateway/default route via DHCP? You can check it with ip route

 

[mymindstorm]

Yes:

default via 192.168.100.1 dev eth0 proto dhcp metric 100 
default via 192.168.150.1 dev eth1 proto dhcp metric 101 
default via 192.168.1.1 dev eth2 proto dhcp metric 102 
192.168.1.0/24 dev eth2 proto kernel scope link src 192.168.1.206 metric 102 
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.16 metric 100 
192.168.150.0/24 dev eth1 proto kernel scope link src 192.168.150.5 metric 101

In this case, vmbr0 is 192.168.1.0/24 on eth2

tcpdump from internal network:

[root@$HOST]# tcpdump -i eth2 port 8080
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:15:14.377740 IP 192.168.1.39.56474 > $HOST.webcache: Flags [F.], seq 1042506034, ack 1703356995, win 502, options [nop,nop,TS val 3603580827 ecr 1117333670], length 0
17:15:14.377801 IP $HOST.webcache > 192.168.1.39.56474: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 1117338670 ecr 3603580827], length 0
...

tcdump from port-forwarded client:

[root@$HOST]# tcpdump -i eth2 port 8080
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:17:10.808601 IP $WEB_IP.56914 > $HOST.webcache: Flags [S], seq 2651131125, win 65535, options [mss 1376,sackOK,TS val 3867513970 ecr 0,nop,wscale 10], length 0
17:17:10.811006 IP $WEB_IP.58262 > $HOST.webcache: Flags [S], seq 2202891635, win 65535, options [mss 1376,sackOK,TS val 3867513969 ecr 0,nop,wscale 10], length 0
17:17:11.074643 IP $WEB_IP.21593 > $HOST.webcache: Flags [S], seq 940990364, win 65535, options [mss 1376,sackOK,TS val 3867514222 ecr 0,nop,wscale 10], length 0
17:17:11.847829 IP $WEB_IP.56914 > $HOST.webcache: Flags [S], seq 2651131125, win 65535, options [mss 1376,sackOK,TS val 3867514977 ecr 0,nop,wscale 10], length 0
17:17:11.849405 IP $WEB_IP.58262 > $HOST.webcache: Flags [S], seq 2202891635, win 65535, options [mss 1376,sackOK,TS val 3867514977 ecr 0,nop,wscale 10], length 0

 

[Lorenz.S]

There is your problem. If the system has multiple default routes, the one with lowest metric will be used, in this case 192.168.100.1, see [0].

An ugly fix would be to set the metric manually to a lower value.
However it would be better to configure the DHCP server on eth0/1 to not send a default gateway, because your problem seems to suggest, that this default gateway is not able to deliver these packets, otherwise you would not have this problem.

[0] https://en.wikipedia.org/wiki/Routing#Path_selection

 

[mymindstorm]

Thanks, that was it. Changing the metric yielded no results, but I was able to ignore the routes using networkmanager like so:

nmcli con modify "System eth0" ipv4.never-default yes