[SOLVED] [NETWORK] Routing with MASQUERADE from vm lans to wan interface not receiving traffic

C

Christian Barbieri

New Member
Jul 24, 2020
5
0
1
33
Hello Fellow proxmoxers,
I've been using this VE for a while now, and I have to say I have been pretty pleased with it through the years.
Alas, now I am facing a weird issue that I cannot seem to find the solution of.

PVEVERSION: pve-manager/6.2-10/a20769ed (running kernel: 5.4.44-2-pve)

CURRENT /etc/network/interface OF PVE
auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet static
address 192.168.12.2/24
#PVE VLAN

auto vmbr0
iface vmbr0 inet dhcp
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.33.1/24
gateway 192.168.33.1
bridge-ports none
bridge-stp off
bridge-fd 0
#Virtual VMs LAN

With this setup, the current situation is:
1. eno1 [bridge vmbr0] is the NIC with public IP of the pve itself for network access
2. eno2 is a dedicated NIC that bridges across multiple pve instances [dedicated zfs snapshot channel]
3. vmbr0 bridge is the network that gives internet access + is the network given to those VMs that require a public IP themselves [ for IP FAILOVERS ]
4. vmbr1 is the VMs network [ for inter communication among various virtual machines] and should be the way that some virtual machines [e.g database instances] talk to the outside world, via MASQUERADING of connections with src vmbr1 that go through vmbr0

All points from 1 to 3 work fine (and half of 4), vms can talk with eachothers, vms can ping the vmbr1 pve address, zfs replication via eno2 is working fine and eno1 network access is good as well.
What is unusual is that i cannot seem to give vms access to internet via MASQUERADING of vmbr0 [or eno1] :
-A POSTROUTING -s 192.168.33.0/24 -o vmbr0 -j MASQUERADE
iptables -P FORWARD is on ACCEPT for sake of testing
ipv4_forward in sysctl is set to 1
rt_filters = 0 (all of them)

The routing table is [route -n]:
Code: 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         [secret]        0.0.0.0         UG    0      0        0 vmbr0
[secret-wan]    0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
192.168.12.0    0.0.0.0         255.255.255.0   U     0      0        0 eno2
192.168.33.0    0.0.0.0         255.255.255.0   U     0      0        0 vmbr1

With ip route:
default via [redacted-gateway] dev vmbr0
[redacted-wan-network] dev vmbr0 proto kernel scope link src [redacted-wan-ip]
192.168.12.0/24 dev eno2 proto kernel scope link src 192.168.12.2
192.168.33.0/24 dev vmbr1 proto kernel scope link src 192.168.33.1

PING from a VM with IP 192.168.33.5 to 8.8.8.8 does not see to work. During this ping, tcpdump -i vmbr0 [or eno1] give the following output:
11:14:42.958458 IP proxmox6 > dns.google: ICMP echo request, id 19562, seq 93, length 64
11:14:42.959951 IP dns.google > proxmox6: ICMP echo reply, id 19562, seq 93, length 64

With iptables LOGs i only see this line currently repeated:
proxmox6 kernel: [73972.733861] IN=vmbr1 OUT=vmbr0 [redacted MACs] SRC=192.168.33.5 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=44844 DF PROTO=ICMP TYPE=8 CODE=0 ID=19562 SEQ=179

With rules iptables -j LOG as :
-A PREROUTING -p icmp -j LOG
-A POSTROUTING -j LOG
-A FORWARD -j LOG
-A OUTPUT -p icmp -j LOG

So... from tcpdump it seems packets do come back from google dns to the vmbr0 interface, as they are being MASQUERADED correctly.. but from there, poof, gone. Also, from logging of iptables -j LOG i cannot see a coming back packet...

Last but not list, the same thing happens if i ping from PVE itself via ping -I vmbr1 (as in, 192.168.33.1 which is the gateway for the VMs assigned to pve itself)

I am open to any idea really, been working on this 2 days and i cannot really figure out wtf is going on, need a routing table for the subnet? But the kernel seems to know that packets for 192.168.33.0 have to go via vmbr1...

Thanks for eventual responses
 
Last edited:
Hello, thanks for the heads up @jlebherz about the gateway on the subnet, can't believe i missed that even though that was not set as gw on routing rules. Oddly enough. routing still does not seem to work
Also, pve firewall is disabled in GUI [second commend reply], I can't exactly replicate that setup as i need a bridge for public IPs on the wan interface, the rest yes of it yes, it's replicated. I will read a bit more, just in case, thanks
[EDIT]
What I am trying to setup is basically points 3.3.4. and 3.3.6. together of that guide. I could try to migrate point 3.3.4 to 3.3.5 but that would leave me with 2 public IPs assigned to the pve mmmh.. I'll fiddle a bit with it thanks
 
Last edited:
About the fwbr prerouting conntrack, firewall is disabled but i tried it nonetheless, with no luck (i directly added those rules via iptables rather than using them in interfaces up/down actions).
[PVE FIREWALL STATUS]
Code: 
root@proxmox6:~# pve-firewall status
Status: disabled/stopped

[IP FORWARD STATUS]
Code: 
root@proxmox6:~# cat /proc/sys/net/ipv4/ip_forward
1

[IPTABLES RULES]
Code: 
root@proxmox6:~# iptables -S

-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i eno2 -p tcp -m tcp --dport [ssh-port] -j ACCEPT
-A INPUT -i vmbr0 -p tcp -m tcp --dport [ssh-port] -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -m limit --limit 4/sec -j ACCEPT
-A INPUT -s [redacted-whitelist-input-ip] -j ACCEPT
-A FORWARD -i vmbr0 -o vmbr1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.33.5/32 -i vmbr1 -o vmbr0 -j ACCEPT
-A FORWARD -i tun0 -o vmbr1 -j ACCEPT
-A FORWARD -i vmbr1 -o tun0 -j ACCEPT

root@proxmox6:~# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 192.168.33.0/24 -o vmbr0 -j MASQUERADE

root@proxmox6:~# iptables -t raw -S
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A PREROUTING -i fwbr+ -j CT --zone 1

tun0 interface has been disabled for the sake of testing before reporting here, it's an openvpn connection for some services that unfortunatelly cannot be migrated to tls/ssl

Routing inside a centos vm [ALL iptables policies to ACCEPT]:
[VM ROUTING]
Code: 
[root@z[hostname] ~]# ip route
default via 192.168.33.1 dev vmlan 
169.254.0.0/16 dev vmlan  scope link  metric 1002 
192.168.33.0/24 dev vmlan  proto kernel  scope link  src 192.168.33.5

VMs are having their 192.168.33.0/24 ip given via dhcp if that can help (which i tried to disable and assign static ips..).
[DHCP SERVER CONFIGURATION]
Code: 
option domain-name-servers [internal], 208.67.222.222, 8.8.8.8;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
subnet 192.168.33.0 netmask 255.255.255.0 {
        option routers 192.168.33.1;

        host [hostname] {
                hardware ethernet [macaddress];
                fixed-address 192.168.33.2;
        }
        host 2,3,4 etc
}

Last but not list, ping 8.8.8.8 from a VM with IP 192.168.33.2.
I have added the following TRACES, to the proxmox instance:
[IPTABLES TRACE PACKETS]
Code: 
root@proxmox6:~# iptables -t raw -S
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A PREROUTING -i fwbr+ -j CT --zone 1
-A PREROUTING -s 192.168.33.0/24 -p icmp -j TRACE
-A PREROUTING -s 8.8.8.8/32 -p icmp -j TRACE
And the log reports:
[IPTABLES TRACE LOG OUTPUT]
Code: 
Jul 25 23:41:02 proxmox6 kernel: [ 3965.728976] TRACE: raw:PREROUTING:policy:4 IN=vmbr1 OUT= MAC=[MAC-ADDRESSES] SRC=192.168.33.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12287 DF PROTO=ICMP TYPE=8 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.731396] TRACE: mangle:PREROUTING:policy:1 IN=vmbr1 OUT= MAC=[MAC-ADDRESSES] SRC=192.168.33.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12287 DF PROTO=ICMP TYPE=8 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.733715] TRACE: mangle:FORWARD:policy:1 IN=vmbr1 OUT=vmbr0 MAC=[MAC-ADDRESSES] SRC=192.168.33.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=12287 DF PROTO=ICMP TYPE=8 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.736091] TRACE: filter:FORWARD:policy:5 IN=vmbr1 OUT=vmbr0 MAC=[MAC-ADDRESSES] SRC=192.168.33.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=12287 DF PROTO=ICMP TYPE=8 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.738426] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=vmbr0 SRC=192.168.33.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=12287 DF PROTO=ICMP TYPE=8 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.742283] TRACE: raw:PREROUTING:policy:4 IN=vmbr0 OUT= MAC=[MAC-ADDRESSES] SRC=8.8.8.8 DST=[PUBLIC-IP] LEN=84 TOS=0x00 PREC=0x00 TTL=114 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=196 SEQ=423 

Jul 25 23:41:02 proxmox6 kernel: [ 3965.744411] TRACE: mangle:PREROUTING:policy:1 IN=vmbr0 OUT= MAC=[MAC-ADDRESSES] SRC=8.8.8.8 DST=[PUBLIC-IP] LEN=84 TOS=0x00 PREC=0x00 TTL=114 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=196 SEQ=423

So, packets are lost after PREROUTING of mangle when they come back. They come back correctly (it seems?) and just blatantly dropped. This behaviour is explained as some form of spoof protection, and to disable it rp_filter system variable should be set to 0...
[SYCTL CONFIG]
Code: 
root@proxmox6:~# sysctl -p
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.all.log_martians = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

[RP_FILTER FOR INTERFACE STATUS]
Code: 
root@proxmox6:~# find /proc/sys -name "rp_filter" | while read ffrp
> do
> echo "$ffrp" && cat "$ffrp"
> done
/proc/sys/net/ipv4/conf/all/rp_filter
0
/proc/sys/net/ipv4/conf/default/rp_filter
0
/proc/sys/net/ipv4/conf/eno1/rp_filter
0
/proc/sys/net/ipv4/conf/eno2/rp_filter
0
/proc/sys/net/ipv4/conf/fwbr100i1/rp_filter
0
/proc/sys/net/ipv4/conf/fwbr101i0/rp_filter
0
/proc/sys/net/ipv4/conf/fwbr101i1/rp_filter
0
/proc/sys/net/ipv4/conf/fwln100i1/rp_filter
0
/proc/sys/net/ipv4/conf/fwln101i0/rp_filter
0
/proc/sys/net/ipv4/conf/fwln101i1/rp_filter
0
/proc/sys/net/ipv4/conf/fwpr100p1/rp_filter
0
/proc/sys/net/ipv4/conf/fwpr101p0/rp_filter
0
/proc/sys/net/ipv4/conf/fwpr101p1/rp_filter
0
/proc/sys/net/ipv4/conf/lo/rp_filter
0
/proc/sys/net/ipv4/conf/tap101i0/rp_filter
0
/proc/sys/net/ipv4/conf/tap101i1/rp_filter
0
/proc/sys/net/ipv4/conf/veth100i1/rp_filter
0
/proc/sys/net/ipv4/conf/veth106i1/rp_filter
0
/proc/sys/net/ipv4/conf/vmbr0/rp_filter
0
/proc/sys/net/ipv4/conf/vmbr1/rp_filter
0

I Am sO CoNfuSeD...
 
I tried removing vmbr0 and using eno1 directly rather than having a bridge. The connectivity WORKS like that. Unfortunately, that is not viable for me. I will try playing with the routing tables tomorrow, thanks again for the inputs
 
Found an acceptable solution I think... somehow what gives troubles is the DHCP directive in /etc/network/interfaces for vmbr0 (iface vmbr0 inet dhcp)
By setting this as static the routing seems to work... I have no idea what changes given the usage of interfaces rather than pre-set addresses in the routing rules and on iptables.
Well, guess this will do... would like to find out one day why it would not work.. Thanks for the contributions
 
Indeed vmbr0 with dhcp config messes up SNAT. Any idea why? Is the dhclient process doing something special?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom