[SOLVED] NOT again! NAT again.. (MASQUERADE failing)

[pico_prob]

Requests from VM in subnet are forwarded to pve's gateway, but with subnet source IP.

My masquerading with iptables fails in this common use-case:
Internet <-eth0-> pve host <-vmbr0-> VM (10.10.0.1)

With sudo tcpdump -n -i enp0s31f6 icmp I get:

17:26:35.506557 IP 10.0.0.2 > 94.130.yy.zz: ICMP echo request, id 601, seq 1151, length 64

Hence request leave the host, but can't be responded to (as source not routable).



Further Infos:

• Current Proxmox VE (ie 7.3-6), installed from ISO
• headless bare metal server (hetzner), on network card with one public IP
• VM: plain debian, static IP 10.0.0.2, gateway: 10.0.0.1.
• The usual NAT command (post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/30' -o enp0s31f6 -j MASQUERADE) seems to have no effect on the forewarding. If I delete the rule with iptable, the requests still appear on tcpdump
• En-/disabling the host's pve firewall has no effect.
• Using the recommended setup from wiki does not forward any requests (not oneth0, not on vmbr0), so is this working interfaces
• suggested addition post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 has no effect.

I have to add, that I am not that firm on networking. But I am trying hard and I am pretty sure, I read all posts to masquerading in this forum. I think I am on this problem for almost a week. Though this is probably due to missing background

Any help is very appreciated!
Kind regards,
Mathis

My /etc/network/interfaces is a bit more complex, as my true intention (german source) is a VM net behind a pfsense. However, I think this should not make any difference:

Spoiler: interfaces file

auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet manual
#outside world

auto vmbr0
iface vmbr0 inet static
    address 94.130.yy.xx/26
    gateway 94.130.yy.zz
    bridge-ports enp0s31f6
    bridge-stp off
    bridge-fd 0
#ip endpoint to outside world

auto vmbr1
iface vmbr1 inet static
    address 10.0.0.1/30
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#in front of firewall

auto vmbr2
iface vmbr2 inet static
    address 192.168.77.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#VMs behind firewall

# Natting
# 1. Allow traffic routing
post-up   echo 1 > /proc/sys/net/ipv4/ip_forward

# 2. Input to firewall
post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22767 -j DNAT --to 10.0.0.2
post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -j DNAT --to 10.0.0.2

# 3. Output Firewall -> outside
post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/30' -o enp0s31f6 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/30' -o enp0s31f6 -j MASQUERADE

# 4. Conflict with proxmox firewall iptables chain PVEFW-FORWARD
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

 

[B.Otto]

Hmm, the examples on the wiki assume that there is a single WAN-interface and not a 'WAN-bridge'. You might have to replace the interface enp0s31f6 in your rules with the bridge holding your external IP vmbr0 like so:

post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/30' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/30' -o vmbr0 -j MASQUERADE

The naturally created routing table should be something like
0.0.0.0 via router 94.130.yy.zz and interface vmbr0
so then the rule would only work if the output interface is vmbr0.

Kind regards,
Benedikt

 

[pico_prob]

Hey Benedikt, thanks for the quick response.
Actually, I have tried this before writing this post in several interfaces configurations to no avail.

However, due diligence, I checked again. And it worked! I assume that there were other parameter involved. I am astonished, that I never tried it with this "original version" of interfaces.

For the record:

• I also needed to change the ip of vmbr2 to not conflict with pfsense (192.168.77.1). I used 192.168.77.251.
• I didn't need (yet?) the last block in interfaces on -j CT --zone 1.

Last question:

0.0.0.0 via router 94.130.yy.zz and interface vmbr0

This refers to the output of ip route show, right?

Thank you again..!

 

[B.Otto]

Hey Benedikt, thanks for the quick response.
Actually, I have tried this before writing this post in several interfaces configurations to no avail.

However, due diligence, I checked again. And it worked! I assume that there were other parameter involved. I am astonished, that I never tried it with this "original version" of interfaces.

For the record:

• I also needed to change the ip of vmbr2 to not conflict with pfsense (192.168.77.1). I used 192.168.77.251.

Good to hear! Maybe you forgot to apply the configuration, or had a tiny mistype. These kind of configurations are finnicky. IP conflicts are always bad.

I didn't need (yet?) the last block in interfaces on -j CT --zone 1.

Iirc that rule refers to interfaces of LXC containers and is irrelevant to VM interfaces.

This refers to the output of ip route show, right?

Kinda, mine was in pseudocode, but it refers to the content of ip route show or route -n