How to enable loopback (hairpin) NAT so that a container can reach another container via PVE public IP?

S

stevops

New Member
Aug 7, 2022
21
1
3
Preface
Hi together, this thread is highly connected to the issue I explained here: Connected issue
I think it has the same root cause but since I got no answer there I tried to narrow down the problem, reframe it. So now I have a different symptom that is based on a more "common" scenario, that I want to explain in this new thread here. I hope this is ok.

Issue
So, here is the overview:
Untitled Diagram(2)(2).drawio.png

I have a base network (192.168.2.0/24) for PCs etc. where also a PVE node sits and hosts some services in a internal network (10.0.0.0/24) that is SNATed (Gateway = 10.0.0.1). More details down below.

To make services from the internal network available to the outside world I implemented firewall rules via iptables:
Code: 
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 5140 -j DNAT --to-destination 10.0.0.3:5140

Connections from the PC to the syslog server are working fine (the green line).
Connections from another instance of the internal network that are pointing to the public IP address of the PVE node are not working (red line). This is expected so far because I need another rule here to enable loopback/hairpin NAT. Thats the rule I used:
Code: 
iptables -t nat -I POSTROUTING -s 10.0.0.3 -p tcp --dport 5140 -j SNAT --to 192.168.2.10

But thats not working too.

When I trace the packet flow this is what I get:

Code: 
TRACE: raw:PREROUTING:policy:5 IN=vrfbr_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: mangle:PREROUTING:policy:1 IN=vrfbr_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: nat:PREROUTING:policy:10 IN=vrfbr_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: raw:PREROUTING:rule:4 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: mangle:PREROUTING:policy:1 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: mangle:INPUT:policy:1 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: filter:INPUT:policy:3 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: security:INPUT:policy:1 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN
TRACE: nat:INPUT:policy:1 IN=vrf_inevpn1 MAC=7e:bf:55:eb:01:42:12:79:af:38:d5:b4:08:00 SRC=10.0.0.3 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=34947 DF PROTO=TCP SPT=41818 DPT=5140 SEQ=1911070483 ACK=0 WINDOW=64860 SYN

So the packet seems to never reach the point where I can SNAT it to force loopback/hairpin NAT. Do you have any ideas how to troubleshoot this one?

Internal Network
Some more infos to my internal network:
It is realized as EVPN/VNET/SUBNET with SNAT option enabled:
1671007338278.png
 
Is there any particular reason why you want to go via the PVE host for this? Would it make sense for you to just connect via the 10.0.0.0/24 IP?
 
Hi @shanreich , thank you for joining this thread. Actually there are three use cases for this solution:
  1. Monitoring: Within the internal network there is a monitoring solution (uptime kuma) that verifies that all services are up. This service shall also test services as if it is positioned in the outer network (uptime kuma -> check DNS on port 53 on 192.168.2.10). The best place for uptime kuma is actually the internal network to check all the internal resources that should not be public facing ones. I otherwise need to setup a second uptime kuma instance in the outer network.
  2. DNS: In the internal network is DNS resolving service. That resolves IP addresses of the outer network and is used by some internal services. Without the ability of loopback NAT I would need to setup a DNS resolver for the internal and outer network seperately.
  3. Learn what is the root cause for this issue to adapt the solution for the origin of this thread :) Connected issue
 
In this case could you post the VM configuration, network configuration and output of ip a of the xyz-Server, as well as the output of ip route on that server?
 
Of course! :) Thank you for helping me out. Actually it is a LXC container.

Bash: 
## xyz server
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether f2:d2:86:5c:88:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.76/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f0d2:86ff:fe5c:8804/64 scope link
       valid_lft forever preferred_lft forever
      
# ip route
default via 10.0.0.1 dev eth0 proto static
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.76

1671020397175.png

Is that the correct output you expected?
 

Attachments

  • 1671020367754.png
    1671020367754.png
    7.9 KB · Views: 2
I will have to find some time to reproduce this setup on my machine, since it is bit more complicated (but I wanted to play around with this anyway, so this is a good opportunity).

One suspicion I have - might it be that the xyz-Server simply doesn't know where to route packets with 192.168.0.0/24 IPs to? But as I said, I will have to look a bit further into this, this is just a first guess after taking a glance at the output. I don't have that much experience with EVPN setups yet.

edit: nvm this should be covered by the default route....
 
Last edited:
What is the exact network configuration on the host and the guest (located in /etc/network/interfaces)

interesting would also be the SDN config files located at /etc/pve/sdn
 
Here it is:

Guest

The exact network configuration of the guest (the LXC containers like xyz Server is one) is very simple like I showed above:
1671042282450.png
All the other machines just have a different IP. If you need more please tell me what I can gather from where.

Host

Code: 
# /etc/network/interfaces.d/sdn
#------------------------------                                                                                                                 
#version:7

auto invnet1
iface invnet1
        address 10.0.0.1/24
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 192.168.2.10
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 192.168.2.10
        post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
        hwaddress 3E:4D:DC:0B:6E:27
        bridge_ports vxlan_invnet1
        bridge_stp off
        bridge_fd 0
        mtu 1450
        ip-forward on
        arp-accept on
        vrf vrf_inevpn1

auto vrf_inevpn1
iface vrf_inevpn1
        vrf-table auto
        post-up ip route del vrf vrf_inevpn1 unreachable default metric 4278198272

auto vrfbr_inevpn1
iface vrfbr_inevpn1
        bridge-ports vrfvx_inevpn1
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_inevpn1

auto vrfvx_inevpn1
iface vrfvx_inevpn1
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.2.10
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_invnet1
iface vxlan_invnet1
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.2.10
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450
        
        
# /etc/network/interfaces
#------------------------------ 
auto lo
iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.2.10/24
        gateway 192.168.2.1
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0

iface wlo1 inet manual

auto vmbr99
iface vmbr99 inet static
        address 10.0.99.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#PVE SSH restriction network

source /etc/network/interfaces.d/*

# /etc/pve/sdn/controllers.cfg
#------------------------------                                                         
evpn: evpnctl1
        asn 65000
        peers 192.168.2.10,192.168.2.11
        

# /etc/pve/sdn/subnets.cfg
#------------------------------ 
subnet: inevpn1-10.0.0.0-24
        vnet invnet1
        gateway 10.0.0.1
        snat 1

# /etc/pve/sdn/vnets.cfg
#------------------------------         
vnet: invnet1
        zone inevpn1
        tag 11000

# /etc/pve/sdn/zones.cfg
#------------------------------   
evpn: inevpn1
        controller evpnctl1
        vrf-vxlan 10000
        exitnodes pve1,pve2
        exitnodes-primary pve1
        ipam pve
        mac 3E:4D:DC:0B:6E:27
        mtu 1450
 
With this iptables rule it works for me (adjust your subnet accordingly):

Code: 
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -p tcp --dport 5140 -j MASQUERADE

edit: with SNAT it works as well:

Code: 
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -p tcp --dport 5140 -j SNAT --to-source 192.168.2.10


edit2: I think the issue in your case was that the source IP address (the one specified via -s) was wrong. It needs to be the address of the container that is trying to connect to 10.0.0.3, not 10.0.0.3 itself. If I substitute the IP address in your command it works as well:

Code: 
iptables -t nat -I POSTROUTING -s 10.0.0.76 -p tcp --dport 5140 -j SNAT --to 192.168.2.10
 
Last edited:
Hi, unfortunately I am not able to reproduce it. I inserted the rule and now my postrouting chain looks like this:

Code: 
Chain POSTROUTING (policy ACCEPT 22 packets, 2222 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 SNAT       tcp  --  *      *       10.0.0.0/24        0.0.0.0/0            tcp dpt:5140 to:192.168.2.10
 176K   13M SNAT       all  --  *      vmbr0   10.0.0.0/24        0.0.0.0/0            to:192.168.2.10

Two thoughts so far:
1) The rule I inserted never gets hit (the second rule is the one that is auto-generated by PVE SDN module if you tick the "SNAT" option for the subnet)
2) As you can see in my (or at least as far as I understand it) trace logs the packets do not arrive at the postrouting chain at all. The packets disappear after nat:INPUT.

Could you provide a trace of yours?

Code: 
# Enable tracing logs if not already done
sudo modprobe ipt_LOG 

# Add iptables rules
sudo iptables -t raw -A OUTPUT -p tcp --destination 192.168.2.10 --dport 5140 -j TRACE
sudo iptables -t raw -A PREROUTING -p tcp --destination 192.168.2.10 --dport 5140 -j TRACE

# "telnet 192.168.2.10 5140" from internal network container like from 10.0.0.76

# Get the traces for the source IP
sudo cat /var/log/pve-firewall.log | grep 10.0.0.76
 
Last edited:
I'll try to check it, hopefully tomorrow

How do your iptables rules look in general?

iptables -t nat -L --line-numbers
 
How mine look atm (had to use some different subnets, since they were already used in my virtual cluster):

Code: 
root@pve-lana-dev-2:~# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DNAT       tcp  --  anywhere             anywhere             tcp dpt:5140 to:10.123.0.3:5140

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    SNAT       tcp  --  10.123.0.76          anywhere             tcp dpt:5140 to:192.168.23.200
2    SNAT       all  --  10.123.0.0/24        anywhere             to:192.168.23.200
3    SNAT       all  --  10.123.0.0/24        anywhere             to:192.168.23.200


and a quick trace:

Code: 
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22774 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830705 ACK=0 WINDOW=64860 SYN
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: nat:PREROUTING:rule:1 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22774 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830705 ACK=0 WINDOW=64860 SYN
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22774 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830705 ACK=0 WINDOW=64860 SYN
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=22774 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830705 ACK=0 WINDOW=64860 SYN
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: nat:POSTROUTING:rule:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=22774 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830705 ACK=0 WINDOW=64860 SYN
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22775 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22775 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:22 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=22775 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=22776 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=22776 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=57 TOS=0x00 PREC=0x00 TTL=63 ID=22776 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830706 ACK=3486955086 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22777 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22777 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=22777 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=22778 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=22778 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=57 TOS=0x00 PREC=0x00 TTL=63 ID=22778 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830711 ACK=3486955091 WINDOW=507 ACK PSH
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22779 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22779 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:23 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=22779 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22780 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK FIN
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22780 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK FIN
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=22780 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830716 ACK=3486955096 WINDOW=507 ACK FIN
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: raw:PREROUTING:policy:5 IN=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=192.168.23.200 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22781 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830717 ACK=3486955097 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: raw:PREROUTING:policy:5 IN=vrf_inevpn1 MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22781 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830717 ACK=3486955097 WINDOW=507 ACK
0 6 - 15/Dec/2022:16:14:25 +0100 TRACE: filter:FORWARD:policy:1 IN=vrf_inevpn1 OUT=test MAC=aa:aa:aa:52:80:b7:aa:aa:aa:ef:06:d4:08:00 SRC=10.123.0.76 DST=10.123.0.3 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=22781 DF PROTO=TCP SPT=46818 DPT=5140 SEQ=404830717 ACK=3486955097 WINDOW=507 ACK
 
I'll compare these lines later on. Thank you so far. I need to catch a train now ;) But here is my NAT table:


Code: 
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DNAT       tcp  --  anywhere             anywhere             tcp dpt:10022 to:10.0.99.1:22
2    DNAT       tcp  --  anywhere             anywhere             tcp dpt:943 to:10.0.0.2:943
3    DNAT       udp  --  anywhere             anywhere             udp dpt:openvpn to:10.0.0.2:1194
4    DNAT       tcp  --  anywhere             anywhere             tcp dpt:5140 to:10.0.0.3:5140
5    DNAT       udp  --  anywhere             anywhere             udp dpt:5140 to:10.0.0.3:5140
6    DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:10.0.0.95:443
7    DNAT       tcp  --  anywhere             anywhere             tcp dpt:1883 to:10.0.0.86:1883
8    DNAT       tcp  --  anywhere             anywhere             tcp dpt:domain to:10.0.0.65:53
9    DNAT       udp  --  anywhere             anywhere             udp dpt:domain to:10.0.0.65:53
10   DNAT       tcp  --  anywhere             anywhere             tcp dpt:smtp to:10.0.0.26:25

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DNAT       tcp  -- !127.0.0.0/8          anywhere             tcp dpt:5140 to:10.0.0.3:5140

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    SNAT       tcp  --  10.0.0.0/24        anywhere             tcp dpt:5140 to:192.168.2.10
2    SNAT       all  --  10.0.0.0/24        anywhere             to:192.168.2.10
 
Hi, no findings so far from my side. I made an test environment where I have the exactly same behaviour. The important SNAT rule is never hit and the packet flow ends in nat:INPUT. -.-

Are you running an active cluster firewall? I am not.
Do you also have to instances coupled as a cluster?
 
Hi @shanreich , I am stuck. I made myself an ansible playbook that builds up an test environment that mostly the same than my production setup. Same here. There must be something different on your setup compared to mine :(
 
I tested this in a cluster and didn't have an active firewall, so this doesn't seem to be it. I'll see if I have time to look at this a bit later today - maybe something springs into my mind, but for now I cannot really think of any difference as well.
 
  • Like
Reactions: stevops
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom