Use PVE node's external IP address to reach service of a internal network via DNAT

[stevops]

Hi everybody,

I am struggeling with a problem where I did not figure out yet if it is a "basic" networking problem or something that has to do with my SDN configuration.
The setup is the following:

I have two VEs (192.168.2.10 and .11) coupled as a cluster. Within this cluster there is an internal network (10.0.0.0/24). It is realized as EVPN/VNET/SUBNET with SNAT option enabled:


In this network there is a server (syslog server hosted on PVE1 listening on port 5140). To make this server available for resources outside of the internal network there is a active IPtables DNAT rule on each PVE node:

Chain PREROUTING (policy ACCEPT 654K packets, 32M bytes)

pkts bytes target     prot opt in     out     source               destination         

10   748 DNAT       tcp  --  vmbr0  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5140 to:10.0.0.3:5140

That is basically working. All hosts (including PVE2) of the network 192.168.2.0/24 can call the port 5140 via telnet ($> telnet 192.168.2.10 5140).

Now I want PVE1 to also use this syslog server by calling its own IP (192.168.2.10). But this does not work. Now I am struggeling to figure out the underlying problem. Is it a problem of the SDN configuration or just the NATting that needs more config?

Normally I would like to configure PVE1 to use the internal IP of the server (10.0.0.3) but this doesn't work either. I think this is because of the SDN and how it works. I thought that the "Exit Nodes local routing" option should have done the trick. But that one did not work as well and brought further networking problems I just did not want to troubleshoot yet as long as there is an option to use the external IP (192.168.2.10) and DNAT.

 

[stevops]

Update 1: I realized that my DNAT rule in the PREROUTING chain is not affected by packets initiated by the host itself (PVE1). Therefore I added a nearly same rule into the OUTPUT chain. So, now the NAT table looks like this:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5140 to:10.0.0.3:5140

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  -- !127.0.0.0/8          0.0.0.0/0            tcp dpt:5140 to:10.0.0.3:5140

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
SNAT       all  --  10.0.0.0/24        0.0.0.0/0            to:192.168.2.10

The rule in the output chain gets hit and now a timeout appears when trying to connect from PVE1 to 192.168.2.10 on port 5140 instead of a "connection refused" message. But something is still missing here. I am asking myself if I need an additional SNAT rule in the POSTROUTING chain like in more traditional NAT loopback scenarios?

 

[stevops]

When tracing the packet flow it looks pretty ok including the NAT POSTROUTING step:

0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: raw:OUTPUT:policy:2 OUT=lo SRC=192.168.2.10 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: mangle:OUTPUT:policy:1 OUT=lo SRC=192.168.2.10 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: nat:OUTPUT:rule:1 OUT=lo SRC=192.168.2.10 DST=192.168.2.10 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: filter:OUTPUT:policy:2 OUT=lo SRC=192.168.2.10 DST=10.0.0.3 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: security:OUTPUT:policy:1 OUT=lo SRC=192.168.2.10 DST=10.0.0.3 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: mangle:POSTROUTING:policy:1 OUT=invnet1 SRC=192.168.2.10 DST=10.0.0.3 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: nat:POSTROUTING:policy:2 OUT=invnet1 SRC=192.168.2.10 DST=10.0.0.3 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=50797 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655029 ACK=0 WINDOW=65495 SYN 
---
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: raw:OUTPUT:policy:2 OUT=lo SRC=192.168.2.10 DST=192.168.2.10 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655030 ACK=0 WINDOW=0 RST 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: mangle:OUTPUT:policy:1 OUT=lo SRC=192.168.2.10 DST=192.168.2.10 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655030 ACK=0 WINDOW=0 RST 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: filter:OUTPUT:policy:2 OUT=lo SRC=192.168.2.10 DST=10.0.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655030 ACK=0 WINDOW=0 RST 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: security:OUTPUT:policy:1 OUT=lo SRC=192.168.2.10 DST=10.0.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655030 ACK=0 WINDOW=0 RST 
0 6 - 08/Dec/2022:19:41:04 +0100 TRACE: mangle:POSTROUTING:policy:1 OUT=invnet1 SRC=192.168.2.10 DST=10.0.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=34382 DPT=5140 SEQ=378655030 ACK=0 WINDOW=0 RST

The SYN sequence seems good to me. But it is followed by an RST sequence immediately