TCP RST,ACK between two VM

J

jeannotp

Member
Dec 26, 2015
10
3
23
41
Hello,

I have a problem for many years, and I suddenly decided to resolve it...
I have it with different hardware, and I had it also with PVE 4, I think.

I have servers with only one IPv4 so I do NAT.

I have for the main interface:


Code: 
auto vmbr0
iface vmbr0 inet static
        address 10.10.8.1/21
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.8.0/21' -o eno3 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.8.0/21' -o eno3 -j MASQUERADE
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
        post-up   /root/nat-rules

Let's take two VM:
  • VM 106, FreeBSD with nginx acting as reverse proxy, with IP 10.10.10.106 (net:virtio)
  • VM 103, Linux with a Web server, with IP 10.10.10.103 (net:virtio)
The incoming TCP 443 traffic is redirected to 10.10.10.106:443
Nginx does proxy to 10.10.10.103:443 etc.

Everything is OK, but sometimes I have 502 errors (Nginx does not "find" 10.10.10.103).

It has been years. Since it's only personal traffic, I haven't cared much about... And I use also IPV6, everything is fine without NAT.

I ran tcpdump and saw that on the nginx VM, FreeBSD makes a TCP connection and the Linux answers with a RST, ACK.

Code: 
Internet Protocol Version 4, Src: 10.10.10.106, Dst: 10.10.10.103
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 60
    Identification: 0x0000 (0)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0x11d8 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.10.10.106
    Destination Address: 10.10.10.103
Transmission Control Protocol, Src Port: 37373, Dst Port: 443, Seq: 0, Len: 0
    Source Port: 37373
    Destination Port: 443
    [Stream index: 13]
    [Conversation completeness: Incomplete (37)]
    [TCP Segment Len: 0]
    Sequence Number: 0    (relative sequence number)
    Sequence Number (raw): 1858107831
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 0
    Acknowledgment number (raw): 0
    1010 .... = Header Length: 40 bytes (10)
    Flags: 0x002 (SYN)
    Window: 65535
    [Calculated window size: 65535]
    Checksum: 0x2913 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (20 bytes), Maximum segment size, No-Operation (NOP), Window scale, SACK permitted, Timestamps
    [Timestamps]

Code: 
Internet Protocol Version 4, Src: 10.10.10.103, Dst: 10.10.10.106
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 40
    Identification: 0x0000 (0)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0x11ec [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.10.10.103
    Destination Address: 10.10.10.106
Transmission Control Protocol, Src Port: 443, Dst Port: 37373, Seq: 1, Ack: 1, Len: 0
    Source Port: 443
    Destination Port: 37373
    [Stream index: 13]
    [Conversation completeness: Incomplete (37)]
    [TCP Segment Len: 0]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 0
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 1858107832
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x014 (RST, ACK)
    Window: 0
    [Calculated window size: 0]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x28ff [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]

It's on bare metal, the CPU use is very very low (with 4 or 12 cores 3Ghz+ ) and traffic is very very low too (with an average of 5 Mbps).
I had also the same problem with Nginx on Linux. I have it also with other web servers than VM 103.

I use the PVE firewall.

Would you have any idea?
 
I had a cron doing a curl every 10 minutes:
Code: 
$ grep HTTP debugwwwcurlv4|sort|uniq  -c
     42 HTTP/2 302
     24 HTTP/2 502

More than sometimes, in fact...
 
Some other details:

Code: 
# uname -a
Linux pve1 5.13.19-1-pve #1 SMP PVE 5.13.19-3 (Tue, 23 Nov 2021 13:31:19 +0100) x86_64 GNU/Linux

Code: 
# cat /etc/pve/firewall/103.fw
[OPTIONS]

log_level_out: debug
enable: 1
log_level_in: debug
policy_out: DROP
policy_in: DROP

[RULES]

OUT Web(ACCEPT) -log nolog
IN HTTP(ACCEPT) -source 10.10.10.106
IN HTTPS(ACCEPT) -log nolog

Code: 
# cat /etc/pve/firewall/106.fw
[OPTIONS]

log_level_out: debug
enable: 1
log_level_in: debug
policy_out: DROP

[RULES]

OUT HTTPS(ACCEPT)
IN HTTP(ACCEPT)
IN HTTPS(ACCEPT)
OUT HTTP(ACCEPT)


Code: 
# tcpdump -nS -i fwpr103p0
15:23:36.444060 ARP, Request who-has 10.10.10.105 tell 10.10.10.106, length 28
15:23:36.589866 IP 10.10.10.106.14410 > 10.10.10.103.443: Flags [S], seq 1290963469, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1394042321 ecr 0], length 0
15:23:41.629336 ARP, Request who-has 10.10.10.106 tell 10.10.10.103, length 28
15:23:41.629565 ARP, Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28z

# tcpdump -nS -i fwln103i0
15:23:36.589881 IP 10.10.10.106.14410 > 10.10.10.103.443: Flags [S], seq 1290963469, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1394042321 ecr 0], length 0
15:23:36.589977 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [R.], seq 0, ack 1290963470, win 0, length 0
15:23:36.589986 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [R.], seq 0, ack 1290963470, win 0, length 0
15:23:41.629351 ARP, Request who-has 10.10.10.106 tell 10.10.10.103, length 28
15:23:41.629568 ARP, Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28

# tcpdump -nS -i vmbr0 host 10.10.10.106
15:23:36.444096 ARP, Request who-has 10.10.10.105 tell 10.10.10.106, length 28
15:23:36.589903 IP 10.10.10.106.14410 > 10.10.10.103.443: Flags [S], seq 1290963469, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1394042321 ecr 0], length 0
15:23:41.629328 ARP, Request who-has 10.10.10.106 tell 10.10.10.103, length 28
15:23:41.629572 ARP, Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28

# tcpdump -nS -i fwbr103i0
15:23:36.444118 ARP, Request who-has 10.10.10.105 tell 10.10.10.106, length 28
15:23:36.589918 IP 10.10.10.106.14410 > 10.10.10.103.443: Flags [S], seq 1290963469, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1394042321 ecr 0], length 0
15:23:36.590235 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304076930 ecr 1394042321,nop,wscale 7], length 0
15:23:37.597353 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304077937 ecr 1394042321,nop,wscale 7], length 0
15:23:39.613320 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304079953 ecr 1394042321,nop,wscale 7], length 0
15:23:41.629332 ARP, Request who-has 10.10.10.106 tell 10.10.10.103, length 28
15:23:41.629585 ARP, Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28
15:23:43.677328 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304084017 ecr 1394042321,nop,wscale 7], length 0
15:23:51.869361 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304092209 ecr 1394042321,nop,wscale 7], length 0

# tcpdump -nS -i tap103i0
15:23:36.444105 ARP, Request who-has 10.10.10.105 tell 10.10.10.106, length 28
15:23:36.589912 IP 10.10.10.106.14410 > 10.10.10.103.443: Flags [S], seq 1290963469, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1394042321 ecr 0], length 0
15:23:36.590203 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304076930 ecr 1394042321,nop,wscale 7], length 0
15:23:37.597327 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304077937 ecr 1394042321,nop,wscale 7], length 0
15:23:39.613293 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304079953 ecr 1394042321,nop,wscale 7], length 0
15:23:41.629315 ARP, Request who-has 10.10.10.106 tell 10.10.10.103, length 28
15:23:41.629578 ARP, Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28
15:23:43.677296 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304084017 ecr 1394042321,nop,wscale 7], length 0
15:23:51.869336 IP 10.10.10.103.443 > 10.10.10.106.14410: Flags [S.], seq 4290038972, ack 1290963470, win 65160, options [mss 1460,sackOK,TS val 304092209 ecr 1394042321,nop,wscale 7], length 0
 
After that I changed the NAT rules but I don't know if it changes something.
From:
Code: 
DNAT       tcp  --  eno3   *       0.0.0.0/0            0.0.0.0/0       tcp dpt:80 /* HTTP nginx */ to:10.10.10.106:80
DNAT       tcp  --  eno3   *       0.0.0.0/0            0.0.0.0/0       tcp dpt:443 /* HTTPS nginx */ to:10.10.10.106:443

To:
Code: 
root@pve1:~# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 7930 packets, 556K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   22  1064 DNAT       tcp  --  eno3   *       0.0.0.0/0            xxx.154.yy.175       tcp dpt:80 /* HTTP nginx */ to:10.10.10.106:80
   67  3976 DNAT       tcp  --  eno3   *       0.0.0.0/0            xxx.154.yy.175       tcp dpt:443 /* HTTPS nginx */ to:10.10.10.106:443

Chain INPUT (policy ACCEPT 872 packets, 54726 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3097 packets, 198K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 6859 packets, 463K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 423K   31M MASQUERADE  all  --  *      eno3    10.10.8.0/21         0.0.0.0/0
 
And then:
Code: 
# tcpdump -nSe -i vmbr0  host 10.10.10.103
19:05:42.491957 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype IPv4 (0x0800), length 74: 10.10.10.106.42036 > 10.10.10.103.443: Flags [S], seq 3043812723, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 494310551 ecr 0], length 0
19:05:42.492110 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 54: 10.10.10.103.443 > 10.10.10.106.42036: Flags [R.], seq 0, ack 3043812724, win 0, length 0
19:05:42.492120 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 54: 10.10.10.103.443 > 10.10.10.106.42036: Flags [R.], seq 0, ack 3043812724, win 0, length 0

# tcpdump -nSe -i fwln103i0
19:05:02.948706 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:08.077013 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:24.079225 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:40.080996 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:42.491993 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype IPv4 (0x0800), length 74: 10.10.10.106.42036 > 10.10.10.103.443: Flags [S], seq 3043812723, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 494310551 ecr 0], length 0
19:05:47.709564 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype ARP (0x0806), length 42: Request who-has 10.10.10.106 tell 10.10.10.103, length 28
19:05:47.709818 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype ARP (0x0806), length 42: Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28

# tcpdump -nSe -i fwpr103p0
19:05:24.079182 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:40.080949 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:42.491939 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype IPv4 (0x0800), length 74: 10.10.10.106.42036 > 10.10.10.103.443: Flags [S], seq 3043812723, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 494310551 ecr 0], length 0

# tcpdump -nSe -i tap103i0
19:05:24.079235 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:40.081006 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:42.492023 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype IPv4 (0x0800), length 74: 10.10.10.106.42036 > 10.10.10.103.443: Flags [S], seq 3043812723, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 494310551 ecr 0], length 0
19:05:42.492212 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win 65160, options [mss 1460,sackOK,TS val 317402831 ecr 494310551,nop,wscale 7], length 0
19:05:43.517608 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win 65160, options [mss 1460,sackOK,TS val 317403857 ecr 494310551,nop,wscale 7], length 0
19:05:45.533596 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win 65160, options [mss 1460,sackOK,TS val 317405873 ecr 494310551,nop,wscale 7], length 0
19:05:47.709551 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype ARP (0x0806), length 42: Request who-has 10.10.10.106 tell 10.10.10.103, length 28
19:05:47.709824 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype ARP (0x0806), length 42: Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28
19:05:48.473639 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:49.757617 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win 65160, options [mss 1460,sackOK,TS val 317410097 ecr 494310551,nop,wscale 7], length 0

# tcpdump -nSe -i fwbr103i0
19:05:02.948725 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:08.077029 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:24.079246 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:40.081017 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
19:05:42.492034 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype IPv4 (0x0800), length 74: 10.10.10.106.42036 > 10.10.10.103.443: Flags [S], seq 3043812723, win 65535, options
[mss 1460,nop,wscale 6,sackOK,TS val 494310551 ecr 0], length 0
19:05:42.492238 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win
 65160, options [mss 1460,sackOK,TS val 317402831 ecr 494310551,nop,wscale 7], length 0
19:05:43.517635 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win
 65160, options [mss 1460,sackOK,TS val 317403857 ecr 494310551,nop,wscale 7], length 0
19:05:45.533627 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype IPv4 (0x0800), length 74: 10.10.10.103.443 > 10.10.10.106.42036: Flags [S.], seq 360535944, ack 3043812724, win
 65160, options [mss 1460,sackOK,TS val 317405873 ecr 494310551,nop,wscale 7], length 0
19:05:47.709569 5a:ab:95:29:4a:7a > 46:2c:c5:c4:27:53, ethertype ARP (0x0806), length 42: Request who-has 10.10.10.106 tell 10.10.10.103, length 28
19:05:47.709833 46:2c:c5:c4:27:53 > 5a:ab:95:29:4a:7a, ethertype ARP (0x0806), length 42: Reply 10.10.10.106 is-at 46:2c:c5:c4:27:53, length 28
19:05:48.473649 1e:87:b4:23:af:09 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: fe80::1c87:b4ff:fe23:af09 > ff02::1: ICMP6, router advertisement, length 56
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back