TCP RST Packets dropped by pve-firewall

[gslongo]

Hi All,

We are experiencing a very strange issue or bug. Here is the case :

• Assume we have a service running on TCP port 12354
• Clients can communicate with it while running
• While service is down, clients recieved "Connection timed out" (no answer) even if OS send TCP RST packets :

[root@cluster-dev-1 ~]# tcpdump -ni any tcp port 12345
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
14:28:11.985896 IP 172.16.38.128.41470 > 172.18.0.20.italk: Flags , seq 906491996, win 29200, options [mss 1460,sackOK,TS val 3812309330 ecr 0,nop,wscale 7], length 0
14:28:11.985919 IP 172.18.0.20.italk > 172.16.38.128.41470: Flags [R.], seq 0, ack 906491997, win 0, length 0

However, these RST packets are dropped somewhere in PVE firewall.
On the VM options :

• Firewall > Options > Firewall = No , Has no effect
• Firewall > Options > * Policy = ACCEPT , Has no effect (even with NO rule in active for this VM)
• Hardware > Network Device >firewall=0 , allows packets RST to pass !

Any idea ?

Thank you !

Here is version informations :

~# pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.15.18-18-pve)
pve-manager: 5.4-11 (running version: 5.4-11/6df3d8d0)
pve-kernel-4.15: 5.4-6
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-53
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-39
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[gslongo]

Hi, Does anyone have any idea ?

Thank you !

 

[wolfgang]

Hi,

I would update your kernel and test it again.
The kernel you use is quite old and no basis to analyze the problem.
Also, what NIC do you use and how is your network settings (bond, bridge,...)

 

[spirit]

if you can reproduce, you can also tcpdump on fwbrX bridge, and vmbrX bridge, and see if packet is dropped somewhere.

(the fwbr bridge is create when you enable firewall on the vm nic)

 

[spirit]

also, can you try to change value of sysctl:

nf_conntrack_tcp_be_liberal
"If it's non-zero, we mark only out of window RST segments as INVALID"



(I have seen problem with tcp_be_liberal for a totally different problem, with with some connection marked as invalid, and all packet was dropped by the main conntrack rule)

 

[iBug]

A follow-up 4 years later: See my new thread #134423 and my bug report #4983.