How to disable nf_conntrack completely

pgolik

Member
Nov 2, 2010
3
0
21
Recently I'm having lots of problems on our small server, probably related to DDoS attacks. The logs fill with messages like

Code:
Mar  3 15:26:58 mothership kernel: nf_conntrack: table full, dropping packet.
Mar  3 15:26:58 mothership kernel: nf_conntrack: table full, dropping packet.
Mar  3 15:26:58 mothership kernel: nf_conntrack: table full, dropping packet.
Mar  3 15:26:58 mothership kernel: nf_conntrack: table full, dropping packet.
Mar  3 15:26:58 mothership kernel: nf_conntrack: table full, dropping packet.
Mar  3 15:27:03 mothership kernel: __ratelimit: 14222 callbacks suppressed
Mar  3 15:27:03 mothership kernel: nf_conntrack: table full, dropping packet.

And the server crashes hard (kernel panic, hard reset required). The admins of our network are helpless (it's an understaffed research intsitution with no professional IT personnel), they know there are some infected PCs on the network, but are unable to deal with the problem.
I do not use conntrack functionality in any of the containers, and I want to disable this module completely. I can't, however remove the nf_conntrack modules, as they are shown to be used by other modules, including vzrst.

Code:
lsmod|grep conntrack
nf_conntrack_ipv4       9978  2 nf_nat
nf_defrag_ipv4          1531  1 nf_conntrack_ipv4
nf_conntrack           79944  6 xt_state,nf_conntrack_ipv4,vzcpt,nf_nat,vzrst,xt_NOTRACK

I do not have conntrack specified in vz.conf

Code:
~# cat /etc/vz/vz.conf|grep IPTABLES
IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length"

and I did not configure iptables on the proxmox server (only installed fail2ban).

I tried to disable tracking by doing
Code:
iptables -t raw -A PREROUTING -j NOTRACK
iptables -t raw -A OUTPUT -j NOTRACK
this helped a bit (the server stayed up a few days longer), but the problem returned.

How do I get rid of conntrack for good to keep my server running?
BTW, this is on pve-manager/3.1-21/93bf03d4 (running kernel: 2.6.32-26-pve)
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
17,137
532
133
Austria
www.proxmox.com
You need to increase netfilter.nf_conntrack_max, for example:

# sysctl -w net.netfilter.nf_conntrack_max=200000

to avoid this kind of messages.
 

thheo

Active Member
Nov 17, 2013
130
1
38
Bucharest
I doubt that your server crashes because of conntrack table filling up.. Better try to find out what is the panic message you see about..
 

pgolik

Member
Nov 2, 2010
3
0
21
I did try increasing that value, and also decreasing net.netfilter.nf_conntrack_generic_timeout and net.netfilter.nf_conntrack_tcp_timeout_established, but it is only a temporary solution and doesn't prevent crashes. From what I've read, it is best to disable conntrack if it is not to be used. Does proxmox need conntrack to function (as none of my CTs and VMs do), assuming that I'm only doing brdged networking, no NAT.
And I do believe the crashes are related to the conntrack table filling up. It has been observed elsewhere, and there is 100% correlation - first the network slows down, the logs fill with the nf_conntrack: table full messages (thousands of them per minute), and then the crash (which leaves no trace in the logs, only on the hung console). This is a known DDoS symptom.
So, the main question is: is conntrack required for proxmox, and if no, how do I remove it?
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,858
699
133
www.odiso.com
do you use proxmox kernel ? because it should be disabled for bridge by default

you can try to add in /etc/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0


then

#sysctl -p /etc/sysctl.conf
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
17,137
532
133
Austria
www.proxmox.com
Please use the pve firewall GUI to change that value (Node/Firewall/Options/nf_conntrack_max
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!