Linux Server DDoS Hardening: Every sysctl, conntrack and nftables Setting

This guide answers one question: how to harden the Linux server itself, with no device in front of it, against the state-exhausting part of a DDoS attack. We are not discussing any intermediate protection layer; we are discussing the server’s own kernel settings.

Most writing on the subject is a sysctl list: twenty lines, no explanation. The list appears to work because the values are harmless. On the day of an attack, a team that does not know what each line does also does not know which counter to read. Below, every parameter comes with its value, its function and its verification command. What an attack can exhaust on the host is finite and countable; each resource gets its own section.

Appliance	Resource	Symptom	Verification command
SYN / accept queue	New connections time out silently	nstat -az | grep -i listen
conntrack table	dmesg: 'nf_conntrack: table full, dropping packet'	conntrack -C; conntrack -S
File descriptors	accept() returns EMFILE; port listens but accepts nobody	cat /proc/<pid>/limits; cat /proc/net/sockstat
TIME_WAIT / orphan sockets	Ephemeral ports or memory exhaust; 'Out of socket memory' in log	ss -tan state time-wait | wc -l; cat /proc/net/sockstat
Socket buffers	Legitimate traffic drops under load	netstat -s | grep -i prune; cat /proc/net/softnet_stat

All five rows are this article's subject and all five are measured on the host. Packet rate saturating the CPU (softirq) is a separate layer, covered in the network-stack guide.

0. Baseline first: you cannot tune what you have not measured

Before hardening, record a normal week’s numbers. Every threshold below is chosen against them.

# Snapshot of sockets (established, syn-recv, time-wait breakdown)
ss -s

# Socket counters: tcp inuse / orphan / tw, tcp mem
cat /proc/net/sockstat

# conntrack occupancy and ceiling
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Cumulative TCP events — handshake, queue drops
nstat -az | grep -Ei 'syn|listen|drop|overflow'

# To watch over time
watch -n1 'cat /proc/net/sockstat; echo; nstat | grep -Ei "listen|syncookie"'

The output of these five commands is your baseline. A week later you run the same commands and see what changed. The trouble with a copied threshold is that it skips this step.

1. The SYN side: there are two queues, not one

The kernel keeps two queues per listening socket. The half-open queue holds connections whose handshake is incomplete (SYN_RECV); its size is tcp_max_syn_backlog. The accept queue holds completed handshakes waiting for accept(); its ceiling is the smaller of somaxconn and the application’s listen() backlog.

Because of this split, raising only the sysctl often changes nothing: if the application listens with 128, what you wrote to the kernel is irrelevant.

# /etc/sysctl.d/90-ddos.conf — SYN layer
net.ipv4.tcp_syncookies = 1          # cookie once the queue overflows; modern default
net.ipv4.tcp_max_syn_backlog = 8192  # half-open queue size
net.core.somaxconn = 8192            # accept queue ceiling
net.ipv4.tcp_synack_retries = 2      # shortens the life of an unanswered SYN_RECV
net.ipv4.tcp_syn_retries = 3         # for outbound connections; client side

Raise the application backlog at the same time, or somaxconn alone does nothing:

# nginx: listen backlog aligns to somaxconn
listen 443 ssl backlog=8192;

Verification. Read sockets in SYN_RECV and the queue drops:

# How many connections are half-open right now?
ss -tan state syn-recv | wc -l

# Accept queue occupancy: Recv-Q is the current backlog, Send-Q the ceiling
ss -ltn

# Queue overflow counters — if these leave zero, the accept chain is narrow
nstat -az | grep -E 'TcpExtListenDrops|TcpExtListenOverflows'

# Did SYN cookies actually engage?
nstat -az | grep -i syncookie

If SyncookiesSent is above zero, your queue has overflowed at least once; that is the signal to raise the backlog or engage an upstream layer. While SYN cookies are active and TCP timestamps are off, window scaling and SACK are lost for those connections; so rather than relying on the cookie, aim for a queue that never overflows.

tcp_abort_on_overflow sends an RST instead of silently dropping when the accept queue overflows. The default (0) is a silent drop and is usually right — it leaves the client room to retry. If you have a load balancer with its own retry logic behind it, 1 is a more honest signal:

sysctl -w net.ipv4.tcp_abort_on_overflow=1   # only with a reason

2. conntrack: once the table is full, no new flow gets in

Netfilter connection tracking keeps an entry per flow. When the table fills, the kernel drops the new flow’s packet. The symptom is insidious: existing sessions live, everyone new stays out, and the graph says “traffic normal”.

# Current occupancy and ceiling
conntrack -C
cat /proc/sys/net/netfilter/nf_conntrack_max

# How many flows in which state? (a SYN_SENT flood shows up here)
conntrack -L 2>/dev/null | awk '{print $4}' | sort | uniq -c | sort -rn

# Have drops started?
dmesg | grep -i 'nf_conntrack: table full'
conntrack -S | grep -Eo 'drop=[0-9]+'

There are three levers. Size, lifetime and not tracking at all.

# /etc/sysctl.d/90-ddos.conf — conntrack
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 3600   # default 432000 (5 days)
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 30        # clear half-open fast
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_loose = 0                    # do not track one-way flows

The hash table size (buckets) is set from the module parameter, not sysctl; keeping it at least an eighth of the ceiling reduces collisions:

echo 262144 > /sys/module/nf_conntrack/parameters/hashsize
# persistent: in /etc/modprobe.d/nf_conntrack.conf
#   options nf_conntrack hashsize=262144

The third lever keeps stateless high-volume traffic entirely out of the table. The nftables raw chain:

table inet raw {
  chain prerouting {
    type filter hook prerouting priority raw; policy accept;
    udp dport 53 notrack
    tcp dport 53 notrack
  }
}

A stateless service like authoritative DNS becomes immune to a conntrack flood with this rule. The cost is plain: ct state rules do not apply to that traffic.

Alarm threshold: wire occupancy at seventy percent of the ceiling into monitoring.

awk -v max="$(cat /proc/sys/net/netfilter/nf_conntrack_max)" \
    -v cur="$(cat /proc/sys/net/netfilter/nf_conntrack_count)" \
    'BEGIN{ printf "conntrack %d/%d = %.0f%%\n", cur, max, 100*cur/max }'

3. File descriptors: the shortest link in the chain wins

Every open connection is a file descriptor. When they run out, accept() returns EMFILE; the service is up, the port listens, but it accepts nobody. Slow-connection attacks target exactly this.

The limit is a chain, and the lowest link governs:

# System-wide ceiling
cat /proc/sys/fs/file-max
# Per-process absolute ceiling
cat /proc/sys/fs/nr_open
# How many descriptors are open now / what is the ceiling?
cat /proc/sys/fs/file-nr

# /etc/sysctl.d/90-ddos.conf — system wide
fs.file-max = 2097152
fs.nr_open = 1048576

The real bottleneck is almost always the service unit limit, because it stays low on most distributions. The durable fix is a systemd drop-in; do not fight with ulimit, which describes your shell, not the service:

# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=262144

systemctl daemon-reload
systemctl restart nginx

# The service's REAL limit (not the shell's):
systemctl show nginx -p LimitNOFILE
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'

# Overall socket usage:
cat /proc/net/sockstat   # sockets: used ...; TCP: inuse ...

4. TIME_WAIT and orphan sockets: the silent exhaustion

A short-lived connection flood exhausts two resources through TIME_WAIT buildup and orphan sockets: the ephemeral port range and TCP socket memory. The symptom is a TCP: out of memory or Out of socket memory line in dmesg.

# State distribution
ss -tan state time-wait | wc -l
ss -tan state fin-wait-1 | wc -l

# Orphan / tw counts and tcp mem pressure
cat /proc/net/sockstat
# tcp_mem: low pressure high  (third number is the hard ceiling in pages)
cat /proc/sys/net/ipv4/tcp_mem

# /etc/sysctl.d/90-ddos.conf — TIME_WAIT / orphan
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_tw_reuse = 1            # safe for outbound connections
net.ipv4.ip_local_port_range = 1024 65535

tcp_tw_reuse is safe only for outbound connections and while timestamps are on; do not use the old tcp_tw_recycle, which breaks clients behind NAT and has been removed from modern kernels anyway. When tcp_max_orphans is exceeded, the kernel closes the excess with an RST and writes too many orphaned sockets to dmesg; that is defensive behaviour, so keep the limit above your legitimate load.

5. Socket buffers and the receive queue

This layer does not stop an attack; it reduces the crushing of legitimate traffic during one. If the queue on the receive path overflows, the good packet drops with the bad.

# /etc/sysctl.d/90-ddos.conf — buffers and receive queue
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 16384      # NIC → kernel queue
net.core.netdev_budget = 600             # packets processed per softirq
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Verification — did the receive path drop, did the softirq budget run out:

# softnet_stat: col 1 processed, col 2 DROPPED, col 3 left by budget/time-limit
cat /proc/net/softnet_stat

# TCP-side prune / collapse (a sign of buffer pressure)
netstat -s | grep -Ei 'prune|collapse|out of'
nstat -az | grep -Ei 'TcpExtTCPRcvQDrop|PruneCalled'

If the second softnet_stat column climbs, raise netdev_max_backlog; if the third climbs, raise netdev_budget. A chronic rise in the third column is also the early sign that packet rate is starting to saturate the CPU — that line is this article’s boundary and passes to the network-stack tuning guide.

6. Spoofed source, ICMP and the routing surface

A small but necessary group. Against spoofed source addresses and the reflection surface:

# /etc/sysctl.d/90-ddos.conf — anti-spoof and ICMP
net.ipv4.conf.all.rp_filter = 1              # reverse-path filtering (strict)
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1     # close the smurf surface
net.ipv4.icmp_ratelimit = 1000
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_timestamps = 1                  # required for tw_reuse and PAWS

rp_filter=1 (strict mode) is correct only on a single-homed server. On an asymmetrically routed machine that receives traffic over more than one path, strict mode drops legitimate packets; there you choose 2 (loose mode). The wrong mode can cause an outage with no attack at all; do not set 1 without knowing which mode you need:

# Effective rp_filter per interface (the max of 'all' and the interface applies)
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo "$i = $(cat $i)"; done

# How many packets did rp_filter drop?
nstat -az | grep -i 'IPReversePathFilter'

7. Per-source rate limiting with nftables

Enlarging queues is the passive half. The active half limits a single source’s share, inside the kernel. An nftables dynamic set does this per source address:

table inet filter {
  set flood4 { type ipv4_addr; flags dynamic; timeout 60s; }
  set flood6 { type ipv6_addr; flags dynamic; timeout 60s; }

  chain input {
    type filter hook input priority filter; policy accept;

    ct state established,related accept
    ct state invalid drop                 # drop half/broken flows

    # Per-source new-connection rate — IPv4 and IPv6 separately
    tcp dport { 80, 443 } ct state new \
      add @flood4 { ip  saddr limit rate over 30/second } drop
    tcp dport { 80, 443 } ct state new \
      add @flood6 { ip6 saddr limit rate over 30/second } drop
  }
}

Verification — which sources hit the limit, how many packets the rule dropped:

# Sources that hit the limit and entered the set
nft list set inet filter flood4

# Rule counters (if you add 'counter' to the rule)
nft list ruleset | grep -A2 flood

# How much did ct state invalid drop — a counted rule example
nft add rule inet filter input ct state invalid counter drop

If you skip the IPv6 set, half the defence is missing. The threshold comes from your measurement: for a NAT-fronted but legitimate crowd, 30 may be too low; for an enterprise API it may be high. Run the rule with counter instead of drop for the first week, look at who gets caught, and switch to drop once you are sure it does not cut legitimate traffic.

There is also synproxy — the kernel takes the handshake and only completed connections reach the service behind — but its interaction with asymmetric routing and conntrack settings is delicate; do not put it in production without validating in a lab.

The honest limit of per-source rate limiting: it works against a flood from few addresses. A flood spread across thousands of addresses, each staying under the per-address threshold, can fill your circuit in aggregate, and this rule does not see it. We covered the prefix-scale form of the same logic in the carpet-bombing guide.

8. Application-layer connection behaviour

A connection that passes the kernel queue reaches the application, where a waiting connection also holds a resource. This article is about the kernel, but two sysctls affect application behaviour directly:

net.ipv4.tcp_keepalive_time = 300     # detect a dead connection early
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3

Slow-connection attacks (opening a connection and dribbling data) are not fully solved at the kernel level; the real defence is the web server’s own timeouts. We handle that per server: the nginx hardening and Apache hardening guides give those timeouts directive by directive.

The full file and one-shot load

The sections combined:

sudo tee /etc/sysctl.d/90-ddos.conf >/dev/null <<'EOF'
# --- SYN ---
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.somaxconn = 8192
net.ipv4.tcp_synack_retries = 2
# --- conntrack ---
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 30
# --- file descriptors ---
fs.file-max = 2097152
fs.nr_open = 1048576
# --- TIME_WAIT / orphan ---
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_tw_reuse = 1
# --- buffers ---
net.core.netdev_max_backlog = 16384
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# --- anti-spoof / ICMP ---
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_timestamps = 1
EOF

sudo sysctl --system
# verify it applied
sudo sysctl -a 2>/dev/null | grep -E 'somaxconn|syncookies|conntrack_max|file-max|rp_filter'

None of the values requires a reboot and all are reversible. But all are starting points, pulled up or down against your own baseline. A guide that gives you exact numbers is misleading, because the right number depends on your session count, your source distribution and your hardware.

The counters to wire into monitoring

Hardening is not felt, it is read. Wire these six signals into the alarm system:

# 1. SYN queue drops
nstat -az | grep -E 'ListenDrops|ListenOverflows'
# 2. SYN cookie use (an overflow indicator)
nstat -az | grep -i syncookiessent
# 3. conntrack occupancy ratio (>70% warn)
cat /proc/sys/net/netfilter/nf_conntrack_count
# 4. file descriptor use
cat /proc/sys/fs/file-nr
# 5. receive-path drops
awk '{s+=$2} END{print "softnet drops:", s}' /proc/net/softnet_stat
# 6. socket memory pressure
grep -E 'TCP:' /proc/net/sockstat

These six lines answer, in under a minute during an attack, the question “is the host narrowing or is the circuit filling”.

The honest limit of this layer

Everything up to here reduces to one sentence: the server is defensible against attacks that exhaust its own state. Beyond that there are two cases, and neither is solved on the host.

The first is circuit saturation. Volume above your access circuit never reaches the kernel; sysctl cannot drop a packet it does not see. Above this line the subject is no longer the server but the network architecture.

The second is packet rate. Before the circuit fills, packets per second can saturate the kernel’s interrupt-handling capacity; the symptom is the third softnet_stat column and CPU going to softirq. That layer’s settings — interrupt distribution, driver queues, RSS/RPS, the receive path — are a separate world, covered command by command in the network-stack tuning guide.

Above these two thresholds host hardening does not become useless; on the contrary, by raising the floor it is what keeps the server standing until an upper layer engages and after it does. But it is not sufficient on its own, and treating it as if it were produces false confidence.

Order of application

1. Record the baseline (Section 0). It takes a week of patience; it is the most expensive step.
2. Write 90-ddos.conf, load with sysctl --system, verify with grep.
3. Align the application backlog (listen ... backlog=) with somaxconn.
4. Add a LimitNOFILE drop-in to the service units, verify with systemctl show.
5. Tune the conntrack size/lifetime; take stateless services out of the table with notrack.
6. Run the nftables rate limit in counter mode for a week, then switch to drop.
7. Wire the six counters into monitoring as alarms.

None of the seven steps needs a reboot and all are reversible.