Connection Table Exhaustion on Game Servers: Tuning conntrack Against SYN and Flow Floods

The attack that doesn’t look like an attack

Most game server owners prepare for volumetric DDoS — a flood of gigabits that saturates the uplink. Modern attackers increasingly skip that. Instead they exhaust connection state. A few thousand short-lived TCP or UDP flows, opened and abandoned, can fill a server’s connection tracking table long before bandwidth becomes a problem. The result is a server that reports low CPU and plenty of free bandwidth while new players simply cannot connect, and existing connections time out with no obvious cause. This is connection-table exhaustion, and it is one of the most under-defended failure modes in game hosting.

This article covers how to detect it, how to size your connection tracking for realistic player loads, and how to harden a Linux host so a moderate flood degrades gracefully instead of taking the whole box offline. It assumes a dedicated or bare-metal machine where you have kernel-level control.

How connection tracking works and where it breaks

The Linux kernel’s netfilter subsystem — the thing behind iptables/nftables — tracks every network flow it sees when stateful rules or NAT are in use. Each tracked connection consumes memory in the conntrack table. Two limits apply: the maximum number of entries (nf_conntrack_max) and the rate at which new entries can be created before the kernel gives up. When either is exceeded, the kernel drops new packets and logs nf_conntrack: table full, dropping packet.

Game traffic makes this worse than typical web traffic. UDP is stateless from the application’s perspective, but conntrack still tracks UDP “flows” with short timeouts, and game servers generate many short flows as players join, leave, and reconnect. A busy server legitimately holds thousands of entries at idle-plus-load. A flood can multiply that by ten in seconds.

Detection: the numbers to watch

Watch three signals together. Any one alone is ambiguous; together they identify table exhaustion.

# Current entries vs maximum
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

# Watch it during a suspected event
watch -n1 'sysctl -n net.netfilter.nf_conntrack_count'

# Kernel drops and invalid packets
dmesg | grep -i "conntrack\|nf_conntrack"
nstat -az | grep -i "TcpExtListenDrops\|TcpExtTCPAbortOnMemory"

If nf_conntrack_count creeps toward the maximum during “connection refused” complaints, and dmesg shows table-full messages, you have your answer. If instead the count is low but ListenDrops is climbing, the issue is application-level accept queue overflow, a different problem with a different fix.

Sizing the table for real player loads

A common mistake is leaving the default table size, which on many distributions is sized for a workstation, not a game host. A rough formula: each active player can hold 5–20 tracked flows depending on how the game handles voice, matchmaking, and telemetry. Add headroom for legitimate reconnects and a flood buffer multiplier of 4–8x.

# Example: raise the table and shorten idle timeouts
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 60

# Keep the hash table large enough to avoid collisions
net.netfilter.nf_conntrack_buckets = 262144

Persist these in /etc/sysctl.d/ so they survive reboots and game server restarts. Shorter timeouts for finished connections are the cheapest defense: they free table entries faster under load without affecting active gameplay.

Hardening the host against connection floods

Sizing the table is necessary but not sufficient. Add rate limits and SYN-flood protections so that a flood is absorbed rather than allowed to fill the table. A minimal iptables/nftables posture:

• Enable SYN cookies so half-open connections do not consume table space: net.ipv4.tcp_syncookies = 1, plus a sensible tcp_max_syn_backlog.
• Rate-limit new connections per source with hashlimit or nftables limit, allowing bursts but capping sustained attempts.
• Drop invalid states early — conntrack --ctstate INVALID -j DROP — so malformed packets never reach the game process.
• Consider NOTRACK for known game UDP ports where you do not need stateful inspection, removing those flows from the table entirely.

Layered defense: on-box and upstream

On-box tuning handles connection exhaustion, but it cannot absorb volumetric floods — those must be scrubbed upstream before they reach your uplink. The best posture combines both: an always-on or on-demand scrubbing layer at the network edge, plus kernel hardening on the host so that what slips through degrades gracefully. Ask your provider whether filtering is always-on, how much it adds in latency when active, and whether it protects protocol-specific game ports. A host that treats DDoS protection as a configurable, documented service is far easier to defend with than one that just says “we have it.”

The combination you want is a well-managed network edge and a host you fully control underneath. If you are evaluating options, review dedicated and bare-metal hosting with DDoS mitigation and confirm the mitigation model matches this layered approach before you commit. You can also pair this with the kernel and network tuning guidance for the host itself.

A hardening checklist

1. Baseline: know your normal nf_conntrack_count at peak.
2. Raise nf_conntrack_max and buckets to match player count and headroom.
3. Shorten UDP/TCP idle timeouts.
4. Enable SYN cookies and rate-limit new connections per source.
5. Drop INVALID state packets at the edge of the host.
6. Use NOTRACK for trusted game ports you do not need to inspect.
7. Layer upstream scrubbing on top for volumetric attacks.

Connection-table exhaustion is quiet, cheap to trigger, and easy to misdiagnose. With the right counters and a sized table, you turn a silent outage into a logged, survivable event.