SYN Proxy and SYN Cookies

A TCP server that receives a SYN allocates something and waits. That allocation is the point of the attack: send SYNs that never complete, from addresses that do not answer, and the table of half-open connections fills with entries that will never become sessions RFC 4987.

The insight behind both defences on this page is the same. Refuse to allocate anything until the client has proved it is really there.

The two approaches

Appliance	Holds state before proof	TCP option fidelity	Where it usually lives
Unprotected server	Yes — one entry per SYN	Full	The server itself
SYN cookies	No — state encoded in the sequence number	Reduced; some options cannot be carried	Server kernel, as a fallback under pressure
SYN proxy	On the protecting device, not the server	Full, if the proxy is faithful	Inline mitigation device or load balancer

The first column is the whole defence. Everything else is a consequence of refusing to allocate anything until the client has answered.

SYN cookies

Instead of storing the half-open connection, the server encodes what it would have stored into the initial sequence number it sends back. If the client is real it answers, and the answer carries that number back — from which the server reconstructs the state and creates the session. If the client never existed, nothing was allocated and nothing needs cleaning up.

The cost is fidelity. Some options negotiated in the original SYN cannot be reconstructed from a sequence number, and window scaling and selective acknowledgement are the ones that matter RFC 4987. Connections established under cookies can therefore perform worse than normal — which is exactly the right trade during an attack and the wrong default outside one, and why the mechanism engages under pressure rather than always.

SYN proxy

A device in front of the server completes the handshake itself. Only when the client answers does the device establish a session to the server — either by opening a new connection or by splicing the flow. The server never sees a half-open connection at all.

The fidelity is preserved because the proxy is a full participant rather than reconstructing from a hash. The cost is architectural: something is in the path, holding state of its own, with its own capacity limit and its own failure modes.

What they work against

SYN floods, and the family of attacks that consume connection-establishment resources rather than bandwidth. This is the class that most often surprises people who sized their defence in gigabits — a modest packet rate of well-crafted SYNs can exhaust a firewall rated for tens of gigabits, because the constraint is table entries rather than throughput.

What they do not work against

Volume. The packets still arrive. If the flood saturates the circuit, nothing the server or an inline device does with those packets matters.

Completed-handshake attacks. An attacker who completes the handshake and then sends slow or expensive requests has passed the proof these mechanisms demand. That is the application-layer problem, answered by request timeouts, concurrency limits and rate limiting rather than here.

Attacks on the proxy itself. A SYN proxy holds state for every handshake it completes, so an attacker who completes handshakes and abandons them has moved the exhaustion target rather than escaping it.

Operational risk

Cookies engaging silently. Most kernels enable cookies as a threshold-triggered fallback, so the first sign is often a performance complaint rather than an alert. Monitor the counter, not just the symptom.

Proxy capacity is a real ceiling. The device is now the connection-establishment bottleneck. Its stated capacity should be verified in connections per second at your traffic mix rather than read from a datasheet.

Splicing subtleties. A proxy that opens a separate connection to the server changes what the server sees as the client address unless the original is preserved. Applications that log or authorise on source address behave differently as a result.

Interaction with anycast and asymmetric paths. A proxy needs both directions of the flow. Designs where the return path bypasses the device break it in ways that look like random connection failures.

False-positive risk

Low for cookies and moderate for proxies, and both mostly show up as degradation rather than blocking. Cookies do not reject legitimate clients; they serve them with reduced options. A proxy can reject clients whose stacks behave unusually, and aggressive per-source connection limits layered on top of it are where genuine false positives appear.

The number to measure is connection success rate for legitimate traffic while the defence is engaged, which is a specific case of the goodput measurement the proof-of-concept methodology sets out.

How to verify

1. Flood SYNs that never complete and confirm the server’s half-open count stays flat.
2. Establish legitimate connections through the same path during the flood and measure success rate and latency.
3. Check whether window scaling and SACK survive — that tells you whether you are proxying or falling back to cookies.
4. Confirm what the server logs as the client address.
5. Push the proxy to its connection-establishment ceiling deliberately and record what happens at the limit.

Sources