TLS Handshake and Renegotiation Attacks

Every TLS session begins with a handshake, and the handshake is not symmetric. The client performs some work; the server performs more, including operations chosen precisely because they are hard to reverse.

That asymmetry is a feature of the cryptography and a liability under load. A client that requests handshakes as fast as it can, and never does anything useful afterwards, converts a small budget into a large one.

At a glance

Appliance	Variant	What the client sends	What the server spends
Renegotiation flood	Repeated renegotiation requests on one connection	A full key exchange each time
Handshake flood	Many new connections, each starting a handshake	A key exchange per connection, plus connection state
Abandoned handshakes	Starts a handshake and stops	Partial state held until a timeout
Expensive parameter selection	Prefers the costliest supported option	More computation per handshake than necessary

Only the first row depends on renegotiation. Removing renegotiation therefore closes one row and leaves three, which is why version alone is not a defence.

What it exhausts

Cryptographic capacity: the CPU or accelerator budget available for key exchange. This is a distinct resource from bandwidth, connection state and application capacity, and it is frequently unmonitored.

Connection state, in the handshake-flood variant, since each new connection needs an entry before any handshake begins.

Partial handshake state, in the abandoned variant: a server that has begun a handshake holds context until a timeout, which is the same shape of problem as a SYN flood one layer higher.

Bit rate remains modest throughout. The traffic is small and the damage is computational, which is why a bandwidth graph shows a quiet afternoon.

How it differs from similar attacks

An HTTP flood is expensive because of what the request asks the application to do. This is expensive before any request exists — the connection never reaches the point of carrying one.

The practical consequence is that request-rate limiting, WAF rules and application caching are all irrelevant here. They operate on something that has not happened yet.

The historical name for the renegotiation variant is a stress tool from around 2011 that demonstrated a single client could occupy a considerably more powerful server. RFC 5746 is often cited as the fix, and it addressed a security flaw in renegotiation rather than its computational cost — a conflation worth avoiding when reading older material.

Observable telemetry

• Handshakes per second rising, against a flat or falling completed-request rate. The most diagnostic pair in this attack.
• CPU or crypto-accelerator utilisation rising with no corresponding application load.
• Handshake failure and abandonment rates rising.
• Session resumption ratio falling, since attack clients do not resume.
• Connection duration distribution collapsing toward very short values.

Most estates monitor none of these. Handshake rate in particular is available from every serious termination point and is graphed almost nowhere.

Common false positives

Clients that never resume. Some embedded and scripted clients open a fresh session every time by design, producing a genuinely high handshake-to-request ratio.

Mobile networks. Address changes and connection churn produce more handshakes per user than a fixed-line population.

A resumption cache change. Reducing a session cache or shortening a ticket lifetime raises handshake rate estate-wide overnight, which will be read as an attack by anyone who was not told.

Mitigation by layer

Disable client-initiated renegotiation wherever the protocol version still permits it. On modern versions there is nothing to disable, which is one good reason to be on them.

Limit handshakes per connection and per source. This is the control that addresses the mechanism rather than a variant, and it survives whatever the next variant turns out to be.

Terminate TLS somewhere sized for it, separate from application servers. A proxy or load balancer concentrates the cost where it can be measured and bounded — the tuning surface for one common case is in HAProxy hardening.

Prefer efficient parameters and do not offer costlier options than you need, so an attacker cannot select the expensive path.

Enable session resumption properly. It lowers legitimate cost substantially, which widens the gap between normal traffic and attack traffic and makes the ratio easier to alert on.

Timeout abandoned handshakes aggressively. Partial state should not persist, and the default is often generous.

Operational pitfalls

Assuming a modern protocol version is sufficient. It closes one of four rows in the table above.

Not monitoring handshake rate. Without it the attack is invisible until CPU saturates, at which point the cause is not obvious.

Rate-limiting connections instead of handshakes. The renegotiation variant uses very few connections, so a connection limit does nothing.

Safe validation

Drive handshakes at increasing rate from a small number of clients and record CPU or accelerator utilisation against handshake rate, then find the rate at which legitimate clients begin failing to connect.

Run it twice: once with session resumption available and once without. The difference is the value of your resumption configuration, and it is usually large enough to change a procurement decision about how much termination capacity is required.

Isolation and authorisation requirements are in the test lab guide and authorised testing.