DDoS Incident Response Runbook

The purpose of a runbook is not to tell an experienced engineer how to read a graph. It is to remove the decisions that are hard to make well while a service is down and a director is asking for an estimate — which of two costly options to take, who is allowed to authorise it, and what has to be captured before it disappears.

Everything below assumes the mitigation itself already exists. A runbook does not compensate for an absent defence; it makes an existing one usable by whoever is actually on call.

Preparation: what has to exist before the day

This is the part that gets skipped, and it is the part that determines how the rest goes.

A measured time-to-mitigate. End to end, from the first attack packet to stable clean traffic, taken from your own exercise rather than a datasheet. Nearly every decision below is made against this number.

A named decision owner, with a named deputy. Not a role, a person, with a phone number that is answered. The escalation matrix is where that mapping lives. Diversion and sacrifice decisions have commercial consequences and an on-call engineer should not be carrying them alone at 3am.

A rehearsed upstream path. Who you call at your transit provider, what they will accept from you, and how long each instrument takes to take effect. If your design depends on remotely triggered black hole filtering RFC 5635, FlowSpec rules RFC 8955 or a signalled request to an upstream mitigator RFC 9132, each of those has been tested at least once outside an incident.

A written significance threshold. The service metrics that, when crossed, make an incident notifiable — decided in advance, approved at the level your regulator expects, and expressed in numbers rather than adjectives.

A telemetry retention decision. What is captured, where it is stored, for how long, and who can export it without a support ticket.

The first five minutes

The goal of this window is a correct classification, not a fix.

1. Confirm the exhaustion. Read bits per second, packets per second, session-table occupancy and request concurrency together. The metric at its ceiling identifies the attack family; the others are consequences.
2. Establish the destination pattern. One address, a handful, or spread across a prefix. Spread changes both the detection story and the available instruments, because per-host thresholds will not have fired.
3. Check what is still working. Which services are degraded, which are down, and whether management access to your own equipment survives. A defence you cannot reach is a defence you do not have.
4. Start the evidence capture if it is not automatic. This is the cheapest action available and the only one that becomes impossible later.
5. Open the incident record with a timestamp. Every subsequent decision gets logged against it with its reason.

Do not tune thresholds yet. Threshold changes made before classification are the most common cause of self-inflicted damage during an incident.

The first fifteen minutes

Appliance	What you are deciding	Evidence it needs	Who signs it
Is this an attack	Traffic shape against your own baseline, not against a feeling	On-call engineer
Which resource is exhausted	Bits, packets, session-table occupancy, request concurrency	On-call engineer
Divert or absorb	Measured time-to-mitigate against measured attack duration	Named decision owner, in advance
Sacrifice a destination	What is behind the same circuit and what it costs to lose it	Named decision owner, in advance
Notify a regulator	Your written significance threshold, applied to service metrics	Legal or compliance owner
Stand down	Sustained clean traffic, plus attacker behaviour after the last change	Named decision owner

Every row that says "in advance" is a decision that cannot be made well at 3am. Write the name in, not the role.

Escalate on the named path. Notify the decision owner and the upstream provider in parallel rather than in sequence. Upstream engagement has lead time and starting it early costs nothing if it turns out to be unnecessary.

Take the diversion decision against the number, not the feeling. If measured time-to-mitigate exceeds the likely remaining attack duration, diversion is a decision to arrive late. Say so in the record and move on to what can act now.

Decide about sacrifice explicitly. If saturation is the problem and the upstream instruments available to you are blunt, the question is whether losing one destination is cheaper than losing everything behind the same circuit. Answer it out loud, log the answer, and name who authorised it.

Communicate before you are asked. A status update that says what is happening and when the next update will come buys more patience than one that waits for certainty.

The first hour

By now the shape is known and the work becomes iterative.

Tune from evidence, one change at a time. Each change gets a timestamp, a reason and an observed effect. Two changes at once produce an incident you cannot learn from.

Watch goodput, not blocked volume. The metric that tells you whether mitigation is working is how much legitimate traffic is getting through — blocked attack volume can rise while your customers are also being dropped.

Validate that real users are actually served. Synthetic checks from inside your own network prove very little. Test from outside, ideally from the networks your customers actually use.

Watch for the second vector. Multi-vector campaigns frequently open with the loud component and follow with the quiet one. A clean bandwidth graph is not the end of the incident.

Check the notification clock. If a regulatory duty applies, the clock started when you became aware, not when you finished mitigating NIS2 Directive. Involve the compliance owner while there is still time rather than at the end.

Escalation and upstream coordination

The upstream relationship does more work than any single device, and it works best when it was established before the incident. What to have written down: the contact path and its out-of-hours form; what the provider will accept from you and in what format; what they will do unilaterally and at what threshold; how long each action takes to converge; and what telemetry you receive back about actions they took on your behalf.

Providers vary enormously on the last of these, and it matters after the incident rather than during it: a mitigation you cannot describe is a mitigation you cannot report.

Evidence preservation

Reporting obligations are data obligations, and the data has a short life.

Capture, at minimum: start and end timestamps for the event and for each mitigation change; the vectors observed and the counters that identified them; the decisions taken with their authorisation; the observed effect on customers; and the traffic samples your own policy allows you to keep.

Store it somewhere the incident itself cannot take down. Evidence held only on the platform that is currently under attack is not evidence you can rely on, and telemetry that lives only in a provider’s console arrives on the provider’s export schedule rather than on your regulator’s.

Failback: coming down without a second outage

Standing down is a decision with its own failure mode, and it is made badly more often than the escalation was.

Require two conditions rather than one: clean traffic sustained for a defined period, and no change in attacker behaviour following your most recent mitigation change. An attacker who is still probing will notice a withdrawal, and a relapse ten minutes after an all-clear costs more credibility than the original incident.

Withdraw in the reverse order of application, one change at a time, watching between each. Return routing to normal before removing local mitigation, not after — the reverse order leaves a window where neither tier is protecting you.

Post-incident review

Hold it within a week, while the detail is still recoverable, and structure it around what the next incident needs rather than around blame. NIST’s incident-handling guidance NIST SP 800 61R3 is the usual external reference for the shape, and the CSF NIST CSF 2 is a workable frame for organising what comes out of it.

The post-incident review template sets the structure out in full. Six questions are enough:

1. What was the measured time from first attack packet to stable clean traffic?
2. Which step in that sequence took the longest, and was it technical or human?
3. What evidence did we want and not have?
4. Which decision was hardest, and can it be pre-decided before next time?
5. What did we break ourselves?
6. What changes in the runbook as a result — specifically, with an owner and a date?

Operational metrics worth tracking

Across incidents rather than within one: time to detect, time to mitigate, time to stable clean traffic, false-positive rate, legitimate goodput during mitigation, packet loss, p95 and p99 latency, connection success rate, circuit utilisation, and the mitigation tier’s own CPU, memory and session-table pressure.

The last group is the one most often missing, and it is what tells you whether your defence was comfortable or nearly overrun.

What this runbook does not cover

It does not cover offensive testing, which is a separate discipline with legal prerequisites — see authorised testing, legally and safely. It does not cover product selection, which belongs in a proof of concept rather than in an incident. And it does not replace an architecture: if the honest answer to “what would we do” is “call the provider and wait”, the gap is in the design rather than in the document.

Sources