How to Test False Positives in a DDoS Mitigation System

A vendor demonstration ends with a number: ninety-nine point something per cent of attack traffic removed. The number is usually accurate. It is also close to meaningless, and its persistence in this industry does real damage.

Consider a device that drops every packet arriving at the protected address. Its attack-traffic drop rate is 100%. Its report is perfect. The service is completely down, and the report will not mention it, because the report measures the attack and the outage happened to the customers.

Every mitigation decision is a classification with two ways to be wrong. Datasheets, demos and post-incident reports overwhelmingly describe one of them.

At a glance

Appliance	Metric	What it tells you	Why it is not enough
Attack traffic dropped, %	The device acted on the flood	A device dropping everything scores 100%
Packets passed, count	Something got through	Packets are not transactions; a broken session still passes packets
Legitimate sessions completed	Users could still connect	Completion is not the same as usable latency
Business transactions completed	The service still did its job	The measurement you want, and the hardest to instrument
Time to first successful retry	How long a refused user waits before succeeding	Rarely measured, and the one users actually experience

Only the bottom three describe the outcome the business cares about. The top two describe the device's opinion of itself.

The asymmetry that hides the problem

A false negative is visible. Attack traffic reaches the service, the service degrades, alarms fire, and everyone involved knows what happened.

A false positive is nearly invisible from inside. The refused user sees a failure that looks exactly like the attack succeeding. They do not call to report that your mitigation refused them, because they have no way to know that is what occurred — and from the operations centre, a chart showing legitimate traffic falling during an attack looks like the attack working, which produces a response of tighten the thresholds.

That feedback loop runs the wrong way. The condition and the treatment are indistinguishable from the console, and the treatment makes the condition worse. Breaking the loop requires measuring the legitimate side directly, which is the whole subject of this page.

What to measure instead

Four measurements, in order of how much they cost to instrument and how much they are worth.

Session completion rate. Of the legitimate connections attempted, what fraction completed the transport handshake and exchanged data? This is cheap to measure and catches the coarse failures.

Business transaction completion. Of the attempted end-to-end operations — a login, a payment, an API call that returns a valid result — what fraction succeeded? This is what the business means by availability, and it is the number to put in an acceptance criterion.

Added latency on surviving traffic. A session that completes in eleven seconds instead of four has not been dropped, and for most services it has effectively failed. Mitigation techniques that hold, challenge or retry impose delay by design, and the delay belongs in the result.

Time to first successful retry. When a legitimate client is refused, how long until an identical attempt succeeds? A hold of two seconds is invisible; a hold of five minutes has cost you the customer. This is the measurement almost nobody takes and the one users actually experience.

Generating legitimate traffic that means something

The measurement is only as good as the traffic it counts, and this is where most tests go wrong.

Replayed production traffic is the best available source. It carries the real distribution of client behaviour, including the awkward tail. It requires care with credentials and personal data, and the handling has to be settled before the capture rather than after.

Synthetic business transactions are the practical fallback: a scripted client performing a complete business path, run continuously through the test at a known rate. The discipline that matters is that the script must exercise the whole path. A synthetic check that opens a connection and closes it will pass while every real login fails.

A held-out control is the part that gets skipped. Run identical traffic against an equivalent unprotected path throughout the test. Without it you cannot separate mitigation damage from the attack’s own effect on shared infrastructure, and that separation is the entire point.

Whatever the source, it must run before, during and after the attack. The before-run is the baseline and the test is worthless without it, because a 96% completion rate means nothing until you know the service completes 99.4% on an ordinary afternoon.

The arithmetic of rare events

A short test cannot measure a small rate, and this trips up otherwise careful evaluations.

If legitimate traffic runs at 200 transactions per minute and the test lasts ten minutes, that is 2,000 samples. A true refusal rate of one in a thousand produces an expected two failures, and observing zero, one or four would all be unremarkable. The test simply cannot distinguish one-in-a-thousand from one-in-ten-thousand, and reporting “no false positives observed” from that sample overstates what was learned.

Two ways out, and they combine well. Run longer at the interesting thresholds rather than running many short scenarios. And report the confidence bound rather than the point estimate: “no refusals in 2,000 transactions” honestly means “the rate is probably below about 0.15%”, which is a defensible sentence where “zero false positives” is not.

State the sample size next to every rate in the report. A result without its denominator is not a result.

Where false positives concentrate

They are not evenly distributed, and knowing the pattern tells you where to aim the test.

Behind shared addresses. Carrier-grade NAT, corporate proxies and large partner gateways concentrate hundreds of users behind one address. Any per-source rate limit sees a single extremely busy client, and the busiest legitimate sources on most networks look statistically like attackers. Test explicitly with a high-volume source that is known to be legitimate.

Unusual clients. Old TLS stacks, embedded devices, scripted integrations and anything with a non-standard connection pattern. Behavioural classifiers learn from the majority and treat the tail as anomalous, which is exactly what it is and exactly why it must not be dropped.

Genuine peaks. A campaign launch, a market open, a results announcement. Thresholds set from an average week refuse traffic at the first real peak — and the first real peak is frequently mistaken for an attack, which brings us back to the feedback loop.

Retry storms. After any interruption, clients retry simultaneously. The recovery pattern resembles an attack closely enough that a defence can prolong an outage it did not cause.

The mechanisms behind several of these are covered in rate limiting and behavioural baselining; the point here is that they are the places to point the test at, not incidental cases.

The test structure

1. Baseline. Legitimate traffic only, no attack, for long enough to establish completion rate, latency distribution and their normal variance.
2. Attack without mitigation, if the environment safely allows it. Establishes what the attack alone does, and separates its damage from the defence’s.
3. Attack with mitigation, at the intended production settings. The primary result.
4. Attack with mitigation at aggressive settings. Establishes the cost of the escalation an operator will reach for at 3am, so the cost is known before the night it is needed.
5. Recovery. Continue measuring after the attack stops. Late-clearing state, expiring blocks and quarantine timers show up here and nowhere else.
6. Repeat with the awkward sources from the section above.

Record for every phase: completion rate with its sample size, latency percentiles including the 95th and 99th, and time to first successful retry for anything refused.

Putting the number in the contract

A false-positive figure belongs in the acceptance criteria, and it belongs there in the business’s own units.

Under the agreed attack scenario, business transaction completion shall not fall below X% of the measured pre-attack baseline, with no legitimate source refused for longer than Y seconds, measured over not fewer than N transactions.

Three numbers, all set before the test runs. The methodology for the surrounding test plan is in the proof-of-concept methodology, and the legal groundwork for generating attack traffic at all is in authorised testing — neither of which is optional before any of this happens.