DDoS Testing and Proof-of-Concept Methodology

A proof of concept exists to measure the properties a specification sheet structurally cannot state. Capacity is on the datasheet — though reading one without being misled takes more care than it appears to. Efficacy is not there at all, because it is a relationship between a product and your traffic, and nobody except you can measure that pair.

This plan is written to be handed to a manufacturer as-is. Everything in it is defensive: it tests your own infrastructure, under authorisation, and produces evidence you keep.

Before anything: authorisation and boundary

No test starts until three documents exist: written authorisation from the asset owner, a defined scope naming exactly which addresses and services are in it, and an acknowledgement from any provider whose shared infrastructure sits in the path. This is covered properly in authorised stress testing, legally and safely, and it is not optional even when everything under test is yours — a test that saturates a shared uplink affects other customers.

Agree the abort condition and who can call it, in writing, before the first packet.

Capture a baseline first

A test against an unmeasured normal produces numbers nobody can interpret.

Record, for a period that covers a genuine business peak: bits per second, packets per second with the packet-size distribution, concurrent sessions, session establishment rate, requests per second per significant endpoint, and the latency distribution your users actually experience. Keep the raw data; the comparison after each test run is against this, not against a memory of it.

The packet-size matrix

Run every network-layer test at the conventional size ladder — 64, 128, 256, 512, 1024 and 1518 bytes RFC 2544 — and record both the bit rate and the packet rate at each step. The number you are looking for is the point at which behaviour changes: latency rises, loss appears, or the device’s own CPU or session table reaches a ceiling.

A single figure at a single packet size is not a result. Two products with identical bit-rate ratings routinely diverge by a large factor at 64 bytes, and 64 bytes is what an attacker sends.

Test classes to run

Volumetric. Floods sized against your circuit, run below saturation deliberately: the point is to measure the device, and a saturated uplink measures the uplink. Where saturation behaviour matters, test it against the upstream tier rather than against the appliance.

Protocol and state exhaustion. SYN floods, ACK floods, fragmentation. Watch session-table occupancy on every stateful device in the path, not only on the product under test — this class frequently takes down a firewall behind a working mitigation tier.

Application layer. Requests that are individually valid and expensive to serve, aimed at the endpoints that are actually expensive in your application. This is where a generic test plan produces a meaningless result, because the expensive endpoints are specific to you.

Encrypted application layer, where applicable. Establish where TLS terminates during the test and confirm that the arrangement is the one you intend to run in production.

Multi-vector. Two or more classes simultaneously, which is how real campaigns arrive. Products that handle each class alone can behave differently when classification has to run against several at once.

Low and slow. Slow request bodies, slow reads, long-held connections at low rate. This class is invisible to flow-based detection by construction RFC 4732, so it tests whether the product sees anything other than volume.

The false-positive test

This is the test most often left out and the one that decides most purchases.

Replay real legitimate traffic — captured from your own service — simultaneously with each attack class, and measure what fraction of it completes successfully. Do it at your genuine business peak rather than at a convenient hour. Record the result as a percentage of legitimate transactions completed, not as an absence of complaints.

Then repeat it with the product tuned more aggressively, so you can see the shape of the trade rather than a single point on it. A product that reaches high efficacy only at a false-positive rate your business cannot absorb has not reached high efficacy.

Degraded-dependency behaviour

Every product depends on something. The test is the same regardless of what it depends on: disconnect the dependency, repeat an identical test, and compare.

For a product that consults a manufacturer-operated service, disconnect its outbound path. For a hybrid, disconnect the cloud tier. For an ecosystem deployment, remove the component that supplies detection. Graceful degradation and step degradation are different answers, and both are legitimate designs — but only one of them matches a requirement that says protection must survive a supplier interruption.

Deployment resilience

Trigger the failure rather than reading about it. Pull power. Kill the software. Force a failover. Time each one and record what reached the origin during the gap.

For inline deployments, establish whether bypass is hardware or software and confirm which one you were sold. For out-of-path deployments, measure diversion end to end on your own routing, and then run an attack shorter than that measured time to see what the design actually does about short bursts.

Operations and telemetry

Have your own engineer, not the manufacturer’s, perform the routine tasks: create a policy, adjust a threshold, investigate an event, export an incident report. Time them.

Then export the telemetry an incident report would need, in a format you can parse, on your own schedule. A product whose evidence can only be produced by a support request has changed your reporting obligation into a dependency.

Recovery and failback

Withdraw mitigation while traffic is still flowing and confirm nothing breaks on the way down. Return routing to normal and confirm sessions survive. This step is skipped in most evaluations and is the source of a good proportion of self-inflicted second outages.

A scoring model with empty defaults

Appliance	Criterion	Suggested weight	Your score
Detection and efficacy	25	Not measured
False positives and goodput	20	Not measured
Deployment resilience	15	Not measured
Operations	15	Not measured
Layer 7 and application coverage	10	Not measured
Integration and telemetry	5	Not measured
Support, region and compliance	5	Not measured
Five-year total cost of ownership	5	Not measured

Weights are a starting point to be argued with, not a standard. Scores are deliberately empty: this site does not publish scores for any product, because a score without a measurement behind it is an opinion wearing a number.

Two things about this table are deliberate. The weights are a starting point for an argument inside your organisation rather than a standard — a service provider will weight multi-tenancy and operations far higher than a single enterprise will. And every score reads “not measured”, because that is what an honest score looks like before a bench has run.

Reporting the result

Record it the way this site records any measurement — source, date and evidence class, as set out in the data methodology. Write the result so that someone who was not present can reproduce it: the configuration under test, the exact model and software version, the generator and its settings, the baseline data, each test run with its measured outcome, and the abort events if any.

Record what you could not test and why. An evaluation that says “we could not test degraded-dependency behaviour because the vendor declined” is more useful than one that silently omits the row.

What a POC cannot settle

It cannot tell you what the product will be like to live with in year three, which is a question about the manufacturer rather than the device. It cannot measure support quality, which only shows up under pressure. And it cannot price the five-year relationship — for which the total cost of ownership model is a better instrument than a bench.

It also cannot be run once and reused. A result is valid for the software version, the traffic mix and the business shape it was measured against, and all three move.

Sources