Kubernetes Ingress DDoS Hardening

Two things make Kubernetes different from a fixed estate under attack, and neither is about the container runtime.

The first is that the ingress controller is an unusually clean choke point: one component, declaratively configured, that every external request passes through. The second is that the cluster’s ability to grow is a liability as well as an asset — the elasticity that absorbs a legitimate spike also absorbs an attack, and pays for it.

Establish the baseline

# What the ingress controller is currently handling
kubectl -n ingress-nginx exec deploy/ingress-nginx-controller -- \
  curl -s localhost:10254/metrics | grep -E 'nginx_ingress_controller_(requests|connections)'

# Current replica counts and what the HPA thinks
kubectl get hpa -A
kubectl top pods -A --sort-by=cpu | head -20
kubectl top nodes

# Which workloads have no limits at all — the blast-radius question
kubectl get pods -A -o json | jq -r '
  .items[] | select(.spec.containers[].resources.limits == null)
  | "\(.metadata.namespace)/\(.metadata.name)"' | head

Ingress controller: connection and request limits

Using the ingress-nginx annotations, which map onto the same nginx directives covered in the nginx hardening guide:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    # Requests per second per source address, with a burst allowance.
    nginx.ingress.kubernetes.io/limit-rps: "50"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    # Concurrent connections per source.
    nginx.ingress.kubernetes.io/limit-connections: "20"
    # Bound the request body so an upload path is not a memory attack.
    nginx.ingress.kubernetes.io/proxy-body-size: "8m"
    # The slow-HTTP answer: bound how long a client may take.
    nginx.ingress.kubernetes.io/client-body-timeout: "10"
    nginx.ingress.kubernetes.io/client-header-timeout: "10"
spec:
  ingressClassName: nginx
  rules:
    - host: example.test
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

Controller-wide defaults belong in the ConfigMap rather than on every Ingress:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  # Worker connection ceiling and keepalive behaviour
  max-worker-connections: "65536"
  keep-alive: "30"
  keep-alive-requests: "100"
  # Slow-HTTP bounds applied globally
  client-body-timeout: "10"
  client-header-timeout: "10"
  # Do not let a single upstream hold connections indefinitely
  upstream-keepalive-timeout: "30"
  # Log the rate-limit rejections so false positives are visible
  log-format-escape-json: "true"

Verify the limit fires where you think it does:

# Should start returning 503 from the ingress, not from the app
for i in $(seq 1 200); do
  curl -s -o /dev/null -w '%{http_code}\n' https://example.test/ &
done | sort | uniq -c

Bounding autoscaling deliberately

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 3
  # The number that decides what an attack costs you. Choose it as a
  # budget decision, not as a capacity guess.
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      # Do not let a burst produce an instant tenfold scale-out.
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300

maxReplicas is the most consequential line on this page. Left high or unset, the cluster answers an attack by buying capacity to serve it — which keeps the service up and transfers the damage to the invoice. That is occasionally the correct trade and should always be a decision someone made on purpose.

Resource limits as blast-radius control

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    # A pod without a memory limit under flood can take neighbours with it.
    cpu: "1"
    memory: 512Mi

# Namespace-wide floor so a workload cannot be deployed without limits
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: budget
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    pods: "150"

A ResourceQuota is the cluster-level equivalent of maxReplicas: it bounds what any one namespace can consume regardless of what its autoscaler wants.

Keeping the ingress itself alive

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx

# Ingress controllers deserve guaranteed resources of their own —
# the component that sheds load must not be the one that is starved.
resources:
  requests:
    cpu: "1"
    memory: 1Gi
  limits:
    cpu: "2"
    memory: 2Gi

Network policy: limiting lateral exposure

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-ingress-only
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

This does not stop a flood. It stops a flood that reached one workload from being able to reach others, which is the difference between an incident and a cluster-wide one.

Observability

# Rejections at the ingress, by status
kubectl -n ingress-nginx exec deploy/ingress-nginx-controller -- \
  curl -s localhost:10254/metrics \
  | grep 'nginx_ingress_controller_requests' | grep -E 'status="(429|503)"'

# Whether the HPA is at its ceiling — the moment cost becomes the constraint
kubectl get hpa web -o jsonpath='{.status.currentReplicas}/{.spec.maxReplicas}{"\n"}'

# Pods being OOM-killed under load
kubectl get events -A --field-selector reason=OOMKilling

The metric worth alerting on is HPA at maximum, because that is the moment the cluster stops absorbing and starts dropping — and it is a much earlier signal than user complaints.

Safe rollout order

1. Add resource limits and a LimitRange first. Lowest risk, immediate blast-radius benefit.
2. Set maxReplicas deliberately, as a budget decision.
3. Add ingress timeouts.
4. Add rate limits in a permissive setting and watch the 429 rate for a full business peak.
5. Tighten only what the measurement supports.

Rollback

kubectl rollout undo deployment/web
kubectl -n ingress-nginx rollout undo deployment/ingress-nginx-controller
# Annotations are per-Ingress; remove the limit annotations to revert instantly
kubectl annotate ingress web nginx.ingress.kubernetes.io/limit-rps-

The honest limit

The ingress controller can only reject what reaches it, and what reaches it has already crossed your cloud load balancer and consumed your egress and ingress bandwidth allocation. A cluster is not a defence against saturation; it is a well-instrumented place to decide what to do about everything below saturation. The tier that answers volume is upstream, as the architecture comparison works through.