Reading performance test results: percentiles, HDR histograms, saturation and Little's Law

A load test produces a lot of numbers, and it is easy to report the wrong ones. "Average response time 180 ms" sounds fine. Meanwhile one request in fifty takes four seconds, and the checkout page makes thirty requests. The test ran correctly, but the reading of it was wrong.

This article covers how to read results so they support a decision. It explains what percentiles do and do not tell you, why HDR histograms exist, how to find the point where a system saturates, how Little's Law checks your numbers, and how to write a report that leads to an action. The code examples work with k6 output, but the ideas apply to any tool.

Why the average lies

Latency distributions are almost never symmetric. Most requests are fast. A small fraction are slow because of garbage collection pauses, cache misses, lock contention, retries or a cold connection. The mean blends these two groups into a number that describes neither.

A simple example: 99% of requests take 50 ms and 1% take 3,000 ms. The mean is 0.99 × 50 + 0.01 × 3,000 = 79.5 ms. That number looks healthy, but no request actually takes 79.5 ms, and the p99.9 is 3,000 ms. The mean hides the users who are having a bad time.

The median (p50) is better at describing the typical request, but it ignores the tail completely. For user experience and capacity, the tail is usually what matters.

Percentiles and why the tail matters

The p95 is the value below which 95% of measurements fall. The p99 covers 99%, and so on. A p99 of 900 ms means one request in a hundred took 900 ms or longer. It says nothing about how much longer. That is why you should also look at p99.9 and the maximum.

"One in a hundred" sounds rare. It is not rare from the user's point of view, because a single page or user journey makes many requests. If a page triggers 20 independent backend calls, the chance that at least one of them is slower than the p99 is 1 − 0.99²⁰ ≈ 18.2%. Nearly one page view in five meets the "rare" slow case. The more calls fan out, the more the user experience is governed by the tail of each service.

By default, k6 prints avg, min, med, max, p(90) and p(95). Add the higher percentiles to the end-of-test summary:

// Report the tail, not just avg and p(95): add to your k6 options
export const options = {
  summaryTrendStats: ['avg', 'min', 'med', 'p(90)', 'p(95)', 'p(99)', 'p(99.9)', 'max'],
};

Thresholds should also target the tail. http_req_duration: ['p(95)<400', 'p(99)<1000'] is a much more meaningful pass/fail gate than any condition on the average.

Never average percentiles

This is the most common reporting mistake. Suppose a dashboard shows the p99 for each minute, and the report states the "average p99 over the test". That number is not a percentile of anything. Percentiles do not combine arithmetically. One minute with a terrible tail and two clean minutes average to a number that understates the real p99 across the run. The same problem appears when you average the p95 of several load generators, or of several service instances.

To combine windows or sources correctly, you need the underlying distribution, not its summary. You either keep all raw samples, or you keep a histogram for each window and merge the histograms. Merging histograms is exact, because adding bucket counts gives the histogram of the combined data.

HDR histograms

Storing every sample is fine for a short test but becomes expensive over hours or many generators. A simple histogram with fixed-width buckets either uses too many buckets or loses precision in the tail. Gil Tene's HdrHistogram solves this. It records values across a wide range (for example 1 µs to 60 s) with a fixed number of significant digits of precision. With 3 significant digits, every recorded value is accurate to within 0.1% of its own size. A 2 ms value and a 20 s value are both tracked with the same relative precision. Memory use stays fixed and small, recording is fast, and histograms merge exactly.

The script below reads the raw output of k6 run --out json=results.json. It builds one histogram per minute plus a merged one, and prints the percentiles. It also prints the average of the per-minute p99s, so you can see how wrong that number is.

"""Percentiles from raw k6 results using an HDR histogram.
    k6 run --out json=results.json script.js
    pip install hdrhistogram
    python percentiles.py results.json"""
import json
import sys
from collections import defaultdict

from hdrh.histogram import HdrHistogram

LOW, HIGH, DIGITS = 1, 60_000_000, 3  # 1 µs .. 60 s, 3 significant digits


def new_hist() -> HdrHistogram:
    return HdrHistogram(LOW, HIGH, DIGITS)


overall = new_hist()
per_minute = defaultdict(new_hist)

with open(sys.argv[1]) as f:
    for line in f:
        rec = json.loads(line)
        if rec.get("type") != "Point" or rec.get("metric") != "http_req_duration":
            continue
        micros = min(HIGH, max(LOW, int(rec["data"]["value"] * 1000)))  # k6 reports ms
        minute = rec["data"]["time"][:16]                       # 'YYYY-MM-DDTHH:MM'
        overall.record_value(micros)
        per_minute[minute].record_value(micros)

ms = lambda us: us / 1000
print(f"{'window':<17} {'count':>8} {'p50':>9} {'p95':>9} {'p99':>9} {'p99.9':>9} {'max':>9}")
for minute in sorted(per_minute):
    h = per_minute[minute]
    print(f"{minute:<17} {h.get_total_count():>8} "
          + " ".join(f"{ms(h.get_value_at_percentile(p)):>9.1f}" for p in (50, 95, 99, 99.9))
          + f" {ms(h.get_max_value()):>9.1f}")

# Correct way to combine windows: merge histograms, then read the percentile.
merged = new_hist()
for h in per_minute.values():
    merged.add(h)
avg_of_p99 = sum(h.get_value_at_percentile(99) for h in per_minute.values()) / len(per_minute)
print(f"\noverall p99 (merged histogram): {ms(merged.get_value_at_percentile(99)):.1f} ms")
print(f"average of per-minute p99s:     {ms(avg_of_p99):.1f} ms   <- not a percentile")

k6 reports http_req_duration in milliseconds as a float. The script stores microseconds as integers, because HdrHistogram records integer values. On a synthetic data set where 1% of requests take 3 seconds, unevenly spread across three minutes, the merged p99 is 3,000 ms. The average of the per-minute p99s is about 1,050 ms, which is a number that does not exist in the data.

The per-minute table is useful in its own right. A p99 that rises steadily over a constant-load test points to a leak, a growing queue or a filling cache. A p99 that jumps in one window and then recovers points to an event such as a deployment, a GC cycle, an autoscaling step or a noisy neighbour.

One caveat: the histogram is only as honest as the measurements recorded into it. If the load generator is a closed model that stops sending when the system slows down, it suffers from coordinated omission and under-reports the tail, no matter how precise the histogram is. Use an open model (an arrival-rate executor in k6) when you measure latency under load.

Finding the saturation point

Every system has a throughput at which it stops coping. Below that point, latency is roughly flat as load grows. Near it, queues form and latency climbs steeply. Beyond it, throughput stops increasing (or drops), latency explodes and errors appear. The bend in the curve is often called the knee.

To find it, run a stepped test. Increase the arrival rate in stages (for example 100, 200, 300 … req/s), hold each stage long enough for the system to reach a steady state (a few minutes at least), and record for each stage:

• offered load (requests sent) and achieved throughput (successful responses per second),
• p50, p95 and p99 latency,
• error rate, and
• utilisation of the main resources.

Plot latency and achieved throughput against offered load. The knee is where achieved throughput stops following offered load, or where p95 starts rising faster than load. A sensible operating limit sits clearly below the knee, not at it. Queueing theory says latency grows sharply as utilisation approaches 100%. In a simple single-server queue model, the mean waiting time is proportional to ρ / (1 − ρ), where ρ is utilisation. Going from 50% to 90% utilisation multiplies that factor by nine.

Explaining saturation with the USE method

Knowing where the knee is only helps half way. The team needs to know why it is there. Brendan Gregg's USE method gives a checklist: for every resource, check Utilisation (how busy it is), Saturation (how much work is queued waiting for it) and Errors.

"Resource" means more than CPU and memory. In web and API systems, the bottleneck is often a logical resource:

• database or HTTP connection pools (busy connections, wait time for a connection),
• thread pools and worker counts,
• locks and row-level contention in the database,
• rate limits of downstream providers, and
• network bandwidth or the file descriptor limit on the load balancer.

A classic pattern: latency rises sharply while CPU sits at 50%. That does not mean there is spare capacity. It means the bottleneck is somewhere else, often a pool that is fully used, with requests queueing for a connection. Saturation metrics (queue length, wait time) find this. Utilisation metrics alone do not.

Little's Law as a sanity check

Little's Law states that, in a stable system, the average number of items in the system equals the arrival rate multiplied by the average time each item spends in it: L = λ × W. It holds regardless of the distribution of arrivals or service times, which makes it a very useful check.

Two practical uses:

Concurrency from throughput. At 400 req/s and an average response time of 250 ms, there are on average 400 × 0.25 = 100 requests in flight. If your service has 64 worker threads in total, the numbers do not add up, and requests are queueing somewhere before they are processed. If your test reports this throughput with far fewer VUs than the result requires, check the test itself.

Capacity of a pool. Rearranged as λ = L / W, the law gives the maximum throughput of a fixed-size resource. A database pool of 20 connections, with each query holding a connection for 50 ms, can serve at most 20 / 0.05 = 400 queries per second. If a request needs one such query, the service saturates at around 400 req/s however many CPU cores you add. Often this calculation predicts the knee before you even run the test, and it tells you which knob to turn.

Use averages for Little's Law, not percentiles. It is one of the few places where the mean is the right statistic.

Checking the test, not just the system

Before you trust a result, confirm that the load generator was not the bottleneck. Check that:

• the achieved request rate matches the configured arrival rate. k6 reports dropped_iterations when it could not start iterations on time, usually because maxVUs was too low;
• the load generator's CPU and network stayed well below saturation;
• the error rate is reported alongside latency. Fast errors can make latency look better, so filter durations to successful responses or report both; and
• the test environment resembles production in the dimensions that matter: data volume, instance sizes, connection limits and caches.

Writing a report people can act on

A performance report is not a dump of charts. Each finding should answer: what did we observe, how sure are we, why does it happen, what does it mean for the business, and what should change. A structure that works:

1. Summary for decision makers: does the system meet its targets at the expected peak? Yes, no, or under which conditions. One paragraph.
2. Targets and workload model: the SLOs tested against, the traffic mix, arrival rates and test durations, and the environment.
3. Findings: one block per finding, ordered by severity, each with evidence and a recommended action.
4. Capacity: the measured knee, the bottleneck resource and the headroom against the planned peak.
5. Limitations: what the test did not cover, and how the environment differs from production.
6. Appendix: raw data, scripts and run IDs, so the results can be reproduced.

Here is what a single finding can look like. The numbers are an illustrative example, not from a real engagement:

FINDING F-02 · High · Capacity
Claim:      POST /v1/payments breaches its p95 SLO (400 ms) above ~560 req/s.
Evidence:   Stress run #3, 2026-10-14 14:05–14:40 UTC, staging (prod-equivalent sizing).
            p95 135 ms at 300 req/s -> 720 ms at 600 req/s; errors 0.1% -> 3.4%.
            App CPU max 58%; DB pool: 20/20 connections busy, pool wait p95 480 ms.
Cause:      DB connection pool saturation (Little's Law: 600 req/s x 40 ms query
            time = 24 connections needed > 20 available).
Impact:     Planned launch peak is 800 req/s -> SLO breach for most checkout traffic.
Action:     Raise pool to 50/node + pgbouncer; re-test at 800 req/s. Owner: platform team.
Confidence: High (reproduced in 3 runs; stable at 300 req/s for 4 h soak).

The finding states a claim with a clear threshold, points to the exact run, gives the resource evidence, explains the cause with arithmetic anyone can check, connects it to the launch, and names an owner. A reader can disagree with it, but cannot misunderstand it.

Checklist

• Report p50, p95, p99, p99.9 and max, not the average alone.
• Never average percentiles. Merge histograms or raw data instead.
• Use an open workload model so the tail is measured honestly.
• Find the knee with a stepped test and set the operating limit below it.
• Explain the knee with utilisation and saturation metrics for every resource, including pools.
• Check your numbers with Little's Law.
• Verify the load generator was not the bottleneck.
• Write findings as claim, evidence, cause, impact, action.