Bimodal latency distribution: what two peaks mean
A bimodal latency distribution means your response times cluster into two distinct groups — say, one peak at 40ms and another at 600ms — instead of one bell around a typical value. It’s one of the most diagnostic shapes in performance work, because it says your endpoint doesn’t have one speed with noise; it has two different execution paths, and finding what separates them usually solves the problem.
Why averages lie about bimodal endpoints
If half your requests take 40ms and half take 600ms, the average is 320ms — a number that describes almost no actual request. Alerts tuned to the mean miss the slow cohort entirely; capacity plans based on it are fiction. Even p50 misleads here: it lands on whichever cohort happens to hold the majority and silently ignores the other. Bimodal shapes are only visible in a histogram or a latency heatmap — which is why teams that only chart averages are routinely surprised by them.
The usual causes
- Cache hit vs miss — the classic. Hits return in milliseconds; misses pay the full query or upstream fetch. Ratio between the peaks is often 10× or more.
- Connection pool waits — under concurrency, requests that get a database connection immediately are fast; requests that queue for one are slow. Appears only under load, at a threshold matching the pool size.
- Cold starts — serverless functions or JIT-warmed services: first hit slow, warm hits fast.
- Two literal code paths — a fast path for common input and a slow path (extra query, fallback provider, pathological regex) for the rest.
- GC or compaction pauses — most requests are fine; the unlucky ones land during a pause.
How to detect it
Collect enough samples per route — 30 is a workable floor; below that, two-peak claims are noise — and look at the distribution, not summary stats. A log-scaled histogram makes the two humps obvious. Algorithmically, you bucket latencies into log-spaced bins, smooth, and look for two significant peaks separated by a valley; that’s precisely what Lobe’s bimodal detector runs on every captured route, reporting the split as fast 43ms ×81 (54%) / slow 604ms ×69 (46%) instead of leaving you to eyeball a chart.
How to find what separates the cohorts
Once you know an endpoint is bimodal, tag requests with which cohort they landed in and diff everything else about them: cache headers (hit/miss), connection reuse (new vs pooled), time of arrival (bursts vs idle), input shape. One of those attributes will correlate almost perfectly with the cohort — and that attribute is your root cause. If nothing correlates and the split only appears under concurrency, suspect a bounded resource and test it directly with a concurrency sweep.