Metric types and percentiles

Record durations in a histogram, add the instances' buckets, then read the percentile: a mean or an average of percentiles hides the slow requests.

Before you read: predict

ExamHall's autosave writer runs on three pods. Each pod's OpenTelemetry SDK exports an autosave-duration histogram with buckets up to 0.05, 0.1, 0.25, 0.5, 1 and 2.5 s, plus one for anything slower. In the 10 minutes after a busy exam start, the pods exported these counts per bucket, and each pod's dashboard tile showed that pod's p99:

pod	≤ 0.05 s	0.05–0.1 s	0.1–0.25 s	0.25–0.5 s	0.5–1 s	1–2.5 s	pod p99
A	3,000	2,800	190	10	0	0	0.21 s
B	1,500	1,400	90	10	0	0	0.22 s
C	300	400	250	10	10	30	2.0 s

What is the p99 autosave time across the whole fleet for those 10 minutes?

The 30-second version

A counter only rises (totals, errors); a gauge holds a level that rises and falls (queue depth); a histogram counts durations into buckets, plus a count and a sum. A thin slow tail barely moves the mean. For a fleet percentile, add the instances' bucket counts, then interpolate inside the bucket holding the rank; never average percentiles.

Record durations in a histogram, add the instances' buckets, then read the percentile: a mean or an average of percentiles hides the slow requests.

How it works

Step 1 of 12

Counter. A counter only rises, such as requests or errors since start. The OpenTelemetry JS SDK 2.11.0 drops a negative add to one.

In the kitchen

the barrier's count only goes up through the day → a counter (step 1)

Analogy: A multi-storey car park's three records (transport infrastructure)

A car park keeps three records: the entry barrier's count, the "spaces free" sign at the ramp, and the pay machines' takings by tariff band (up to 1 h, 1–2 h, 2–4 h, 4–24 h, over a day).

At the car park	In metrics
the barrier's count only goes up through the day	a counter (step 1)
the "spaces free" sign rises and falls with the cars inside	a gauge (step 2)
the takings say how many stays fell in each band, not how long each stay was	a histogram's bucket counts (step 4)
the average stay looks ordinary while one car in a hundred stays three days	the mean against the tail (steps 5–6)
an operator with two car parks adds their counts band by band before asking how long the longest-staying 1% stayed	merge, then compute (steps 7–8)
inside a band she can only assume the stays are spread evenly	interpolation and resolution (steps 9–10)

Breaks down when: a car park's takings close each day, while a histogram's counts keep rising from start-up; a window comes from the difference between two exports (rate()), not a daily total. And tariff bands are set for pricing, while buckets should sit around the times you alert on (step 12).

Contrasting case: a driver's own receipt shows one exact stay. That is a log line or a span, not a metric (obs-three-signals).

In real terms: an instance's histogram is a set of counters, one per bucket; merging instances is addition, and a percentile is read from the sum.

Examples, run for real

4.1 ExamHall EXH-2380: the mean looked fine

On a busy morning the autosave dashboard's mean stayed calm while candidates on the largest exam complained. The writer records each autosave with a partition attribute; partition p7 holds the largest exam:

4.1 sdk-metrics 2.11.0 · 10,000 autosaves · mean (sum ÷ count) 86 ms
4.1 p99 from the summed buckets 1.75 s · average of the 4 partitions' p99s 0.68 s
4.1 p99 by partition: p1 0.10 s · p2 0.10 s · p3 0.10 s · p7 (largest exam) 2.42 s

The mean hides a 2% tail (step 5). The average of the partitions' p99s is a number no autosave had (step 7); the summed buckets give the fleet's p99 (step 8), and the breakdown points at p7, one hot partition. MR !2680 replaces the mean panel with a p99 panel computed from summed buckets, plus that breakdown.

4.2 Ticketing: is 300 ms enough?

A box office promises 95% of checkouts within 300 ms. Its histogram had boundaries 0.1, 0.5, 1 and 2.5 s:

4.2 2,000 checkouts · recorded at or under 0.3 s: 1,850 (92.5%)
4.2 boundaries 0.1, 0.5, 1, 2.5: 0.3 lies inside (0.1, 0.5] · interpolated share 79.3%
4.2 boundaries 0.1, 0.3, 0.5, 1, 2.5: le="0.3" bucket read directly · share 92.5%

With no boundary at 0.3 s, the share is interpolated across a 0.4 s-wide bucket and comes out far too low (steps 9–10). Adding a boundary at 0.3 turns it into a count. Either way the promise is missed, but only the second says by how much.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

A box office's refund worker keeps the length of its retry queue in const depth = meter.createCounter('refund_queue_depth'), with the OpenTelemetry JS SDK. It calls depth.add(1) when a refund joins the queue and depth.add(-1) when one is done. From start-up, the calls run in this order: +1 +1 +1 -1 +1 +1 -1 -1 +1 +1 +1 -1 +1.

What does the first export read for refund_queue_depth?

Question 2 of 5

A ticket-checkout histogram has buckets up to 0.1, 0.25, 0.5, 1 and 2.5 s, plus one for anything slower. In one hour it counted 2,000 checkouts: 1,500 in the first bucket, 300 in the second, 160 in the third, 30 in the fourth and 10 in the fifth. A trace sample shows that the 30 in the fourth bucket all took between 0.52 and 0.58 s.

What does histogram_quantile(0.99, …) report for that hour?

Question 3 of 5

A video service records each play's start time in a histogram with buckets up to 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 and 10 s. Two alerts read it over 10 minutes: one pages when the mean (sum ÷ count) is over 150 ms, the other when histogram_quantile(0.99, …) is over 1 s. In one 10-minute window of 20,000 plays, 1 in 400 plays took 8 s to start and the rest took 80 ms.

Which alerts page?

Question 4 of 5

A payments API runs in two regions, with three pods in eu and two in us. Each pod exports a payment_seconds histogram with buckets up to 0.1, 0.25, 0.5, 1 and 2.5 s, plus one for anything slower, and a region's payments spread evenly over its pods. The status page's p99 panel runs avg(histogram_quantile(0.99, sum by (region, le) (rate(payment_seconds_bucket[5m])))) and reads 1.3 s. The last 5 minutes held these counts per bucket in each region, and every 5 minutes of the past hour looked the same:

region	≤ 0.1 s	0.1–0.25 s	0.25–0.5 s	0.5–1 s	1–2.5 s	over 2.5 s
eu	6,000	2,700	240	60	0	0
us	300	400	160	100	40	0

Which query makes the panel show one number: the p99 of all the payments in those 5 minutes?

Question 5 of 5

A ticket checkout service runs on 3 pods. Each pod records every checkout's time with checkoutSeconds.record(seconds), where checkoutSeconds = meter.createGauge('checkout_seconds'), and the OpenTelemetry SDK exports once a minute. In one minute each pod completes 400 checkouts, spread through the minute: 8 of each pod's 400 take 4 s and the rest take 0.2 s. On-call needs the export to show how many of that minute's checkouts took longer than 1 s.

Which change gives that number?

Go deeper

Pro tips

When a percentile panel sits flat on a round number, check whether the rank has reached the +Inf bucket: histogram_quantile then returns the highest finite bound, however slow the requests are. When: a p99 line that never rises above your top boundary during an incident (step 10). Verify: tip 30 of 1,000 checkouts at 8 s, then at 30 s · top finite boundary 2.5 s · histogram_quantile(0.99) 2.5 s and 2.5 s Status: RAN.

Source

Prometheus and OpenTelemetry docs can change: the facts below are as of 2026-10-01. No item is keyed on a source; every key is executed, and the quoted rules are what lib/prom.mjs and lib/promql.mjs re-implement. Valid as of 2026-10-01.

Prometheus docs, Histograms and summaries: https://prometheus.io/docs/practices/histograms/ (prom-histograms.txt). The curriculum row's first source. Quoted: "The φ-quantile is the observation value that ranks at number φ*N among the N observations"; "averaging the quantiles yields statistically nonsensical values"; histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) // GOOD.; quantile error "is limited by the width of the bucket the quantile is located in"; "the even distribution within the relevant buckets is exactly what the interpolation algorithm for classic histograms assumes"; "The calculated fraction is accurate if there happens to be a bucket boundary precisely at 0.3."
Prometheus docs, Metric types: https://prometheus.io/docs/concepts/metric_types/ (prom-metric-types.txt). Quoted: a counter's "value can only increase or be reset to zero on restart"; "Do not use a counter to expose a value that can decrease"; a gauge "can arbitrarily go up and down"; a classic histogram's buckets are "cumulative counters for the observation buckets, exposed as <basename>_bucket{le="<upper inclusive bound>"}", with _count "identical to <basename>_bucket{le="+Inf"}". Supports steps 1, 2 and 4.
Prometheus docs, Query functions, histogram_quantile(): https://prometheus.io/docs/prometheus/latest/querying/functions/ (prom-functions.txt). Quoted: "it assumes a uniform distribution of observations within the bucket (also called linear interpolation)"; "If a quantile is located in the highest bucket, the upper bound of the second highest bucket is returned"; "To aggregate all classic histograms, specify only the le label". Supports steps 9–11.
OpenTelemetry, Metrics Data Model: https://opentelemetry.io/docs/specs/otel/metrics/data-model/ (otel-data-model.txt). The row's second source. Quoted: for a Gauge, "only the last value is reported in the metric stream via OTLP"; a Histogram has "A count (count) of the total population of points" and "A sum (sum) of all the values"; "Bucket upper-bounds are inclusive (except for the case where the upper-bound is +Inf) while bucket lower-bounds are exclusive." Supports steps 3–4.
OpenTelemetry, Metrics API: https://opentelemetry.io/docs/specs/otel/metrics/api/ (otel-metrics-api.txt). Quoted: a Counter "supports non-negative increments"; a Histogram "is intended for statistics such as histograms, summaries, and percentile"; UpDownCounter example uses include "the number of items in a queue". Supports steps 1–2. That the JS SDK drops a negative add, and its default boundaries, are executed facts of @opentelemetry/sdk-metrics 2.11.0, not quotes.
OpenTelemetry, Supplementary Guidelines: https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/ (otel-supplementary.txt). Background on choosing instruments; nothing here rests on it alone.

Checked against the tools and docs as of 2026-10-01.

Predict first