What observability metrics keep and leave out

Consider a hypothetical release whose latency dashboard improves while more requests time out. The panel measures successful requests only. As slow requests disappear into the timeout outcome, the population left on the chart gets faster. The displayed number can be accurate and still encourage the wrong rollout decision.

Metrics make large amounts of activity manageable by compressing observations into counts, current values and distributions. That compression is their strength, but it also leaves things out. To understand a favorable trend, the reader needs to know what went into the number and which question it can answer. Customer latency, queue age and resource use may all matter to the same service without being substitutes for one another.

What information does the measurement keep?

A counter accumulates events, such as requests or failures. Because a process restart can reset it, a calculation of its rate must account for resets. A gauge records a current value, such as queue depth or active connections. Adding gauges across instances makes sense when each instance represents a distinct part of the total; adding overlapping views counts some work twice.

A distribution keeps variation that a single average loses. Most requests might become faster while a smaller group waits much longer. Histograms can help answer duration-threshold questions, with precision depending on the buckets and query. Prometheus explains how histograms and summaries aggregate differently; in particular, averaging the percentiles reported by individual instances does not yield a fleet-wide percentile.

In the hypothetical release, improving the successful-request percentile calculation still leaves timeouts outside the panel. The rollout comparison needs the eligible requests, observation window and failed or missing work beside that percentile. Together they reveal whether requests got faster or the slowest requests merely stopped contributing to the graph.

Match the measurement to the operational question

Read diagram description

Conceptual comparison. Keep units, eligible population and collection window with each metric; choosing the convenient graph can answer the wrong question. Diagram labels: Latency distribution: How long did requests take?; Queue age: How long has pending work waited?; Error fraction: What share of eligible requests failed?; Resource usage: Which capacity might constrain the work?.

Two identical percentages can tell different stories

Suppose one release serves 100,000 eligible requests and 400 fail. Another serves 500 requests and two fail. Both error fractions are 0.4%, but the second percentage rests on far fewer observations. Keeping the counts with the percentages makes that difference visible before someone treats the releases as equally well understood.

The counting rule matters just as much as the amount of data. If a failed attempt succeeds on retry, should the metric describe each attempt or the user's overall transaction? Either can answer a useful question. Counting attempts while making a claim about user transactions, however, changes what the denominator means without telling the reader.

The same issue appears when results are combined. For disjoint populations, add failed events and eligible events separately, then divide. Averaging percentages from services with different traffic gives each percentage equal influence regardless of how much work it describes. Even summed counts are misleading when populations overlap. Establish an event identity or a single measurement boundary before calling the result a customer-level rate.

Grouping can uncover what aggregation hides. A region or release-version dimension may reveal a localized regression concealed in the total. Each distinct label combination also creates another time series, so unbounded values such as user IDs and raw URLs make poor metric dimensions. Keep the bounded comparisons in metrics and the detailed identities in an appropriately controlled event store. This preserves a route to the detail without making every identity another series.

From a version-specific latency rise to a request trace

In an illustrative checkout investigation, the latency increase appears only on one application version. That is a useful lead, but it does not yet establish cause. The version might serve a different traffic mix or use a dependency differently. Compare those conditions with rollout timing, then follow a representative request through targeted tracing to see where the time was spent.

An incident dashboard helps this handoff when it keeps the affected population, a baseline and corresponding logs or traces close to the chart. The metric narrows the question; the detailed evidence helps explain the mechanism. Making that route easy is often more useful than adding another summarized panel.

Exploratory analysis can begin without a predetermined action. A correlation becomes subject to a different standard when it is trusted to trigger a consequential change: its population, exclusions and connection to that action need to be understood. That distinction leaves room for discovery while keeping an interesting pattern from silently becoming a release policy.

Adding the timeouts back to the opening rollout comparison changes the question from “Did this line go down?” to “What happened to all the requests?” Counts and distributions then do complementary work: one describes which outcomes occurred, and the other describes the timing of the population it actually measured.