Grafana dashboards for incident response

The first useful feature of an incident dashboard may be the ability to disagree with it. A fleet summary can accurately show normal latency while checkout is failing in one region. The chart is not necessarily wrong; it is describing a population broader than the one the responder needs to understand.

A Grafana incident view should make that difference easy to investigate. Begin with the affected user operation, show the traffic behind its measurements and provide a route into dependency and resource evidence. More panels help only when they answer the next question. Grafana displays the data its queries request, so a misleading denominator or an absent signal has to be addressed in the query and its presentation.

An incident view starts with a user operation

This guide assumes an approved Grafana instance with an existing metrics source such as Prometheus. Use the current installation and authentication guidance for that deployment rather than exposing a default local installation as a production service. Viewer permissions also affect the investigation: a link to a dataset the on-call rotation cannot read is not a usable route into evidence.

Grafana's dashboard guidance recommends organizing a view around a question. A service owner looking for failing requests and an infrastructure responder looking for saturation need different starting points, but their views should connect. That connection lets the team move from evidence of customer impact toward an explanation without treating every busy resource as an incident.

Designing around the conditions that could invalidate a health claim gives those links a clear purpose. If regional checkout failures would change the assessment, their request population and underlying evidence should be immediately reachable from the healthy fleet summary.

Exploratory dashboards can be broader than incident views. Separate those uses so that a rich analysis workspace does not become the mandatory route to recognizing immediate customer impact.

The request population behind each number

Consider a hypothetical checkout dashboard with normal fleet latency and failing requests in one region. Narrowing the view to that region, with its request count and time window still visible, explains how both observations can be true. The responder can then investigate resources supporting those requests instead of searching the entire fleet for an unusual graph.

This Prometheus query is one resource-level view. It calculates the non-idle share of CPU time separately for each instance:

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
))

The counter records accumulated idle CPU time. Its rate over five minutes describes how quickly that idle time is increasing; averaging within each instance gives the idle share for that instance. Subtracting from one and multiplying by 100 produces the percentage recorded in modes other than idle. That includes I/O wait, so the result should not be read as the percentage of time spent executing application work. Retaining the instance label keeps a high non-idle share on one host visible instead of averaging it into the fleet.

That result can support an explanation of the checkout problem, but it does not establish customer impact by itself. Compare it with the relevant service latency, failure rate and saturation evidence before deciding whether it justifies a page. Otherwise the dashboard can send a responder after a busy host that is doing exactly the work expected of it.

Memory illustrates a different measurement type:

100 * node_memory_MemAvailable_bytes
    / node_memory_MemTotal_bytes

Available and total memory are byte quantities measured at a point in time. These gauges can be divided to produce the available share without first taking a rate. Following the units is a useful check on query meaning: bytes subtracted from bytes per second would not produce a sensible anomaly measurement.

Latency needs the same care with populations. Inspect histogram buckets and aggregation dimensions before calculating a percentile. Averaging percentiles that have already been calculated does not produce a fleet percentile, so combining regional panels is not enough to answer a fleet-level latency question.

An incident dashboard needs a route through the evidence

Read diagram description

Suggested reading order. Start with the affected user operation, then narrow by traffic and population before investigating resources. Check freshness throughout. Diagram labels: User symptom: Errors or latency for the affected operation; Traffic + population: Region, tenant or other relevant scope; Dependency and resource signals: Compare against the same period; Logs or traces: Inspect the requests behind the aggregate.

Healthy, quiet and unknown are different states

A zero error value, zero traffic and missing data can look reassuring in a hurried glance while describing different conditions. Keep the service, environment, time range, units and measurement freshness visible. Test what happens when a target disappears or a query errors, as well as when traffic falls to zero. Those states should preserve the difference between observed health and an inability to observe it.

Deployment and configuration annotations provide useful leads when they overlap the symptom. Retain region or version labels and compare matching time windows to investigate the relationship. The annotation tells the responder what changed nearby in time; the rest of the evidence must explain whether that change accounts for the affected requests.

Carry that context through links. A service panel leading to logs should preserve the service, environment and absolute incident window wherever the tools support it. A link that silently returns to the latest fifteen minutes can make historical evidence look absent and force the responder to rebuild the search.

A historical incident tests the dashboard links

1. Choose a known historical incident and set its absolute time window.
2. Identify the user symptom, then follow the linked evidence toward a plausible dependency or resource explanation.
3. Compare panel values with direct data-source queries, including their labels and units.
4. Repeat the exercise with missing data and the restricted permissions used by the rotation.

Version queries and review changes that affect their interpretation. Choose refresh intervals with backend load in mind: frequent refreshes are a cost even when they add little useful information. The metrics guide develops the distinctions among counters, gauges and distributions when a query needs closer inspection.

In the regional checkout case, the dashboard should let a responder narrow the healthy fleet summary to the failing request population and carry its time window into logs or traces. The CPU and memory panels can then help explain those requests. That order makes a busy host relevant because of its relationship to the failure, rather than because its graph is the most conspicuous one.