Prompting an assistant to investigate an incident

“What caused this incident?” is an appealing prompt because it asks for the answer everyone wants. With an incomplete record, however, a more useful question may be “What should we check next?” The first request can encourage a settled explanation before the evidence distinguishes it from another plausible cause. The second gives a responder a way to make progress while that uncertainty remains.

Prompt engineering for operators starts with defining a useful answer and supplying the material needed to assess it. The template below focuses on choosing a read-only investigation step. It asks the model to connect facts, alternatives and missing information so the responder can decide whether the proposed check is worth doing.

The incident record inside the prompt

Include the affected service, incident window and time zone, measured user impact, source identifiers and relevant constraints. Separate that source material from instructions. A log line or ticket comment belongs to the evidence being analyzed, even if its text tells the model to do something.

Task: Help the responder choose the next read-only check.
Service and window: [service, start, end, timezone]
Observed impact: [measurement and source ID]
Evidence: [numbered, timestamped excerpts]
Constraints: Do not execute changes or invent missing values.

Return:
- Observed facts, each linked to an evidence ID
- Plausible explanations and evidence against each
- The next read-only check and what its result would distinguish
- Missing information that could change the recommendation
If the evidence is insufficient, say so.

The output fields serve different purposes. Evidence IDs let a responder find the observation behind a claim. Plausible explanations keep competing interpretations visible. Asking what the next check would distinguish makes the recommendation testable: a query is valuable because its result can change the investigation, rather than because it looks technically sophisticated.

The template is a reusable starting point, not an enforcement mechanism. A request not to execute changes cannot replace tool permissions that prevent execution. Likewise, a concise account of supporting evidence is more useful for review than a demand for a transcript of internal reasoning.

Make the source material part of the prompt

Read diagram description

A proposed incident-analysis prompt structure. Supply the task and evidence, ask for supported findings and unknowns, then review the answer against the source material. Diagram labels: Task + supplied evidence: Question, logs, timeline and source identifiers; Answer instructions: Separate observations, hypotheses and missing facts; Model draft: Findings with references and unresolved questions; Responder review: Check claims before using the draft.

Examples can demonstrate a useful incomplete answer

Few-shot examples are sample questions and answers that demonstrate the format expected from the model. They also demonstrate what the prompt rewards. An example that jumps from high CPU to scaling up leaves out workload, limits and the possibility that additional capacity will send more pressure downstream.

In a hypothetical investigation, an example answer can test the deployment explanation by proposing a read-only comparison of the earliest error timestamp with the rollout window. Errors that began before rollout would weaken that explanation and leave another cause open for investigation. The answer should say which evidence is missing rather than select a decisive root cause to satisfy the requested format.

This does not mean the model should always ask for more information. It means the examples should show when the available evidence supports a conclusion and when the valuable output is a discriminating question. That is the behavior the evaluation needs to examine later.

For commands, establish environment and scope before producing an executable form. Even a read-only Kubernetes query needs the intended context and namespace. Logs can contain sensitive information, so collection should be limited to the workload that can answer the question rather than automatically gathering an entire cluster.

Testing proposed checks against held-out cases

Use redacted incidents and constructed cases that were not used while developing the prompt. Check whether claims have support, constraints survive, missing information is identified and the suggested investigation could separate the proposed explanations. Repeat important cases to see whether a single convincing response concealed variation.

Response latency is only part of the cost. Measure the time spent reviewing and correcting the answer as well, and record the model, prompt version, retrieval configuration and tool catalog. A change to any of these can alter what the assistant sees or recommends, so an earlier evaluation may no longer describe the deployed system.

OWASP’s prompt-injection guidance explains why instructions alone do not establish a security boundary. Include source text that attempts to redirect the model and inspect the surrounding permissions separately. The answer can look compliant in ordinary cases while an integration still grants actions beyond the intended task.

The prompt should preserve the evidence a responder needs to make the decision, while the surrounding system enforces consequential action constraints. Keeping those responsibilities separate lets a well-explained recommendation remain useful without treating the model's obedience as authorization.

How an answer should change with new evidence

Before an answer enters the incident record, verify the claims that would affect action. If a citation is unclear, request the relevant excerpt and compare it with the authoritative source. Retain alternatives that still fit the record instead of smoothing them away to make the summary more decisive.

Then revise at the point where the evidence changed. For example, “The error increase began before the deployment; reconsider the deployment hypothesis” gives the model a concrete reason to alter its explanation. Preserve that observation and the correction in the record. Repeatedly generating a whole new answer until one sounds convincing makes it harder to understand why the conclusion moved.

The NotebookLM dossier guide develops source-document work, while an independently enforced skill contract addresses execution. Each extends a different part of the workflow introduced here: finding defensible information and controlling what may be done with it.

Some tasks need only a useful draft, so exhaustive citations can add more work than value. Increase evidence requirements with the consequence of the decision, and keep enforced permissions in place regardless of how well the answer follows the prompt.

As the incident record grows, the assistant’s answer should change for a traceable reason: an earlier error timestamp weakens the deployment hypothesis, or a missing dependency log leaves two explanations open. A useful prompt makes those changes inspectable and asks for the next query that could resolve them. The responder can act on that query while the cause remains uncertain.