Runbook example: checkout diagnosis and bounded mitigation

Adapt an illustrative procedure with stop conditions, rollback, escalation and independent recovery checks.

Illustrative dataset · Tool version 2 · Reviewed September 29, 2026

Could another responder follow the procedure?

The illustrative checkout procedure compares rollout timing, application pool waits and database pressure. Its scope section tells the responder to stop if target ownership or rollback compatibility is uncertain. The queue-worker variant instead distinguishes work arriving faster than capacity from retries caused by a failing dependency. These are starting instructions to adapt and exercise for a service; neither example includes executable production commands.

Section	Illustrative starting text
Service and environment	Illustrative checkout API / staging
Owner and escalation	Service owner on call; escalate through the team’s approved incident route.
Required access	Confirm responder role and read-only monitoring access before the exercise.
Applies when / stop conditions	Checkout failures after a deployment. Stop if rollback compatibility or target ownership is uncertain.
Confirm impact	Check a synthetic checkout and eligible request failures in the same window.
Diagnosis and branches	Compare rollout timing, application pool waits and database pressure. Do not infer database saturation from pool occupancy alone.
Bounded mitigation	Use only the service-approved, bounded mitigation after checking preconditions. Record the target and expected effect.
Rollback and partial failure	Record the exact service-approved reversal. If partially applied, stop and establish current state before another action.
Recovery verification	Verify successful user outcomes independently; record the agreed observation window and unresolved impact.
Coordination and updates	Record incident owner, incident link and next update time.
Last exercise and review	Illustrative starting example only. Replace with service-specific instructions and exercise evidence.

What this example does not establish

The exercise record belongs beside the procedure so a later responder can distinguish its last edit from the last attempt to follow it.

How a different input changes the result

Open the checkout and queue-worker examples separately. Checkout diagnosis compares rollout timing, pool waits and database pressure; the queue example distinguishes incoming load from dependency-driven retries. The different branches show why a procedure needs the service’s own failure behavior before it can guide a responder.

These datasets and worksheets may be adapted for your team or training. Attribute AIOpsSRE and link to this example when redistributing. Third-party sources retain their own terms.

Get AIOpsSRE by email · Report a calculation or explanation issue