Recovering an agent task after OpenShell containment

Stopping an agent can leave a responder with a surprisingly urgent question: did its last write already reach production? NVIDIA’s Open Agent Safety Platform, launched September 28, 2026, combines the OpenShell runtime’s access controls with Sentry, a separate monitoring and enforcement design on BlueField-4 data processing units (DPUs). The NVIDIA launch announcement describes stronger ways to interrupt unsafe execution. An interruption still leaves earlier work to account for.

The difficult case is a write accepted just before the agent loses the ability to verify it. The service may already have changed, another action may be queued, and the original customer problem may persist. Restarting the agent does not resolve any of those questions; it creates another opportunity to act on an incomplete picture.

Recovery begins with the action the agent was allowed to take and the system state that decision described. A responder then establishes which earlier actions reached their destinations, whether current state still fits the approved plan, and what an independent check says about the result. OpenShell enforces important limits around execution. The responder’s job is to understand what happened inside those limits. A hypothetical Kubernetes scaling request shows how the two jobs meet.

What is outside the agent’s own judgment

OpenShell places an agent in an isolated sandbox and enforces policy outside its reasoning loop. NVIDIA documents kernel-level control of filesystem access, system calls, processes and outbound connections. Provider controls can hold real credentials outside the agent workload and attach them only to requests that pass applicable authorization checks. These controls constrain the resources available to execution rather than relying on the model to remember a rule. OpenShell runtime and credential controls

Its policy prover can formally examine proposed access changes before they take effect, exposing an expansion for review. The guarantee is limited to features represented in the prover’s model; it is not proof that every possible permitted agent behavior is safe. The OpenShell policy prover documentation describes the scope of that formal check.

The current policy model also divides changes by when they can take effect. Filesystem and process permissions are established at startup. Network policy can change dynamically after validation, and the documentation describes provider attachments as runtime-updatable. The OpenShell policy architecture explains this division, which becomes practical when a responder needs to alter a blocked workflow.

A validated policy update may be sufficient for one missing network destination. A newly needed startup permission can instead require a fresh sandbox. Knowing which restriction fired helps avoid trying to repair the task through a control that cannot change that part of the running environment.

Sentry supplies an additional, out-of-band watchdog on BlueField-4 DPUs. NVIDIA says it continuously observes agent behavior and can quarantine an agent that tries to move beyond its permitted boundary, with containment in milliseconds. That timing is a vendor claim, not an AIOpsSRE benchmark. The NVIDIA Sentry announcement describes this independent enforcement layer.

Sentry is optional in the reference design. OpenShell is open source and can run outside the complete Vera and BlueField architecture, so evaluation of the runtime should not assume that every installation includes the hardware watchdog. NVIDIA layered reference architecture

Whatever the enforcement arrangement, preventing the next operation cannot undo an operation already accepted elsewhere. The recovery procedure starts with that boundary in time.

The API accepted the request; the agent never checked it

Consider a hypothetical agent permitted to scale checkout-api from 12 replicas to at most 20 when an approved saturation condition holds. The decision uses Deployment resource version 8841. It chooses 16 replicas and submits this bounded request:

PATCH Deployment checkout-api

current replicas: 12
requested replicas: 16
state precondition: resourceVersion 8841

In the example, Kubernetes accepts the write. Before checking the rollout, the agent sends another outbound request that violates policy and is denied, or an independent control quarantines it. Enforcement has stopped later execution as intended. The accepted scale request is now an unfinished verification problem for the responder.

The responder needs to establish whether the write committed, whether the desired 16 replicas became available and whether another actor changed the Deployment afterward. Queued actions and the customer symptom also matter. A confirmed scale write may have solved the original problem, or may have left both the service and the remaining plan in a different state from the one the agent expected.

Collect that evidence before restarting or broadening access. The recovery steps below first distinguish an operational interruption from a routine denial, then follow the scale request to its destination and back to the customer-facing result.

Does this denial leave production work unresolved?

A forbidden outbound write stopped before anything consequential happens can be a successful, routine policy event. The workflow may continue safely with no responder needed. The Kubernetes example differs because an earlier action may have changed production and verification was interrupted. These conditions help identify when a denial has also created an operational question:

Condition	Operational consequence
A consequential write may have happened first	Destination state may be unknown
The agent is quarantined	The workflow cannot safely continue without reconciliation
Required production work is interrupted	Reliability is now affected even if the security control succeeded
The agent requests unexpected additional authority	The task or policy may no longer describe actual behavior
The same unexpected deny repeats after correction	The workflow may be drifting or the policy remains wrong
Destination state cannot be inspected	Safe recovery cannot yet be established

Route the event according to that consequence. Routine telemetry can record an expected denial with no production effect; an unresolved write needs someone able to inspect the destination. Treating every successful block as a page would consume attention without helping the scaling request that really does need a decision.

Pausing writes preserves a traceable Deployment state

Pause new consequential writes from the interrupted workflow so that investigation does not compete with another change. Preserve the relationship among task, policy, blocked operation and possible destination effects. This operating template records enough of that relationship to begin:

containment_event:
  agent_id:
  sandbox_id:
  task_id:
  policy_version:
  event_time:
  blocked_operation:
  last_confirmed_operation:
  possible_inflight_operation:
  target:
  target_state_version:
  execution_id:
  queued_actions:

This is an SRE template, not an OpenShell-native schema. It connects the control event to a particular execution and to the state that execution might have changed. Keep safe read-only inspection available where possible; reopening the write path should not be a prerequisite for learning what the earlier request did.

For the scaling request, the resulting record lets a responder locate the API operation while queued writes remain paused. Without that pause, another action could change the Deployment before the first result is understood, making both reconstruction and approval harder.

Kubernetes records the result of the scale request

The agent’s history is useful for finding the request, but can stop before the remote system records success. Evidence of a remote effect therefore needs to come from the destination. The absence of a success message in the agent’s transcript is evidence about the transcript, not proof that Kubernetes rejected the write.

Inspect current Deployment resource version, desired and available replicas, rollout conditions, and relevant audit or controller evidence. Together these can establish both whether a change took place and whether the object has moved on since then.

The next action depends on which result that evidence supports:

Evidence	Next decision
Approved write is confirmed and remains current	Verify the service postcondition before another change
Write occurred, but state changed afterward	Reassess from current state
Destination proves the write did not occur	Decide whether a fresh attempt is still justified and authorized
Evidence is incomplete or conflicting	Preserve an explicit unknown state
Destination cannot be inspected safely	Hold further writes and assign the unresolved action to an owner

Even an empty lookup can remain inconclusive, depending on the destination’s visibility delay and operation records. Preserve an explicit unknown outcome until available evidence resolves it. The dangerous shortcut begins with a valid observation:

I did not receive proof that the write succeeded

and quietly turns it into a new action unsupported by that observation:

The write failed, so send it again

Ordinary automation timeouts create the same ambiguity. The unknown write reconciliation guide explains the implementation pattern; containment makes it more visible because the agent may no longer be able to investigate its own last operation.

An old approval may describe a Deployment that has changed

While the agent is stopped, a person, autoscaler or controller can continue changing the Deployment. The pause does not freeze the rest of production. A short interruption can therefore be enough to invalidate assumptions used in the original request.

Suppose the approved decision was based on this state:

replicas = 12
resourceVersion = 8841
requested replicas = 16

The responder now reads this instead:

replicas = 18
resourceVersion = 8897

Scaling from 12 to 16 may have been sensible when the request was authorized. With 18 replicas at a new resource version, replaying the earlier plan no longer follows from that decision. Read current state, check whether the original saturation objective still needs work, and obtain the fresh decision required by the workflow rather than treating old approval as perpetual authority.

This issue is independent of where the model runs. A local agent and a cloud agent both need permission that applies to the present target and action. The guide to local models and agent permission boundaries follows the same question through a local deployment.

Repairing the specific access the task needs

A blocked task can make broader access feel like the quickest repair. First identify the operation that failed and the capability it actually requires. That gives the reviewer a specific proposed change instead of a general request to make the agent less constrained.

If the legitimate next step reads one additional endpoint, specify that destination, operation and credential scope. Unrestricted outbound access would solve a much larger problem than the task has demonstrated. Let the policy-validation path examine the narrow expansion before relying on it.

For missing network access, the documented model allows dynamic policy changes after validation. That may let the intended work proceed without replacing the complete sandbox.

A new filesystem or process permission is different because those controls are established at startup. A fresh sandbox with a reviewed policy is a distinct recovery path, often clearer than attempting to mutate an environment whose startup restrictions no longer fit the task. The OpenShell policy lifecycle explains that lifecycle.

The same reasoning applies at other boundaries: one endpoint rather than a network range, one method rather than both reads and writes, one namespace rather than a cluster, or one repository rather than an organization. For the interrupted plan, it can mean one reconciled operation instead of replaying everything.

That last limit matters even when no permissions change. Earlier decisions may no longer suit the current Deployment. Disposing of obsolete queued work lets the legitimate remainder proceed without dragging stale assumptions back into execution.

Sixteen replicas is not yet a recovered checkout

Return to the scale request. Sixteen available replicas establish a Deployment result, but the request was motivated by a customer symptom. An independent observer also needs to assess that symptom. A possible verification record is:

verification:
  deployment_available_replicas: 16
  request_error_rate: recovered
  p95_latency: recovered
  synthetic_checkout: pass
  unexpected_policy_denials_after_resume: 0

These checks and thresholds are illustrative. Their job is to be capable of disagreeing with the agent. Healthy replicas alongside a failing customer path mean the operational task remains open. A recovered service alongside the same unexpected denial after resumption means the execution path still needs investigation before another restart.

The guide to authorizing actions and independently verifying outcomes places that independent service check in the broader production-agent workflow. Containment and successful API execution are useful evidence, but neither alone answers whether the requested operational improvement occurred.

From containment to verified recovery

The recovery record exposes a useful interval: from intervention to understanding what changed and establishing service recovery. The proposed measures below describe that work. They are SRE measures for a local implementation, not native OpenShell metric names:

Measure	Question it answers
Expected policy denials	Is policy enforcing known restrictions?
Unexpected policy denials	Are normal workflows hitting unplanned limits?
Quarantined agents	How often does execution reach stronger containment?
Unknown in-flight actions	How much consequential state still lacks a known outcome?
Reconciliation latency	How long does it take to establish what actually changed?
Stale-authority rejections	How often does production state move before execution or resume?
Repeat denial after resume	Did the recovery actually correct the control path?
Containment to verified recovery	How long until both service state and execution state are understood?

Keep operation IDs and similar high-cardinality identifiers in controlled logs or traces, where they can support reconstruction without becoming metric labels.

A zero-denial target would be misleading. Expected denials can demonstrate that a needed restriction works; overly broad permissions could improve that number. Instead, observe whether unexpected denials are explained and whether consequential actions remain unresolved. Those measures preserve the distinction between enforcement doing its job and recovery work still waiting for a person.

A recovery rehearsal with interrupted writes

Put the recovery procedure beside the policy configuration before production authority is enabled. When the agent cannot explain its last action, a responder needs a record that connects classification, preserved evidence, destination state and permission to resume. This compact template keeps those decisions together:

containment_runbook:
  classify:
    expected_or_unexpected:
    possible_prior_side_effect:
    production_work_interrupted:

  preserve:
    agent_id:
    sandbox_id:
    task_id:
    policy_version:
    last_confirmed_operation:
    possible_inflight_operation:

  reconcile:
    destination_source_of_truth:
    current_state:
    operation_evidence:
    result: confirmed | rejected | unknown

  authority:
    original_precondition:
    current_precondition:
    fresh_approval_required:

  resume:
    minimum_policy_change:
    new_sandbox_required:
    queued_work_disposition:

  verify:
    service_postcondition:
    control_postcondition:
    accountable_owner:

A synthetic or nonproduction rehearsal can interrupt the workflow before a write, after a write and while its result is unknown. Let state change while the agent is quarantined. Exercise both a dynamic policy repair and a fresh-sandbox recovery, plus failed service verification and repeated denial after resumption. Each case asks the responder a different question that a clean end-to-end run would not reveal.

Judge the rehearsal by whether someone can explain state at each interruption and decision. Eventual completion is insufficient if the path included a duplicate write or a stale approval. The record should make those errors visible rather than letting the final green status hide them.

Why an immediate restart can complicate recovery

Restarting before reconciling the earlier write and widening access before explaining the failed operation have the same weakness: they add consequential activity while its premise is unresolved. Destination and customer checks may show that no resumption is needed at all. Classification also keeps harmless expected denials out of that heavier process, preserving attention for cases such as our interrupted scale request.

When a denied connection needs only a record

The complete procedure is unnecessary when strong evidence shows that consequential state could not have changed. Imagine, as a hypothetical contrast, a disposable test sandbox with no write-capable tool available. Its agent attempts a forbidden outbound connection, the policy denies it and no production work is affected. The event demonstrates the intended boundary rather than an unknown production write.

Record that expected denial and correct the task or policy if needed. Reserve the heavier recovery process for production authority, incomplete evidence, stale state or customer impact. This boundary makes the procedure usable: responders do not need a full reconciliation exercise to acknowledge every blocked test connection.

The scaling request after recovery

OpenShell moves important restrictions outside the model’s own judgment, and Sentry adds an independent hardware enforcement domain in deployments that use it. The NVIDIA reference architecture describes how those layers fit together. Their value is clearer when the operational procedure acknowledges the work that remains after a successful block.

For the hypothetical checkout, three answers establish recovery: what happened to the write, what action is now authorized against current state, and what the customer-facing check shows. Keeping them together gives the next responder a starting point even when the agent stopped between execution and verification. The record then connects the interrupted execution to the service a responder is trying to recover.

Related AIOpsSRE reading

• Unknown write reconciliation
• Production agents: authorize the action, then verify the outcome
• Evaluate incident-response agents against service outcomes

Primary sources

• NVIDIA Sentry announcement
• OpenShell runtime and credential controls
• OpenShell policy prover documentation
• OpenShell policy architecture
• NVIDIA reference architecture
• OpenShell policy lifecycle
• OpenShell provider and credential documentation

Evidence and scope: Technical sources were reviewed September 29, 2026. NVIDIA’s timing and containment results remain attributed vendor claims, not independent benchmarks. The Kubernetes sequence, recovery templates and measures illustrate an SRE procedure; they do not report a production incident or define native OpenShell schemas.