AI Agent Incident Response: What to Do When an Agent Gets It Wrong
An incident response plan for AI agents: detection, kill switches, containment, reversing actions, root cause analysis and preventing repeats.
Quick answer
Prepare before anything goes wrong. Every production agent needs a way to detect problems (outcome monitoring and alerts, not just uptime), a tested kill switch that stops it without a deployment, a log detailed enough to list and reverse its actions, a fallback to the manual process, and a runbook naming who decides what. When an incident happens: contain first, assess which actions and customers were affected, reverse or correct them, find the root cause from traces (context, tools, permissions, model change or manipulation), fix the control that failed, re-run evaluations and update the runbook.
What counts as an agent incident
| Type | Example | Typical cause |
|---|---|---|
| Wrong action | Refunds issued twice; records overwritten | Non-idempotent tool, retry loop, bad data |
| Wrong statement | Agent promises a policy that does not exist | Ungrounded answer, outdated source |
| Data exposure | Agent includes another customer's details | Retrieval without tenant scoping |
| Manipulation | Agent follows instructions hidden in an email | Prompt injection (goal hijack) |
| Runaway cost | Token spend jumps tenfold overnight | Loop, oversized context, traffic spike |
| Silent degradation | Accuracy drops after a model update | Unevaluated change |
Before an incident: the minimum preparation
- Detection: outcome metrics (error rate, escalation rate, refunds, complaints), cost alerts and anomaly alerts on action volume; see AI agent observability
- Kill switch: a configuration flag or credential revocation that stops the agent in minutes, tested regularly
- Degraded modes: read-only mode, approval-required mode, or routing all cases to people
- Action log: every write with enough detail to identify and reverse it
- Compensating actions: scripted reversals for common writes (void credit, restore record)
- Runbook: who is on call, who decides to stop the agent, who talks to customers
- Version records: model, prompt, tool and policy versions per run
Key takeaway
If stopping an agent requires a code deployment, you do not have a kill switch. Test the switch the same way you test backups.
During an incident: contain, then assess
Contain first. Switch the agent to a degraded mode or stop it. Do not wait to understand the cause; agents can repeat a mistake hundreds of times while you investigate. If manipulation is suspected, revoke the agent's credentials and those of any connected tools.
Assess impact. Use the action log to list everything the agent did in the affected period: which records, customers, amounts and messages. Narrow by the run IDs and tools involved. This list drives remediation and any notifications.
Reverse and remediate. Run compensating actions where possible, correct records, and contact affected customers with a clear explanation and fix. Some actions (messages sent, information disclosed) cannot be undone; correct them and record what was done.
Finding the root cause
Agent incidents rarely have a single bug. Read the traces of affected runs and work through the layers:
| Layer | Question |
|---|---|
| Context | Did the agent have wrong, stale, missing or injected information? |
| Tools | Did a tool return an error, partial data or allow an unsafe action? |
| Permissions | Could the agent do more than its task required? |
| Approvals | Did an approval step exist, and was it meaningful? |
| Changes | Did a model, prompt, tool or data change precede the incident? |
| Input | Was there adversarial content (emails, documents, web pages)? |
After an incident: fix the control, not just the case
Patching the prompt for the specific case is rarely enough. Fix the control that let the mistake through: narrow a tool, add a validation, scope retrieval per tenant, add a threshold for approval, make a write idempotent, add a cost limit. Add the incident to your evaluation set so it is tested on every future change, re-run evaluations and record the change. The OWASP Top 10 for Agentic Applications is a useful checklist for which class of control failed; see the OWASP agentic guide.
Need agents you can stop, audit and repair?
ZSpace Labs builds kill switches, degraded modes, action logs and runbooks into production agents. See AI automation services.
Communicating with customers and stakeholders
Be direct: what happened, who was affected, what you have done and what changes. Do not blame "the AI"; customers and regulators hold the business responsible, as covered in who is responsible when an AI agent makes a mistake. Where personal data is involved, follow your data breach procedures and legal notification duties.
Practise
Run a short exercise each quarter for important agents: simulate a manipulation or a runaway loop, trigger the kill switch, list the affected actions from logs and reverse a sample. The first exercise usually reveals a missing log field or a switch nobody can find. See AI red teaming for adversarial testing.
Conclusion
Agents will make mistakes; the difference between a minor incident and a serious one is preparation. Detect outcomes, keep a tested kill switch and degraded modes, log every action so it can be reversed, contain before investigating, and fix the control that failed. Then practise, so the plan works when it matters.
The detail behind two steps of this plan is covered separately: how to safely undo autonomous actions and how to build an audit trail for agent actions.
Common questions.
Any event where an agent causes or nearly causes harm: wrong actions on records or money, incorrect statements to customers, data exposure, runaway costs, or behaviour suggesting manipulation such as prompt injection.