TL;DR:
- AI agents fail differently from stateless services: errors compound across steps, the blast radius of a bad deployment is larger, and rollback must account for in-flight workflows
- Shadow mode (running new agent versions alongside production, comparing outputs without acting on them) is the safest pre-production validation pattern for agentic systems
- Define rollback triggers before you deploy, not after something goes wrong
Deploying a new version of a REST API is mostly a solved problem. Feature flags, canary releases, health checks, and automated rollback are well-understood. AI agents add complications that change how you need to think about each of these.
An agent that processes a five-step workflow can fail at step three in ways that are only visible by inspecting the full execution trace, not a single response. An agent that costs £0.08 per workflow at v1 might cost £0.40 per workflow after a prompt change — not an error, but a significant regression. A new model version might handle edge cases differently in ways that pass all your evals but cause problems with a specific customer segment’s data patterns. These issues require deployment strategies adapted to the agentic context.
The Core Problem: Compounding Errors
Standard services have a well-defined failure mode: the request either returns an error or it doesn’t. You monitor error rates, latency, and availability.
Agent workflows fail in subtler ways. A prompt change that causes the agent to occasionally misinterpret tool output doesn’t show up as a 500 error — it shows up as a downstream task performed incorrectly. A change to how the agent decides to call a tool might increase token consumption by 40% without any visible errors. These are regressions that standard HTTP monitoring doesn’t catch.
This means your deployment process needs to observe agent-specific signals:
- Workflow completion rate (what percentage of workflows reach a successful terminal state)
- Step-level error rate (where in the workflow are failures occurring)
- Token consumption per workflow (cost regression signal)
- Tool call frequency and distribution (unexpected tool use patterns indicate prompt or logic changes)
- Output quality score if you have LLM-as-judge or embedding-based evaluation in your pipeline
Define baseline metrics from your current production version before you start the rollout. You need a comparison point.
Shadow Mode: The Pre-Production Safety Net
Before a canary rollout of a new agent version, shadow mode is the most valuable validation step. The idea is straightforward: route production traffic through both the current agent version and the new version, but only act on the current version’s outputs. The new version processes real inputs in real time, generating outputs that are logged and compared but never executed.
Shadow mode lets you:
- Compare new vs current version outputs at scale on real production data
- Identify edge cases your test suite didn’t cover
- Verify token consumption at production traffic volumes
- Catch model API compatibility issues before they affect users
For agents that take real-world actions (sending emails, updating records, calling APIs), shadow mode is particularly important because you can’t easily undo committed actions. Running shadow mode for 24-48 hours before a canary release catches a meaningful proportion of production-environment surprises.
The practical implementation depends on your framework. With LangGraph or similar stateful frameworks, you can run parallel graph executions against the same input, suppressing the side-effect nodes in the shadow instance. With Temporal or Prefect-based workflows, you can spin up shadow workflow executions that skip the external action activities.
Canary Deployments
Once shadow mode gives you reasonable confidence, a canary rollout sends a percentage of real traffic to the new version. Start small — 5% is sensible for agentic systems given the higher blast radius compared to stateless services.
Route by workflow type, not random sampling. If you can segment traffic by task type, route only the lower-stakes workflow category to the canary first. Don’t start with your most complex or highest-consequence workflows.
Extend the bake time. A canary bake period for a standard microservice might be 15 minutes. For an agent handling multi-step workflows that might take 10-30 minutes each, your observation window needs to be long enough to see completed workflows, not just initiated ones. Four to eight hours minimum.
Monitor the right signals. During canary: compare workflow completion rates between canary and control, compare token consumption, compare output quality if you have scoring. A canary where completion rate drops from 94% to 88% is a rollback trigger even if error rates look similar.
Blue/Green for Major Version Changes
For major changes — new model, substantial prompt restructure, new tools added — a blue/green deployment gives you a clean cutover with a fast rollback path.
Keep the old version (blue) running alongside the new version (green). Route all traffic to green. If something goes wrong, route back to blue immediately — sub-second cutover if you’re using a load balancer or traffic management layer.
The complication with agentic systems: in-flight workflows. If a workflow started on green at step 2 when you decide to roll back, you can’t just move it to blue mid-execution. The workflow is mid-state; moving it requires either:
- Letting it complete on green (accept the risk for already-started workflows)
- Aborting it and restarting on blue (requires your orchestration layer to support workflow cancellation and restart)
- Maintaining state compatibility between versions so in-flight workflows can continue on blue
The cleanest pattern is checkpointed workflows where each step’s output is persisted, and rollback can resume from the last successful checkpoint on the previous version. Temporal’s durable execution model makes this relatively natural; ad-hoc function-based agents do not.
Rollback Triggers and Automation
Define your rollback triggers before the deployment. The adrenaline of an incident is not when you want to be deciding what threshold warrants a rollback.
Specific trigger examples:
- Workflow completion rate drops more than 5% relative to baseline
- Token consumption increases more than 20% relative to baseline
- Any critical tool call error rate exceeds 2%
- Canary output quality score drops below a defined threshold
Automate what you can. If your monitoring system can compare canary vs control metrics and trigger an automated rollback, use it. Human-in-the-loop is fine for the initial canary decision, but once things start moving in the wrong direction you want fast rollback, not a Slack thread discussion.
State and Memory Considerations
Agents with persistent memory (user preferences, conversation history, prior task context) introduce additional rollback complexity. If the new version has written memory entries in a schema the old version doesn’t understand, rollback may fail or produce unexpected behaviour.
The standard solution is memory schema versioning. Store a schema version with each memory entry. Both versions should be able to read entries written by the other — either by supporting multiple schema versions or by running a migration before cutting over. This is the same discipline as database schema migration applied to agent memory stores.
A Practical Rollout Sequence
- Evaluate — run your test suite and evals against the new version
- Shadow mode — route production traffic in parallel, log outputs, compare for 24-48 hours
- 5% canary — route a small traffic slice to the new version, monitor for 4-8 hours
- 25% canary — widen if 5% canary metrics are clean, observe for another 4 hours
- Full rollout — complete cutover, keep old version warm for 24 hours
- Decommission — remove old version once you’re satisfied
The timeline feels slow compared to a standard software deployment, but agentic workflows accumulate errors over time in ways that fast rollouts hide. The patience compounds with the complexity of the workflows you’re running.