TL;DR:
- Incident response is one of the highest-ROI targets for AI agents — the workflows are well-defined, the data is structured, and time-to-resolution is directly measurable
- A useful incident agent does three things: triages the alert, executes safe diagnostic steps autonomously, and hands off to a human with context pre-assembled
- LangGraph is the right framework for this because incident response is inherently stateful — you need the agent to remember what it tried before escalating
Incident response has all the properties that make a workflow good for AI automation: it’s repetitive, it’s time-sensitive, the inputs are structured (alert payloads, metrics, logs), and the cost of being slow is measurable in money and reputation. And yet most teams are still doing it manually, with an on-call engineer being paged at 2am to run the same five diagnostic commands they always run.
Here’s the thing — you probably don’t want an AI agent resolving incidents autonomously. What you want is an agent that does the first fifteen minutes of incident response for you: triage, initial investigation, context assembly. Then it hands off to a human with everything they need to make a decision quickly. That’s achievable today, and it’s where the real productivity gain is.
What the Agent Actually Does
Before writing any code, it’s worth being precise about the scope. The incident response agent we’re building handles the triage and investigation phase:
- Receives an alert from PagerDuty (or your alerting tool of choice)
- Queries your observability stack for relevant signals — metrics from Prometheus or Datadog, recent errors from your logging platform, recent deployment events from your CI/CD system
- Checks a runbook to see if there’s a documented diagnostic procedure for this alert type
- Executes safe read-only diagnostic steps (running queries, calling APIs, checking service health endpoints)
- Produces a structured incident summary and pages the on-call engineer with context pre-assembled
What it doesn’t do: make changes to production infrastructure, execute any write operations, or close an incident without human sign-off. That’s a deliberate constraint. The agent’s job is to get the human to a decision point faster, not to remove the human from the loop.
Why LangGraph for This
Incident response is stateful in a way that simple chain-of-thought prompting doesn’t handle well. The agent needs to remember that it already checked the database connection pool, that it already queried the last three deployments, and that the Prometheus query returned an anomaly it needs to investigate further. If the agent forgets its prior steps, it either repeats work or misses context.
LangGraph models agent execution as a directed graph with persistent state. Each node in the graph is a function — query Prometheus, check PagerDuty history, search the runbook, page the engineer — and the state is threaded through the graph so each step has access to everything that happened before it. That’s the right mental model for incident investigation.
LangGraph v1.2.7 (current stable as of July 2026) is also fully decoupled from LangChain, so you can use it with any LLM provider without pulling in the rest of the LangChain ecosystem.
The Core Architecture
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode
from typing import TypedDict, Annotated
import operator
class IncidentState(TypedDict):
alert: dict # Raw alert payload
metrics: Annotated[list, operator.add] # Accumulated metric queries
logs: Annotated[list, operator.add] # Accumulated log samples
runbook_steps: list # Runbook steps if found
findings: Annotated[list, operator.add] # Investigation findings
severity: str # Assessed severity
summary: str # Draft incident summary
def build_incident_graph():
graph = StateGraph(IncidentState)
graph.add_node("triage", triage_alert)
graph.add_node("query_metrics", query_observability)
graph.add_node("check_runbook", check_runbook)
graph.add_node("investigate", run_investigation)
graph.add_node("summarise", draft_summary)
graph.add_node("page_oncall", notify_engineer)
graph.set_entry_point("triage")
graph.add_edge("triage", "query_metrics")
graph.add_edge("query_metrics", "check_runbook")
graph.add_edge("check_runbook", "investigate")
graph.add_edge("investigate", "summarise")
graph.add_edge("summarise", "page_oncall")
graph.add_edge("page_oncall", END)
return graph.compile()The triage_alert node extracts the key fields from the incoming alert payload, classifies the affected service and alert type, and sets the initial severity. It’s mostly structured data extraction, which you can do reliably without an LLM for well-defined alert formats.
The Investigation Step
This is where the LLM earns its keep. Given the metrics and logs collected in prior steps, and the runbook guidance (if any), the agent needs to reason about what the signals mean and decide whether it has enough information to summarise or needs to run more queries.
async def run_investigation(state: IncidentState) -> IncidentState:
context = {
"alert": state["alert"],
"metrics": state["metrics"],
"logs": state["logs"],
"runbook": state.get("runbook_steps", [])
}
response = await llm.ainvoke([
SystemMessage(content=INVESTIGATION_PROMPT),
HumanMessage(content=json.dumps(context))
], tools=diagnostic_tools)
# Handle tool calls if the LLM decides it needs more data
findings = []
if response.tool_calls:
for call in response.tool_calls:
result = await execute_diagnostic_tool(call)
findings.append(result)
return {"findings": findings, "severity": extracted_severity}The diagnostic_tools available to the agent at this step are strictly read-only: get_service_dependencies, check_error_rate_last_hour, list_recent_deployments, query_database_pool_stats. No writes, no restarts, no config changes.
The Handoff Summary
The summary the agent produces for the on-call engineer should be opinionated — a concise assessment of what’s happening, not a dump of raw data. The prompt for the summary step should explicitly ask for: the most likely cause, supporting evidence, what’s been ruled out, suggested first steps, and current blast radius.
A good handoff summary means the engineer can read it in under a minute and know what to do. If the agent produces a three-page narrative, it’s slower than just paging the engineer directly.
Runbook as RAG
Most teams have runbooks — either in Confluence, Notion, or a wiki of some kind. The runbook lookup step works best as a lightweight RAG query: embed your runbooks at indexing time, then at investigation time query for the most semantically relevant runbook given the alert type and service.
Don’t try to embed the entire runbook library in the system prompt. That approach works fine for small runbook collections but degrades quickly as you add more content. A vector search against embedded runbook chunks, with the top three results passed to the investigation step, is more reliable and much cheaper to run.
Guardrails That Matter
Two guardrails that aren’t optional:
Timeout everything. Diagnostic tool calls should have aggressive timeouts — five seconds, not thirty. If a query is slow, that’s itself a signal worth logging, and a hung tool call shouldn’t block the agent from escalating. An agent that takes twelve minutes to produce a summary is worse than no agent.
Log the full trace. Every investigation the agent runs should produce a structured trace — what it queried, what it found, what reasoning it applied. This isn’t just useful for postmortems; it’s essential for debugging the agent itself. An incident response agent that produces a wrong or confusing summary needs to be debuggable.
Getting Started
The minimal viable version of this agent connects to PagerDuty webhooks, queries one observability source (start with your logging platform — it’s usually the richest signal), and produces a Slack message with the summary. That alone — automated first-look triage delivered to Slack before the engineer opens their laptop — is valuable enough to justify the build.
From there, adding metrics queries, runbook lookup, and deployment correlation each add incremental value without changing the core architecture. Build the integration layer first, then expand the investigation scope.
The incident response agent won’t replace your SRE team. It’ll give them their 2am back.