TL;DR:

  • AI agents for IT operations (ITOps) are moving from research demos to production deployments, with teams using them to run initial incident triage, query observability platforms, execute runbook steps, and summarise alerts.
  • The most successful deployments are narrow in scope: an agent that handles a specific class of incident (disk space, OOM kills, Kubernetes pod restarts) rather than a general-purpose “run any runbook” agent.
  • The critical design decisions are about trust boundaries — what the agent can execute autonomously versus what requires human approval — and about what happens when the agent gets it wrong.

The traditional incident response workflow looks something like this: an alert fires, a page goes out to an on-call engineer, the engineer wakes up at 3am, spends 20 minutes figuring out what the alert actually means, queries a few dashboards, realises it’s a known issue with a documented fix, executes the runbook steps, confirms the fix worked, and goes back to sleep. The whole thing takes 45 minutes, and 40 of those minutes are mechanical.

AI agents can handle a significant portion of that mechanical work. They can query your observability platform, pull recent logs, check related services, and walk through the first several runbook steps — all before a human engineer is paged. If the agent can resolve the issue autonomously, the engineer never wakes up. If it can’t, the engineer gets paged with a summary of what the agent already found and what it already tried, rather than starting from scratch.

This isn’t science fiction. Teams at companies like Slack, LinkedIn, and Cloudflare have published about variations of this architecture. The challenge is building it in a way that’s reliable enough to trust at 3am.

The Core Architecture

An ITOps agent typically has access to several tool categories:

Observability tools. Read access to your monitoring and observability stack: querying Datadog, Grafana, Prometheus, PagerDuty, or whatever you use. The agent should be able to fetch metric data for a time window, pull recent logs for a service, and check the status of related dependencies.

Infrastructure tools. Read access to infrastructure state: Kubernetes cluster status, EC2 instance metadata, RDS instance health, recent deployment history from your CI/CD system. Many teams also give agents restricted write access for safe operations — restarting a specific pod, scaling a deployment, flushing a cache.

Runbook access. Either a structured representation of runbooks (JSON or YAML describing steps and conditions) or access to runbook documentation that the agent can parse and execute. Structured runbooks are much more reliable than asking an agent to interpret prose documentation.

Communication tools. Write access to Slack or PagerDuty for posting status updates, escalating to humans, or logging what it did.

The agent loop: alert fires, agent receives the alert payload, queries relevant monitoring data, evaluates against known runbook triggers, executes allowed autonomous steps if conditions match, posts a summary, and escalates to the on-call human with context if the issue isn’t resolved.

What to Automate and What to Protect

The biggest mistake teams make with ITOps agents is granting too much autonomy too early. Here’s a practical framework:

Safe to automate fully: Actions that are easily reversible, affect only one service, and have a clear success condition. Restarting a stuck pod. Scaling up a deployment that’s running out of headroom. Rotating a cache. These are good candidates for full automation because the blast radius is small and the action is undoable.

Automate with approval gate: Actions that affect multiple services, involve infrastructure costs, or are difficult to reverse. Terminating an EC2 instance, modifying a database parameter, changing DNS records. The agent can prepare and propose these actions, but a human confirms before execution.

Inform but don’t act: Actions that require contextual judgment beyond the agent’s knowledge. Security-related responses, changes to production configurations, anything involving customer data. The agent surfaces the information and recommendation; a human decides.

The most reliable way to implement this is at the tool level, not the prompt level. Don’t rely on the agent’s instructions to prevent it from taking a dangerous action. Build the dangerous actions into a tool that requires an explicit human approval token before executing. If the agent can’t call the dangerous tool without that token, prompt injection and similar failure modes can’t override the restriction.

A Practical Starting Pattern: Disk Space Incidents

Rather than building a general incident response agent, start with a single incident type that has high volume, a well-understood runbook, and a bounded blast radius.

Disk space alerts are a good first candidate. High volume (disk fills happen constantly), clear runbook (check what’s using space, identify whether it’s logs, temp files, or application data, take appropriate action), and bounded blast radius (deleting log files on a server doesn’t affect production services).

Here’s the agent loop for this:

  1. Alert fires: /dev/sda1 is 95% full on hostname web-prod-03
  2. Agent queries: df -h web-prod-03, du -sh /* | sort -rh | head -20 via SSH or your infrastructure tool
  3. Agent checks: is this a log accumulation pattern? Is there a log rotation issue? Is temp space from a recent deployment?
  4. Agent executes (autonomously): rotates logs if log rotation is the issue, removes known-safe temp files
  5. Agent escalates with context if the root cause isn’t a known pattern

This is boring, but boring is exactly what you want for your first ITOps agent. Once it’s working reliably, you add another incident type.

Observability Integration

The agent needs structured access to your observability data, not just the ability to query dashboards. A few practical approaches:

Prometheus/Grafana: Use the Prometheus HTTP API or Grafana’s HTTP API to query metrics programmatically. Define the queries as tools the agent can call with parameters (service name, time window, metric name).

Datadog: Datadog’s API allows querying metrics, logs, and events. Build wrappers around the specific queries your runbooks need, rather than giving the agent raw API access.

PagerDuty: For alert context, the PagerDuty API provides incident details, service information, and recent alert history. This gives the agent context about whether this is a first-time occurrence or a recurring pattern.

OpenTelemetry traces: If your services emit traces, trace data is often the most useful signal for diagnosing issues. Build tooling to query your trace backend (Jaeger, Tempo, Honeycomb) for recent errors on a given service.

Tools That Help

Opsgenie and PagerDuty both offer API access and webhook integrations suitable for triggering agent workflows from alerts.

PagerDuty’s AI incident response features provide some built-in automation that’s worth evaluating before building from scratch. It may cover your use case without custom agent development.

LangChain and LangGraph are common choices for building the agent loop, given their observability integrations and the ability to define custom tools. Temporal is worth considering for the orchestration layer if your runbooks involve multi-step, long-running processes where reliability matters.

Incident.io has been building AI-powered incident management features that sit alongside (rather than replacing) your custom agents, and is worth evaluating for the coordination and communication layer.

The Reliability Question

The honest answer is that ITOps agents aren’t yet reliable enough to fully replace human judgment for novel incidents. Where they work well is in the 70-80% of incidents that are repeating patterns with known solutions. Those are exactly the incidents that shouldn’t require waking someone up at 3am.

For novel incidents — the ones where something genuinely unexpected has gone wrong — the agent’s role is better scoped as a first responder that gathers context quickly and surfaces it to a human, rather than as an autonomous problem-solver.

Build for that scope first. The agent that handles your top five incident types reliably is more valuable than an agent that handles all incident types unreliably. Start narrow, measure mean time to resolution and false positive rates, and expand scope as confidence grows.