TL;DR:

  • AgentOps tracks every LLM call, tool use, token spend, and error at session level, giving you replay and cost visibility that standard APM tools lack.
  • It integrates directly with LangChain, CrewAI, AutoGen, and the OpenAI and Anthropic SDKs with minimal setup.
  • The free tier covers enough for development and small production workloads, but you’ll want to understand the session limits before scaling up.

Here’s the thing about running AI agents in production: the things that go wrong are almost never the things you’d expect. It’s not a server going down or a network timeout that causes the most pain. It’s a tool call that silently returned a malformed result, an agent that looped through the same steps four times before stopping, or a session that spent three times your estimated token budget because of a subtle prompt regression. Standard application monitoring doesn’t catch any of that. It sees an HTTP 200 and considers the job done.

That’s the gap AgentOps is designed to fill.

AgentOps is an observability platform built specifically for the structure of AI agent workloads. Rather than treating your agent as a black box that receives a request and returns a response, it instruments the internal mechanics: every LLM call with its input/output and latency, every tool invocation with its arguments and return values, the full sequence of steps within a session, token counts, costs, and whether the agent finished cleanly or errored out. The result is something closer to session replay than traditional logging.

Why does session replay matter? Because debugging an agent failure without it is genuinely miserable. You get a stack trace, maybe a final error message, and then you’re reverse-engineering what the agent decided to do and why. With session replay, you can step through the exact sequence of calls, see what the model was passed at each stage, and identify exactly where reasoning went off the rails. That’s a qualitative improvement in debugging experience.

Generic APM tools like Datadog, New Relic, or even OpenTelemetry-based setups can capture latency and error rates at the API level. They’re not useless. But they have no concept of what a “session” means in the agent sense, they can’t tell you your tool call success rate across 10,000 runs, they won’t break down your LLM provider latency by model and call type, and they definitely won’t give you per-session cost attribution. AgentOps does all of that out of the box.

Getting set up

Integration is deliberately lightweight. Here’s what a basic setup looks like with the OpenAI SDK in Python:

import agentops
from openai import OpenAI

agentops.init(api_key="your-agentops-api-key")

client = OpenAI()

@agentops.record_action("fetch_weather")
def fetch_weather(city: str) -> str:
    # your tool implementation
    return f"Weather in {city}: 18°C, partly cloudy"

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's the weather in Manchester?"}],
)

agentops.end_session("Success")

The agentops.init() call patches the OpenAI client automatically, so every chat.completions.create call is captured without you needing to instrument each one individually. The @agentops.record_action decorator wraps your tool functions and records their inputs, outputs, and timing. end_session closes the session and marks it with a terminal state.

That’s genuinely most of what you need for basic coverage. You get LLM call traces, tool call records, and session-level metadata in the AgentOps dashboard from that snippet alone.

Framework integrations

If you’re using LangChain, the integration is a single callback handler: AgentOpsHandler drops into any chain or agent executor. CrewAI has native AgentOps support built in; you set the API key as an environment variable and it instruments automatically. AutoGen requires a slightly more manual setup, wrapping the conversation manager, but the AgentOps docs cover it well.

The Anthropic SDK integration works similarly to the OpenAI path. You initialise AgentOps, and it patches the client’s message creation methods to capture traces. If you’re running multi-agent systems where one Claude instance calls another, each LLM call is captured individually and grouped under the parent session.

Trace-level vs session-level monitoring

It’s worth understanding the distinction between these two, because they serve different debugging needs.

Trace-level monitoring is about individual LLM calls: what prompt went in, what came back, how long it took, which model was used, how many tokens were consumed. This is your go-to when you’re investigating a specific bad output or a latency spike.

Session-level monitoring is about the full agent run: how many steps did it take, what tools were called and in what order, what was the total cost, did it succeed or fail, and how does this session compare to the average. This is what you use for understanding systemic issues, regressions after a prompt change, or unusual cost spikes across a time window.

AgentOps gives you both views, and you can drill from session level down to individual traces, which is exactly the workflow you want when something goes wrong in production.

The free tier and its limits

To be honest, the free tier is reasonably generous for development and small-scale production. You get up to 10,000 events per month and access to the core dashboard features including session replay and trace viewer. Once you’re running volume production workloads, you’ll hit the limits fairly quickly, so it’s worth factoring the paid tier into your infrastructure budget before you scale.

The cost-per-session visibility is itself a compelling reason to pay for the tool, because it tends to pay for itself by surfacing runaway sessions that you’d otherwise not notice until you got your LLM provider invoice.

Why not just log everything yourself?

Fair question. You can absolutely build your own logging layer around your LLM calls and tool functions. Plenty of teams do. The problem is that it takes real engineering time to build something that captures everything you need, doesn’t add meaningful latency, handles concurrent sessions correctly, and gives you a useful query interface on the collected data. AgentOps is that system, already built and maintained. Using it means your engineers spend time on your actual product rather than on observability infrastructure.

For any team running agents in production at non-trivial volume, specialised agent monitoring isn’t a nice-to-have. It’s genuinely the difference between operating your system with confidence and flying blind.