TL;DR:

  • AI agents for data quality monitoring work best as a layered system: deterministic rule-based checks handle schema validation and freshness; statistical baselines handle volume and distribution drift; LLMs handle classification, root cause summarisation, and alert triage.
  • The core agent loop — monitor, detect, classify, alert or remediate — can be built with existing tools like Great Expectations, dbt tests, and Soda Core, with an LLM layer added for the classification and communication steps.
  • Auto-remediation is valuable but requires careful scoping; start with alert enrichment and suggested fixes before giving agents write access to production pipelines.

Bad data is expensive in a way that’s easy to underestimate until something catastrophic happens. A model trained on stale features. A financial report built on duplicated rows. A customer-facing dashboard showing null values because an upstream schema changed without notice. These problems are common, they compound quickly, and they’re disproportionately hard to catch through manual review as data volumes scale.

AI agents are increasingly capable of handling the monitoring work that used to require dedicated data engineering attention — not by replacing the humans who design data systems, but by operating the continuous watch layer that most teams build inconsistently or not at all.

This piece covers what a well-designed data quality agent actually does, which tools and frameworks are worth building on, and where LLMs genuinely help versus where rule-based checks are more appropriate.

What Data Quality Agents Actually Monitor

A useful frame is to think of data quality across five dimensions, each requiring slightly different detection methods:

Schema drift is the simplest to detect and often the most damaging. Upstream sources add, remove, or rename columns; data types change; nullable constraints flip. A schema validation check runs on every pipeline load and fails immediately if the incoming data doesn’t match the expected contract. This is pure rule-based work — no ML or LLM needed, just a defined expectation and a binary pass/fail.

Statistical anomalies in volume, distribution, and value ranges are where baseline modelling starts to matter. If your orders table typically receives 50,000–80,000 rows per day and today’s load contains 3,000, something is wrong — but the threshold isn’t fixed, because it varies by day of week, seasonality, and business events. A simple z-score or rolling average model catches most of these without requiring anything sophisticated.

Freshness and completeness monitoring tracks whether data arrived on schedule and whether expected records are present. This is largely deterministic: define an SLA (data should land by 08:00 UTC), check the max ingestion timestamp, alert if it’s outside the window. Completeness checks are similar — if you expect every order to have a corresponding customer record and 5% don’t, that’s a referential integrity failure.

Value-level quality — whether individual field values are plausible, consistent, or within expected ranges — is where more nuanced checks come in. Is this email address formatted correctly? Does this transaction amount fall within a reasonable range? Are there impossible values (negative ages, future birth dates)? These can mostly be handled with rule sets, though the rules need to be maintained as business logic evolves.

Semantic anomalies — where the data is technically valid but contextually wrong — are where LLMs start to add value that rule-based systems struggle to provide.

The Core Tooling Layer

Most teams building data quality agents should start with existing frameworks rather than rolling their own detection logic.

Great Expectations remains the most mature open-source option. You define expectations — essentially assertions about your data — and run them against datasets on a schedule or as part of a pipeline. A basic checkpoint configuration looks like this:

# great_expectations/checkpoints/orders_daily.yml
name: orders_daily_checkpoint
config_version: 1.0
class_name: Checkpoint
run_name_template: "%Y%m%d-%H%M%S-orders-daily"
validations:
  - batch_request:
      datasource_name: warehouse
      data_connector_name: default_inferred_data_connector_name
      data_asset_name: orders
    expectation_suite_name: orders.critical
action_list:
  - name: store_validation_result
    action:
      class_name: StoreValidationResultAction
  - name: send_slack_notification
    action:
      class_name: SlackNotificationAction
      slack_webhook: ${SLACK_WEBHOOK_URL}
      notify_on: failure

Great Expectations handles the detection and reporting; your agent layer sits above it, consuming validation results and deciding what to do next.

dbt tests are the right choice if your transformation layer is already in dbt. Generic tests cover the most common cases:

# models/schema.yml
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('customers')
              field: id
      - name: amount_gbp
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 100000

Custom singular tests handle business-specific rules that generic tests don’t cover. dbt’s --store-failures flag writes failing rows to the warehouse, which gives your agent something concrete to inspect.

Soda Core offers a YAML-based check syntax that’s readable and integrates well with orchestration tools like Airflow and Dagster:

# checks.yml
checks for orders:
  - row_count between 40000 and 120000:
      name: "Daily order volume within expected range"
  - freshness(created_at) < 4h:
      name: "Orders data not stale"
  - missing_count(customer_id) = 0:
      name: "No orphaned orders"
  - duplicate_count(order_id) = 0:
      name: "Order IDs unique"

Monte Carlo Data and Acceldata are managed observability platforms that handle the statistical baseline modelling automatically — they learn your data’s normal patterns and alert on deviations without you needing to define thresholds manually. They’re more expensive than open-source options but significantly reduce the engineering overhead of baseline management.

WhyLogs (from whylabs) is worth knowing for ML-adjacent pipelines — it generates statistical profiles of datasets and detects distribution drift between training and serving data, which is a specific problem that general data quality tools don’t handle well.

Building the Agent Loop

The agent loop for data quality monitoring has four stages:

1. Monitor — run checks on a schedule (or triggered by pipeline completion). This is the tooling layer: Great Expectations, dbt tests, Soda. The output is a structured result set: which checks passed, which failed, and what the failing data looks like.

2. Detect — identify which failures are novel, which are known flaky checks, and which represent genuine severity. A simple statistical comparison against historical failure rates handles most of this. A Python baseline check might look like:

import pandas as pd
from scipy import stats

def is_anomalous_volume(current_count: int, history: list[int], threshold: float = 3.0) -> bool:
    """Return True if current_count is more than `threshold` std devs from the historical mean."""
    if len(history) < 14:  # Require at least two weeks of history
        return False
    z_score = abs((current_count - pd.Series(history).mean()) / pd.Series(history).std())
    return z_score > threshold

# Usage
recent_counts = get_recent_row_counts("orders", days=90)
today_count = get_today_row_count("orders")
if is_anomalous_volume(today_count, recent_counts):
    trigger_investigation(table="orders", count=today_count, baseline=recent_counts)

3. Classify — determine what kind of failure this is, what the likely cause is, and what severity to assign. This is where LLMs earn their place. A rule-based classifier can handle straightforward cases (schema change = schema drift, volume drop on Monday = likely weekend lag). An LLM classifier handles the ambiguous ones: a failure in customer_revenue that correlates with a schema change in transactions three hops upstream, or a value distribution shift that might be a data issue or might reflect a legitimate business event.

The LLM prompt for classification should include: the check that failed, the failing rows (sampled), recent pipeline run history, any related upstream failures, and a description of what the table represents. With that context, a capable model can produce a useful diagnosis in most cases.

4. Alert or remediate — decide what to do with the classified failure. This is a policy decision, not an AI decision: some failures should page on-call, some should open a Jira ticket, some should trigger an automatic backfill, some should just be logged. The agent enforces the policy; it doesn’t make it up.

Where LLMs Add Real Value (and Where They Don’t)

Use LLMs for:

  • Summarising failures into human-readable incident descriptions (“The orders table is missing 47,000 rows for 3 August. This correlates with a failed Fivetran sync from Shopify at 02:14 UTC.”)
  • Classifying ambiguous failures where context matters
  • Generating suggested remediation steps based on failure type and history
  • Triaging alert fatigue — given ten simultaneous failures, which ones are related and which represent the highest business impact?
  • Writing data quality rules in natural language and translating them to Great Expectations or Soda syntax

Use rule-based checks for:

  • Schema validation — always deterministic, always binary
  • Freshness checks — a timestamp comparison needs no LLM
  • Uniqueness and null checks — these are O(n) operations on your data, not interpretation problems
  • Statistical anomaly detection — baseline models are better at this than LLMs, which have no access to your historical data unless you inject it
  • High-volume, low-latency checks where inference latency would be a bottleneck

The temptation to pass everything through an LLM “just in case” is worth resisting. LLMs add latency, cost, and non-determinism. For checks that have unambiguous pass/fail semantics, deterministic code is faster, cheaper, and easier to audit.

Auto-Remediation: Proceed Carefully

The most powerful configuration is an agent that doesn’t just detect and alert but takes corrective action — triggering a backfill, reverting a schema change, pausing a downstream pipeline to prevent bad data propagating. This is achievable but requires careful scoping.

Start with read-only remediation: the agent identifies the issue, describes the fix, and opens a ticket or sends a suggested command. That’s already a significant time saving over manual investigation.

Move to supervised write access only after you’ve built confidence in the classification accuracy. An agent that automatically triggers a backfill for every volume anomaly will create more problems than it solves if its anomaly detection has a 10% false positive rate.

The right escalation model: auto-resolve known safe issues (retry a failed check, re-run a specific dbt model), require human approval for anything that modifies source data or disables a downstream consumer, and always write an audit log of every action taken.

Putting It Together

A production-grade data quality agent isn’t a single model or a single tool — it’s a composed system: a framework layer (Great Expectations, dbt, Soda) that handles detection; a statistical layer that handles baseline comparison and anomaly scoring; an LLM layer that handles classification, summarisation, and suggested remediation; and a policy layer that decides what actions to take based on the classification.

The engineering work is mostly integration and policy design, not ML research. Most of the components exist as open-source tools or managed services. The value an agent adds over a traditional alerting system is in the classification and communication steps — turning a wall of check failures into an intelligible incident summary, and routing it to the right person with enough context to act quickly.

That’s a solvable problem with current tooling, and the teams that invest in building it stop spending engineering time on data firefighting and start spending it on data systems that don’t need fighting.