Competitive intelligence used to mean paying an analyst to spend hours trawling competitor websites, collating pricing tables in spreadsheets, and writing monthly summaries that were already stale by the time leadership read them. The same job is now a good fit for an LLM-backed agent: it can do the web research, synthesise what it finds, and drop structured output into your tooling of choice on a schedule.

This walkthrough covers the architecture for a real competitive intelligence agent, the tooling decisions that make it reliable, and the failure modes to design around.

What the Agent Actually Does

The finished agent operates on a defined competitor list and a set of intelligence categories you care about. On each run it:

  1. Fetches current content from competitor sites (pricing pages, feature announcement blogs, job listings)
  2. Compares findings against a stored baseline from the previous run
  3. Flags changes that cross a relevance threshold
  4. Writes a structured summary — pricing changes, new features, notable messaging shifts, significant hiring moves

You end up with a Markdown or JSON report you can pipe into Slack, a CRM, or a Notion database. The key value isn’t real-time monitoring — it’s consistent, structured coverage across many competitors without human bandwidth.

Tool Stack

Web search: Tavily, Exa, or Brave Search API are the reliable choices here. General-purpose search (Google via SerpAPI) works but noise-to-signal ratio is high for competitive research; purpose-built search APIs with better result filtering produce cleaner context for the LLM.

Scraping for structured data: Playwright or browser-use for JavaScript-heavy pages where an API call won’t get you rendered content. For simpler pages, httpx + selectolax is faster and lighter.

LLM: Claude or GPT-4o for synthesis. For bulk processing across many URLs, the cost per run matters — consider using a smaller model (Haiku, GPT-4o mini) for initial filtering and a full model only for final report generation.

Orchestration: LangGraph, CrewAI, or raw Python with a simple task queue. The agent doesn’t need complex multi-hop reasoning — a sequential pipeline with branching based on what changed is sufficient and much easier to debug.

Storage: A SQLite or Postgres table storing baseline states per competitor. Key fields: competitor name, page URL, content hash, last-fetched date, extracted summary. When the hash changes, trigger detailed comparison.

Architecture

from dataclasses import dataclass
from typing import Optional
import hashlib, json, asyncio

@dataclass
class CompetitorPage:
    competitor: str
    url: str
    category: str  # "pricing", "features", "blog", "careers"
    last_hash: Optional[str] = None
    last_summary: Optional[str] = None

async def run_intel_cycle(pages: list[CompetitorPage], llm, search_client):
    changes = []
    
    for page in pages:
        content = await fetch_page(page.url)
        current_hash = hashlib.sha256(content.encode()).hexdigest()
        
        if current_hash == page.last_hash:
            continue  # no change, skip LLM call
        
        summary = await llm.generate(
            f"Summarise what changed on this {page.category} page for {page.competitor}. "
            f"Previous summary: {page.last_summary}\n\nCurrent content:\n{content[:8000]}"
        )
        
        changes.append({
            "competitor": page.competitor,
            "category": page.category,
            "url": page.url,
            "summary": summary,
        })
        
        # update stored state
        page.last_hash = current_hash
        page.last_summary = summary
    
    if changes:
        return await generate_report(changes, llm)
    return None

The hash check is the key efficiency win. Most runs will see no changes for most pages — you only spend tokens on the LLM for pages where content actually changed.

Handling the Hard Cases

Paywalled or login-required content: For competitor pricing hidden behind registration, structured web search (“site:competitor.com pricing 2026”) surfaces cached or public discussion. It’s imperfect but workable. If you need direct access, consider maintaining logged-in browser sessions per competitor using saved Playwright state.

JavaScript-heavy SPAs: Pricing pages and product feature tables are often rendered client-side. Static fetching returns an empty div. Use Playwright’s page.wait_for_load_state("networkidle") before extracting content, or use browser-use which handles this natively.

Noise from cookie banners, pop-ups, footer changes: Pre-process extracted content to strip navigation, footer, and cookie consent text before hashing. Otherwise you’ll flag changes every time a competitor updates their footer privacy link.

Rate limiting and anti-bot measures: Space requests across competitors — don’t hammer one site. Use respectful headers, randomise delays between requests (2–5 seconds), and consider rotating user agents. If a site actively blocks scraping, fall back to search API results for that competitor.

Structuring the Intelligence Output

The report format matters as much as the agent logic. Aim for structured output the LLM can reliably produce:

## Competitive Intelligence Report — 2026-07-12

### Pricing Changes
- **Acme Corp** updated their Pro plan from £49/mo to £59/mo. Starter plan unchanged.
- **Globex SaaS** removed the free tier from public pricing page (now "contact us").

### New Feature Announcements
- **Initech** announced native Slack integration (blog post 2026-07-10).

### Messaging Shifts
- **MomCorp** has shifted homepage hero copy from "fastest solution" to "most secure platform" — likely responding to recent security incidents in the sector.

### Hiring Signals
- **Acme Corp** posted 4 new ML engineer roles this week, all focused on inference optimisation.

Use Pydantic models or structured output schemas to get consistent JSON out of the LLM, then render to Markdown for the actual report. Structured intermediate output makes the pipeline auditable — you can log what the LLM extracted and replay specific sections without re-fetching.

Scheduling and Alerting

For most businesses, daily or weekly runs are enough. Competitive pricing doesn’t change hourly. Set up a cron job (or APScheduler for in-process scheduling) and deliver the report only when there are changes:

# only send if there's something to report
if report and report.has_meaningful_changes:
    await slack_client.post_message(channel="#competitive-intel", text=report.to_markdown())

Immediate alerting (running every few hours) makes sense for specific high-stakes events — a competitor announcing funding, a major pricing change, a product launch. Use RSS feeds or search API alerts for these rather than polling pages constantly.

What This Doesn’t Replace

The agent produces raw intelligence, not analysis. It can tell you that a competitor changed their pricing page — it can’t tell you whether the change signals a response to your pricing, a move upmarket, or a promotional test. That interpretation still needs a human who understands the business context.

The right framing: the agent handles data collection and first-pass summarisation so your team’s analytical time is spent on interpretation rather than legwork.

References