TL;DR:

  • Groq’s Language Processing Unit (LPU) delivers 800–1,200+ tokens/second on major open models — 10–20x faster than typical GPU inference
  • High throughput changes agent loop design: tight multi-step chains become practical that were too slow on GPU APIs
  • Groq works best as a speed layer for latency-sensitive steps; blend with other providers for cost and capability balance

When people talk about making AI agents faster, they usually mean smarter prompting, fewer steps, or better caching. Groq is different — it’s a hardware approach to the same problem. The company built a custom chip called the Language Processing Unit (LPU) specifically for the sequential token generation bottleneck in transformer inference. The result is inference speeds that, on benchmarks and in practice, are genuinely faster than anything available on GPU-based cloud infrastructure.

For agent builders, this isn’t just a benchmark number. Throughput affects how you architect multi-step workflows.

What Groq’s LPU Actually Delivers

On Groq’s current production hardware, Llama 3.3 70B runs at around 800 tokens per second. Llama 3.1 8B and similar smaller models hit 1,200–1,500 tokens/second. Meta’s Llama 4 models, available on Groq since April 2026, run at similar speeds with the larger context windows those models provide.

For comparison, a GPU-based inference provider typically delivers 80–150 tokens/second on 70B-class models under real load. The difference is roughly 10x.

What this means in practice: a response that takes 3 seconds on a GPU endpoint takes under 400ms on Groq for the same output length. In a five-step agent loop where each step generates 200 tokens of reasoning before acting, you’ve gone from 15 seconds of pure inference time to under 2 seconds.

Where This Changes Agent Architecture

Latency matters most in two agent scenarios: user-facing loops and tight sequential chains.

User-facing conversational agents are the obvious case. If your agent needs to think visibly before responding — generating a scratchpad, looking something up, then composing an answer — the difference between 3-second and 400ms steps is the difference between a usable product and one that feels slow. Groq lets you include more reasoning steps without the UX cost.

Sequential tool-calling chains benefit for a different reason. When each step depends on the previous one (call tool A, parse result, decide which of B/C/D to call next, parse that, compose final answer), you can’t parallelize. You’re waiting on each step. Cutting per-step inference latency from 3s to under 1s halves the perceived response time on a five-step chain even if tool execution takes fixed time.

There’s a subtler architectural benefit too: speculative branching becomes affordable. If inference is cheap in time, you can generate multiple candidate next steps and pick the best one — a technique that’s too slow to justify on GPU endpoints but works at Groq speeds for most use cases.

The Groq API

The API is OpenAI-compatible. If you’re using the OpenAI Python SDK, switching to Groq is changing the base URL and dropping in a Groq API key:

from groq import Groq

client = Groq(api_key="gsk_...")

response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful agent."},
        {"role": "user", "content": "Analyse this data and suggest three next actions: ..."}
    ],
    temperature=0.3,
    max_tokens=500
)

Available models as of mid-2026 include:

  • llama-3.3-70b-versatile — 128K context, best capability
  • llama-3.1-8b-instant — fastest, lowest cost, good for classification and routing steps
  • llama-4-scout-17b-16e-instruct — Llama 4 architecture with long context
  • gemma2-9b-it — Google’s Gemma 2 9B
  • mixtral-8x7b-32768 — Mixtral MoE with 32K context

Groq also supports tool use (function calling) with the same syntax as OpenAI, which is essential for agentic workflows.

Pricing and Where It Fits in a Multi-Provider Stack

Groq’s pricing as of 2026: approximately $0.59/million input tokens and $0.79/million output tokens for Llama 3.3 70B. That’s competitive but not the cheapest — you’re paying a modest premium for the latency advantage. Llama 3.1 8B on Groq runs at around $0.05/$0.08, which is very cheap for high-volume routing steps.

The practical pattern most production teams use: a tiered stack where latency-sensitive steps go to Groq and capability-heavy steps go elsewhere. A research agent might use Groq for fast query decomposition and result classification (where 8B or 70B is more than capable), and route deep synthesis steps requiring 200K context or complex reasoning to Anthropic Claude or Google Gemini 2.5.

This is straightforward to implement with LiteLLM or OpenRouter as the routing layer:

import litellm

# Fast step: route decomposition to Groq
result = litellm.completion(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": decompose_prompt}]
)

# Synthesis step: route to Claude for long-context reasoning
synthesis = litellm.completion(
    model="anthropic/claude-opus-4-5",
    messages=[{"role": "user", "content": synthesis_prompt}]
)

Limitations Worth Knowing

Groq’s context windows are currently shorter than GPU-hosted models’ maximums — most models cap at 128K tokens rather than the 500K–1M contexts available on some GPU platforms. For document-heavy workflows, this is a constraint.

Rate limits on the free tier are restrictive for production use. The paid tiers offer 500–14,400 requests per minute depending on model, which is sufficient for most agent deployments but worth checking against your expected concurrent agent count.

The model selection is smaller than OpenAI or Anthropic’s offerings. You’re working with open-weight models. For most agent tasks — classification, routing, reasoning over structured data, tool call generation — the quality is more than adequate. For tasks that genuinely require frontier model capability, Groq isn’t the right choice.

Getting Started

Create an account at console.groq.com. Free tier includes 14,400 requests/day on smaller models and 1,000 on 70B. The API key is available immediately. Start by profiling your existing agent’s step latencies — the steps that are purely sequential and latency-bound are the right candidates to migrate first.

For most multi-step agent workflows, even moving one or two intermediate reasoning steps to Groq produces a measurable improvement in end-to-end response time. It’s one of the cheapest latency wins available in the current agent tooling ecosystem.