Building Local AI Agent Workflows with Ollama and Open-Source LLMs

TL;DR:

• Ollama makes running production-quality open-source LLMs on local hardware as simple as ollama pull llama3.3
• Local models are now capable enough for most agent workloads — Qwen2.5-Coder and Llama 3.3 70B can handle tool calling, structured output, and multi-step reasoning
• A local agent workflow with LangChain + Ollama can match cloud-hosted performance on most tasks at zero ongoing cost

The economics of AI agents shifted in 2025 and 2026. Cloud API costs that were once an afterthought at low volumes become a significant line item when agents run hundreds of tool calls per session, process long documents, or operate continuously as background services. For teams and individuals who want full data control, work in air-gapped environments, or simply want to stop watching the meter, local LLM inference via Ollama has become a genuinely viable path.

Why Local Agents Make Sense Now

Three things had to happen before local agent workflows could be practical: the models needed to get good enough, the tooling needed to be simple enough, and the hardware needed to be accessible enough. In 2026, all three conditions are met.

Models: Llama 3.3 70B runs comfortably in 43GB RAM on a Mac with 64GB unified memory or a Linux workstation with a modern GPU. Qwen2.5-Coder 32B handles code generation and analysis tasks that previously required GPT-4. Mistral Small and Phi-4 offer excellent quality in the 14B parameter range, fitting on 8-16GB VRAM. All of these models support function/tool calling — the capability essential for agents.

Tooling: Ollama has matured into a proper local model server with an OpenAI-compatible API, automatic GGUF quantization selection, and straightforward model management. LangChain, LlamaIndex, and the Ollama Python library all speak to it natively.

Hardware: A MacBook Pro M3 Max or M4 Pro handles 70B quantized models at reasonable throughput. RTX 4090 or AMD RX 7900 XTX workstations handle 34B models at genuinely fast inference speeds.

Setting Up Ollama

Installation is a single command on macOS or Linux:

# macOS
brew install ollama

# Linux (one-liner installer)
curl -fsSL https://ollama.com/install.sh | sh

# Start the server (runs on port 11434)
ollama serve

# Pull a capable model for agents
ollama pull llama3.3          # 70B, 43GB, best general capability
ollama pull qwen2.5-coder:32b # 32B, 20GB, excellent for code tasks
ollama pull phi4:14b          # 14B, 9GB, fast and capable for lighter tasks

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1, meaning any library that works with OpenAI’s API works with Ollama by changing two lines.

Building a Simple Agent with LangChain and Ollama

Here is a working agent that can use tools with a locally running model:

from langchain_ollama import ChatOllama
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool
import subprocess
import json

# Initialize the local model
llm = ChatOllama(
    model="llama3.3",
    temperature=0,
    base_url="http://localhost:11434",
)

# Define tools
@tool
def run_shell_command(command: str) -> str:
    """Run a shell command and return its output. Use for file operations and system queries."""
    result = subprocess.run(
        command, shell=True, capture_output=True, text=True, timeout=30
    )
    return result.stdout or result.stderr

@tool
def read_file(path: str) -> str:
    """Read the contents of a file."""
    try:
        with open(path, "r") as f:
            return f.read()
    except Exception as e:
        return f"Error: {e}"

@tool
def write_file(path: str, content: str) -> str:
    """Write content to a file."""
    with open(path, "w") as f:
        f.write(content)
    return f"Written {len(content)} characters to {path}"

tools = [run_shell_command, read_file, write_file]

# Create the agent
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant with access to tools. Use them to complete tasks."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Run the agent
result = executor.invoke({
    "input": "List all Python files in the current directory and tell me how many lines each has"
})
print(result["output"])

Structured Output for Reliable Pipelines

One of the most important capabilities for production agent workflows is structured output — getting the model to return JSON that matches a specific schema. Ollama supports this via the format parameter:

from langchain_ollama import ChatOllama
from pydantic import BaseModel
from typing import List

class CodeReview(BaseModel):
    issues: List[str]
    severity: str  # "low", "medium", "high", "critical"
    suggested_fixes: List[str]
    summary: str

llm = ChatOllama(
    model="qwen2.5-coder:32b",
    format="json",
    temperature=0,
)

structured_llm = llm.with_structured_output(CodeReview)

code_snippet = """
def process_payment(user_input):
    query = f"SELECT * FROM payments WHERE id = {user_input}"
    return db.execute(query)
"""

review = structured_llm.invoke(
    f"Review this code for security issues:\n\n{code_snippet}"
)

print(f"Severity: {review.severity}")
print(f"Issues: {review.issues}")

Multi-Step Research Workflow

Here is a more complete workflow that chains multiple LLM calls to research and summarize a topic — entirely locally:

from langchain_ollama import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOllama(model="llama3.3", temperature=0.3)
parser = StrOutputParser()

# Step 1: Break down the research question
decompose_prompt = ChatPromptTemplate.from_template(
    "Break this research question into 3-5 specific sub-questions: {question}\n\n"
    "Return only the numbered sub-questions, nothing else."
)

# Step 2: Answer each sub-question
answer_prompt = ChatPromptTemplate.from_template(
    "Answer this specific question concisely based on your training knowledge:\n{sub_question}"
)

# Step 3: Synthesize the answers
synthesis_prompt = ChatPromptTemplate.from_template(
    "Given these research findings:\n{findings}\n\n"
    "Write a concise, well-structured summary answering the original question: {question}"
)

# Chain the workflow
decompose_chain = decompose_prompt | llm | parser
answer_chain = answer_prompt | llm | parser
synthesis_chain = synthesis_prompt | llm | parser

def research_workflow(question: str) -> str:
    # Decompose
    sub_questions_text = decompose_chain.invoke({"question": question})
    sub_questions = [q.strip() for q in sub_questions_text.split("\n") if q.strip()]
    
    # Answer each sub-question
    findings = []
    for sq in sub_questions:
        answer = answer_chain.invoke({"sub_question": sq})
        findings.append(f"Q: {sq}\nA: {answer}")
    
    # Synthesize
    return synthesis_chain.invoke({
        "findings": "\n\n".join(findings),
        "question": question,
    })

result = research_workflow("What are the main trade-offs between transformer and state-space models for long-context tasks?")
print(result)

Model Selection Guide for Agent Tasks

Performance Expectations

On an M4 Pro MacBook Pro (48GB):

• phi4:14b: ~35 tokens/sec — fast enough for interactive agent workflows
• llama3.3:70b-q4: ~12 tokens/sec — acceptable for autonomous background tasks
• qwen2.5-coder:32b: ~18 tokens/sec — good for code review workflows

For anything latency-sensitive where users are waiting, the 14B class is practical. For autonomous agents running in the background, 70B quality at 12 tok/sec is entirely usable.

When to Use Local vs Cloud

Use local when: data sensitivity matters, you need offline capability, you run high-volume background processing, costs have become a concern, or you’re iterating quickly and want free experimentation.

Use cloud when: you need maximum capability (frontier models outperform local open-source on complex tasks), you have sporadic usage that doesn’t justify keeping hardware warm, or you need the latest reasoning models immediately.

The most practical approach for many teams is hybrid: local models for development, testing, and high-volume routine tasks; cloud APIs for complex reasoning steps that need the best quality. Ollama’s OpenAI-compatible API makes swapping between them a one-line change.