TL;DR:

  • Prompt engineering is the right starting point: it’s fast to iterate, requires no training data, and works well for most use cases
  • Fine-tuning pays off when you have high call volumes, consistent output format requirements, or need to compress long system prompts into the model’s weights
  • The most effective production agents often use both: a fine-tuned base model with runtime prompting for task-specific context

Every team building AI agents eventually hits the same question: should we invest in fine-tuning, or can prompt engineering get us to where we need to be? The honest answer is that it depends on your volume, your output requirements, and how much friction you can tolerate in the update cycle.

Here’s a practical framework for making that call.

What prompt engineering solves (and what it doesn’t)

Prompt engineering — crafting system prompts, few-shot examples, and instruction structures — is where every agent starts. The feedback loop is tight: change the prompt, test the agent, iterate. No training runs, no data pipelines, no waiting.

For most use cases, particularly in the early stages, this is exactly what you want. You’re still learning what your agent needs to do. Your requirements are changing. The cost of a bad fine-tuning run is much higher than the cost of a bad prompt — you lose time and compute budget, and you have to start over.

Prompt engineering has real limits, though. The most common ones:

Consistency degrades with prompt complexity. When you’re relying on a 2,000-token system prompt to instil domain-specific knowledge, output format requirements, and behavioral constraints, you get inconsistency. The model follows some of it most of the time. You end up with extensive output validation to catch the cases where it doesn’t.

Latency increases with prompt length. Every token in your system prompt is a token that has to be processed on every call. If your agent makes thousands of calls per day, a bloated system prompt adds up in both latency and cost.

Context window pressure. If your system prompt is eating 30% of your context window, that’s 30% less space for tool outputs, conversation history, and the actual task. For agents that need long context, this matters.

What fine-tuning actually does

Fine-tuning adjusts the model’s weights to embed knowledge, behaviors, or output patterns that would otherwise require extensive prompting. You’re not teaching the model new capabilities — you’re making certain behaviors the model’s default, so you don’t need to elicit them with every call.

The practical effects:

Compressed system prompts. Domain knowledge, output formatting rules, and behavioral constraints that would take 1,500 tokens to describe in a prompt can become implicit after fine-tuning. Your system prompt gets shorter, your context window stays cleaner.

Improved consistency. Fine-tuned models follow format requirements more reliably than prompted ones, particularly for structured output tasks. If your agent needs to emit specific JSON schemas, call tools in particular patterns, or maintain a consistent voice across thousands of outputs, fine-tuning reduces the variance.

Better latency and cost at scale. Shorter prompts mean faster processing and lower token costs. For a high-volume production agent running tens of thousands of calls per day, this adds up to real savings.

The decision framework

Start with prompting. Always. Fine-tuning requires training data, which means you need labeled examples of good agent behavior. The best source of training data is a working prompting-based system. Run your agent with prompt engineering first, collect outputs, curate the good ones, and use those as fine-tuning examples.

Fine-tune when you hit one of these conditions:

High volume with consistent tasks. If your agent is doing the same type of task thousands of times a day and you’ve locked down the requirements, fine-tuning’s consistency and cost benefits are worth pursuing.

Output format is critical and frequently breaks. If your agent needs to emit structured output and validation failures are a recurring problem, fine-tuning for format compliance is often faster than building increasingly elaborate output parsers.

System prompt is longer than 1,000 tokens of stable content. If you have substantial content that never changes call-to-call — domain knowledge, strict formatting rules, persona — fine-tuning can absorb it and free up context space.

Latency matters. If your agent is user-facing and prompt length is contributing meaningfully to response latency, fine-tuning to compress the prompt is a direct lever on perceived performance.

Stick with prompting when:

Requirements are still changing. Every requirements change in a fine-tuned system means a new training run. If you’re iterating quickly, prompt changes are orders of magnitude faster.

You don’t have enough examples. Fine-tuning with low-quality or insufficient data can make things worse. You generally need hundreds to thousands of high-quality examples for reliable improvement.

The task is genuinely novel. Fine-tuning can’t teach capabilities the base model doesn’t have — it adjusts weights toward behaviors the model can already exhibit. For genuinely new capability requirements, you need a different base model, not fine-tuning of the current one.

Using both together

The most effective production agent setups often combine fine-tuning and runtime prompting. The fine-tuned model handles stable knowledge, format requirements, and behavioral defaults. Runtime prompts inject task-specific context, user-specific information, and dynamic instructions that change call-to-call.

This architecture keeps your fine-tuned model stable (you don’t retrain every time a task changes) while preserving the flexibility to adjust agent behavior through prompting when needed.

A common pattern: fine-tune for output format compliance and domain vocabulary, then use runtime prompts for task description, tool selection context, and user-specific constraints. The fine-tuned model handles the “how to behave” layer; the prompt handles the “what to do right now” layer.

The cost reality

Fine-tuning on major providers (OpenAI, Anthropic, Google) has become significantly cheaper than it was two years ago, but it still requires:

  • Data curation time (often the largest cost — human review of training examples)
  • Training compute
  • Evaluation runs to verify the fine-tune improved what you intended and didn’t regress on other tasks
  • A testing process before promoting to production

Budget several weeks for a proper fine-tuning cycle, including data preparation. If your timeline is shorter than that, prompt engineering is almost always the right call.

The question isn’t which approach is better — it’s which approach is right for your current stage, volume, and requirements. Most agents should start with prompting and consider fine-tuning once they’re stable enough that the cost of a training cycle is lower than the ongoing cost of prompt-induced inconsistency.