Skip to main content
This lesson summarizes three practical patterns for adapting a large language model to your data: retrieval-augmented generation (RAG), model fine-tuning, and long-context (passing documents in the prompt). They can solve similar-looking problems but operate very differently. Use this guide to decide which approach matches your operational constraints, cost tolerance, and desired behavior. The key distinction to remember:
  • RAG and long context supply information at runtime — they change what the model sees during inference but do not alter the model’s weights.
  • Fine-tuning updates model weights and therefore changes the model’s behavior persistently.
A presentation slide explaining that RAG and long context provide knowledge at runtime and change what the model can see during inference rather than altering its trained knowledge. It adds that most “it doesn’t know our stuff” issues are retrieval problems, not training problems.

​
High-level tradeoffs (summary)

  • RAG: Retrieve relevant data at query time and include it with the prompt. Model weights remain unchanged. Best when knowledge is large, frequently updated, or must be auditable/cited.
  • Fine-tuning: Train the model on examples to encode recurring formatting, decisions, or tone into weights. Best when you need consistent behavior that does not change often.
  • Long context: Place documents directly into the request context window so the model can use them immediately. Best for one-off documents or small datasets where operational simplicity matters.
Tip: Start by diagnosing the problem (fresh facts, request-specific context, or consistent behavior). Choose the lever that directly addresses that gap instead of defaulting to fine-tuning.
An infographic titled "The Three Approaches" that outlines three methods—RAG, Fine-Tuning, and Long Context—for adapting a model to your data. Each method has a short description and a "Best Fit" box explaining when to use it.

​
Mechanics and best-fit scenarios

  • RAG
    • How it works: Index your sources, run a semantic search at query time, retrieve top chunks, then prepend/append them to the prompt.
    • Best fit: Large knowledge bases, frequently updated documentation, situations requiring citations or access-control checks.
  • Fine-tuning
    • How it works: Continue training the model on curated examples so desired behaviors become encoded in the weights.
    • Best fit: Stable, repeatable behaviors (formatting, classification rules, or niche workflows) that must be consistent across many requests.
  • Long context
    • How it works: Pass the needed documents directly in the prompt (within the model’s token window).
    • Best fit: One-off documents or small, immediate contexts where simplicity and speed outweigh per-request token cost.

​
Cost, maintenance, and failure modes

A dark-themed slide titled "The Tradeoffs at a Glance" showing a comparison table of three approaches — RAG, Fine-Tuning, and Long Context. The table lists differences across rows like what changes, best use cases, freshness, cost shape, and warnings.

​
Avoid the common reflex

Many teams reflexively reach for fine-tuning when the model “doesn’t know our runbooks or docs.” That often misdiagnoses the problem. If the material changes frequently, encoding it into weights forces repeated retraining and high maintenance costs.
Warning: Don’t fine-tune just because the model missed facts. Check whether retrieval or prompt-context fixes the issue first — these are usually cheaper and more maintainable.
Match the technique to the need:
  • Current, frequently changing facts → use RAG.
  • One-off or small documents for a single request → use long context.
  • Stable, repeatable behavior that must be consistent across requests → consider fine-tuning.
A presentation slide titled "The Instinct vs the Right Approach" comparing two strategies: "The Reflex" (fine-tune the model on runbooks/tickets) and "The Right Call" (use RAG to retrieve facts, the context window for one-off docs, and fine-tune only for behavior). The left column lists costs and downsides of fine-tuning, while the right column highlights cheaper, fresher alternatives.

​
Practical decision path (checklist)

  1. Does the required material fit inside the model’s context window and is it needed only for this task?
    • Yes → use long context.
  2. Is the knowledge large in volume or updated frequently?
    • Yes → use RAG so you can retrieve fresh, relevant chunks at query time.
  3. Do you need consistent, repeatable behavior (formatting, classification, tone) that prompting alone cannot achieve?
    • Yes → consider fine-tuning.
  4. Avoid encoding frequently changing facts into model weights; manage them externally so updates are independent.
  5. Combine approaches where useful: for example, fine-tune for stable behavior while using RAG for organizational knowledge and passing request-specific docs via long context.
A dark-themed slide titled "The Decision Path" with a numbered five-step checklist guiding how to choose AI approaches. It recommends using long context, RAG (retrieval), fine-tuning, avoiding baking in changing facts, and combining methods when needed.

​
Quick examples of when to pick each lever

​
Final takeaway

Start with the problem, not the tool. Diagnose whether the gap is:
  • Missing up-to-date facts → RAG
  • Request-specific context → long context
  • Need for consistent behavior → fine-tuning
In production, you’ll often combine levers: fine-tune for predictable behavior, use RAG for organizational knowledge and freshness, and pass request-specific documents through the context window as needed. That mix gives the best balance of freshness, accuracy, and predictable outputs.

Watch Video