RAG vs Fine-Tuning Decision
Overview
RAG and fine-tuning are not interchangeable knobs. RAG injects retrievable facts at inference time; fine-tuning changes model behavior and style in the weights. Pick wrong and you pay in retraining cycles, stale knowledge, or brittle prompts.
When to Use
- Domain knowledge or tone is missing from base-model behavior
- Team is debating "should we embed our docs" vs "should we train on our data"
- Skip when few-shot prompting already clears your eval bar on real inputs
Quick Reference
| Need | Lean toward |
|---|
| Facts change often; citations required | RAG |
| Stable format, tone, or task behavior | Fine-tuning or strong system prompt |
| Both grounded facts and house style | RAG + prompt; add PEFT only if prompt plateaus |
| Tiny static FAQ | RAG or prompt; fine-tune is usually overkill |
| Classifier with stable labels | Fine-tune or structured prompt + eval |
Implementation
- Classify the gap — is it what the model knows or how it responds?
- Check data shape — RAG needs a retrievable corpus; fine-tuning needs representative labeled pairs.
- Check change frequency — monthly policy updates favor RAG; stable style favors fine-tune.
- Prototype the cheaper path first — RAG + eval fixtures before any training job.
- Measure on held-out real queries — not playground vibes.
- Document the decision — so the next engineer does not re-litigate it.
On This Portfolio
The public assistant uses RAG over published writings and research — content changes when I publish in admin. Job-fit classification is a candidate for fine-tuning only if prompt-only approaches plateau; general Q&A stays retrieval-first.
Common Mistakes
- Fine-tuning to memorize facts that will change next quarter
- Using RAG to fix JSON format or routing (that is prompting / schema validation)
- Skipping evals because "the demo looked good"
Related Skills
- Data availability audit
- Prompt engineering vs fine-tuning tradeoff
- RAG pipeline design