When to Fine-Tune vs Prompt
| Scenario | Fine-Tune | Prompt/RAG | Why |
|---|
| Domain-specific terminology | Yes | Maybe | Model doesn't know your jargon |
| Consistent output format | Yes | Partial | Fine-tuning guarantees format better |
| Tone/style matching | Yes | Partial | Can learn from 50+ examples |
| Factual knowledge | No | Yes (RAG) | Fine-tuning doesn't add facts well |
| Simple task, clear instructions | No | Yes | Prompt engineering sufficient |
| Cost at scale | Yes | Depends | Fine-tuned small model < API costs at volume |
Dataset Preparation
| Item | Description |
|---|
Format: JSONL | {messages: [{role:'system',content:...},{role:'user',content:...},{role:'assistant',content:...}]} |
Minimum examples | 50-100 for basic tasks, 500+ for complex style/tone, 1000+ for difficult reasoning |
Quality over quantity | 10 high-quality diverse examples > 100 near-identical ones |
Diversity | Cover edge cases, different phrasings, varied complexity — not just happy path |
Validation split | 90/10 train/val. Use val to detect overfitting early. |
System message | Include in training — model learns when/how to follow system instructions |
LoRA & QLoRA (PEFT)
| Item | Description |
|---|
LoRA | Low-Rank Adaptation — trains small adapter matrices instead of full model. ~1% of parameters. |
QLoRA | Quantized LoRA — 4-bit base model + LoRA adapters. Fine-tune 70B on single GPU. |
Rank (r) | 8-64 typical. Higher = more capacity, more VRAM. Start at 16. |
Alpha | Usually 2× rank. Scales LoRA contribution. Alpha=32 with r=16 is common. |
Target Modules | q_proj, v_proj (attention). Sometimes k_proj, o_proj, gate_proj, up_proj, down_proj. |
Dropout | 0.05-0.1 — prevents overfitting, especially with small datasets. |
Hyperparameters
| Item | Description |
|---|
Learning Rate | 1e-4 to 5e-4 for LoRA. Lower (1e-5) for full fine-tuning. Use cosine scheduler. |
Batch Size | 4-16 per GPU. Effective batch = per-GPU × gradient_accumulation_steps × num_GPUs. |
Epochs | 1-5 typically. More epochs = more overfitting. Monitor validation loss. |
Warmup Ratio | 0.03-0.1 — gradually increases LR at start. Prevents early instability. |
Weight Decay | 0.01-0.1 — L2 regularization. Higher for small datasets. |
Pro Tip: Fine-tune only when prompt engineering + RAG can't solve the problem. Signs you need fine-tuning: consistent format requirements, domain-specific vocabulary, or when prompt tokens cost more than fine-tuning.