Local LLM Setup Cheat Sheet

Running LLMs locally — Ollama, LM Studio, llama.cpp, quantization levels explained (Q4_K_M, Q5_K_M, etc.

Last Updated: May 1, 2025

Quantization Levels (GGUF)

Quant	Bits	Size (7B)	Quality	Use Case
Q2_K	2-bit	~3GB	~80%	Smallest possible, speed over quality
Q3_K_M	3-bit	~4GB	~87%	Very tight memory, acceptable quality
Q4_K_M	4-bit	~5GB	~95%	Sweet spot — best quality/size balance
Q5_K_M	5-bit	~6GB	~97%	Slightly better, needs more RAM
Q6_K	6-bit	~7GB	~98%	Near-perfect on capable hardware
Q8_0	8-bit	~8GB	~99%	Maximum quality, no noticeable loss
F16	16-bit	~15GB	100%	Full precision, development/evaluation

Ollama Quick Start

ollama pull llama3.1:8b

Download and cache a model

ollama run llama3.1:8b

Interactive chat session

ollama list

Show downloaded models with sizes

ollama rm llama3.1:8b

Remove model to free space

ollama serve

Start the API server (usually auto-started)

POST http://localhost:11434/api/generate

REST API — drop-in OpenAI-compatible via litellm

Modelfile: FROM llama3.1:8b
SYSTEM You are...
PARAMETER temperature 0.7

Custom model with system prompt + parameters

LM Studio

Item	Description
`GUI for llama.cpp`	Browse Hugging Face → download GGUF → chat. No command line needed.
`Local Server`	One-click local API server — OpenAI-compatible endpoint at localhost:1234
`GPU Offloading`	Drag GPU layers slider — How many layers to run on GPU vs CPU. More = faster, needs VRAM.
`Prompt Templates`	Each model has a specific template (ChatML, Llama, Mistral) — LM Studio auto-detects.
`Presets`	Save configs: creative (temp 0.8), precise (temp 0.1), balanced (temp 0.7)

Hardware Guide

Item	Description
`7B models (Llama 3.1)`	8GB RAM minimum, 4GB VRAM for GPU acceleration. Runs on M1 Mac, any modern laptop.
`13B models`	16GB RAM, 8GB VRAM recommended. Sweet spot for quality + accessibility.
`34B models`	32GB RAM, 16GB+ VRAM. Noticeably smarter, needs beefy hardware.
`70B models`	64GB+ RAM or dual GPU. Run with Q4_K_M on 48GB Mac/GPU. Cloud only for most users.
`Apple Silicon`	Unified memory = RAM = VRAM. M1 Max 64GB runs 70B Q4. M3 Pro 36GB runs 34B Q5.

Pro Tip: Q4_K_M is the sweet spot for most users — 4-bit quantization with medium quality. It runs 7B models in <8GB RAM, 13B in <10GB, and preserves ~95% of full-precision quality.