LLM Deployment Cheat Sheet

Production LLM deployment — vLLM, TGI, Ray Serve, quantization (AWQ, GPTQ, GGUF), latency optimization, scaling strategies, and serving infrastructure for large l.

Last Updated: May 1, 2025

Serving Frameworks

Framework	Key Feature	Best For	Throughput
vLLM	PagedAttention — near-zero memory waste	High-throughput APIs, OpenAI-compatible	Best for concurrent users
Text Generation Inference (TGI)	Hugging Face's solution, watermarks, grammar	HuggingFace ecosystem users	Good, with optimizations
Ray Serve	Distributed serving + model composition	Multi-model pipelines, complex routing	Scales horizontally
TensorRT-LLM	NVIDIA-optimized, best GPU utilization	NVIDIA GPU-only, max performance	Highest single-node perf
Ollama	Simple local deployment, GGUF models	Development, local testing	Low (single user)
Llama.cpp	CPU inference, GGUF quantization	Edge devices, CPU-only servers	Low, but runs anywhere
SGLang	Structured generation + radix attention	Structured outputs, JSON modes	Excellent for structured
BentoML	General model serving framework	Teams serving multiple model types	Good, with batching

vLLM Quick Start

pip install vllm

Install vLLM — requires CUDA-compatible GPU

vllm serve meta-llama/Llama-2-7b-chat-hf

Serve a model with default settings — auto-detects best config

vllm serve model --tensor-parallel-size 2

Split model across 2 GPUs for large models (70B+)

vllm serve model --max-model-len 8192

Set max context length — longer = more GPU memory

vllm serve model --gpu-memory-utilization 0.90

Use 90% of GPU memory for KV cache — higher = more concurrent requests

vllm serve model --max-num-seqs 256

Max concurrent sequences — balance throughput vs latency

vllm serve model --quantization awq

Load AWQ quantized model — ~4x less memory, near-identical quality

OpenAI-compatible API on port 8000

vLLM exposes /v1/chat/completions — drop-in replacement for OpenAI SDK

Quantization Guide

Method	Precision	Memory vs FP16	Quality Impact	Load Time
FP16 (baseline)	16-bit	100% (~14GB for 7B)	Reference	Fast
INT8	8-bit	~50%	<0.5% degradation	Fast
GPTQ	4-bit	~25%	1-2% degradation	Slow (calibration needed)
AWQ	4-bit	~25%	<1% degradation	Fast (auto-calibrated)
GGUF (Q4_K_M)	4-bit	~25%	1-3% degradation	Fast (pre-quantized)
GGUF (Q5_K_M)	5-bit	~31%	<1% degradation	Fast (pre-quantized)
GGUF (Q8_0)	8-bit	~50%	Negligible	Fast (pre-quantized)
NF4 (QLoRA)	4-bit + double quant	~20%	2-4% degradation	Fast (bnb config)

Latency Optimization

Item	Description
`Continuous Batching`	Don't wait for full batch — process each token as it's generated. vLLM and TGI do this automatically
`KV Cache Management`	PagedAttention (vLLM) ensures efficient GPU memory use — no fragmentation, higher concurrency
`Speculative Decoding`	Use small draft model to predict tokens → large model verifies in parallel — 2-3x speedup
`Prompt KV Cache Reuse`	Cache system prompt KV states — only compute user prompt tokens. vLLM's automatic prefix caching
`Quantization`	4-bit models run on cheaper GPUs (T4 instead of A100) or fit more concurrent users on same hardware
`Flash Attention 2`	Memory-efficient attention — 2-4x faster than standard attention. Supported by vLLM, TGI, and most frameworks
`Streaming`	Return tokens as they're generated (TTFT optimization) — users see first token in 100-500ms vs 5-30s
`Request Prioritization`	Assign priority to requests — interactive users get low latency, batch jobs get throughput
`GPU Selection`	A100 (best), H100 (2x A100), A10G (budget), L4 (inference-optimized budget), L40S (mid-range sweet spot)
`Load Balancing`	Round-robin across multiple model replicas with session affinity for multi-turn conversations

Pro Tip: Quantization is the single biggest lever for production LLM deployment. Going from FP16 to 4-bit (AWQ/GPTQ) typically gives 3-4x throughput improvement with <1% accuracy loss for most models.