Semantic Search Cheat Sheet

Semantic search implementation — embedding search, hybrid search combining BM25 with vectors, re-ranking with Cohere and cross-encoders, and production search opt.

Last Updated: May 1, 2025

Search Approaches

Approach	How It Works	Strengths	Weaknesses
BM25 (Lexical)	TF-IDF variant — word frequency + document length	Exact matches, code, IDs	Misses synonyms, paraphrases
Dense Retrieval	Embed query + docs → cosine similarity	Semantic understanding	Struggles with rare terms
Hybrid Search	BM25 score + vector similarity combined	Best of both worlds	Higher computational cost
Cross-Encoder Re-rank	Transformer scores query-doc pair jointly	Highest accuracy	Too slow for full corpus
ColBERT	Late interaction — token-level embeddings	Balance of speed + accuracy	More storage than bi-encoders
SPLADE	Learned sparse + dense representations	Strong on rare terms	Model size, training complexity

Embedding Search Pipeline

text-embedding-3-small

OpenAI embedding model: 1536 dims, $0.02/1M tokens — best cost/performance balance

text-embedding-3-large

OpenAI: 3072 dims, $0.13/1M tokens — higher accuracy for complex queries

sentence-transformers/all-MiniLM-L6-v2

Open-source: 384 dims, fast, runs locally — great for prototyping

intfloat/e5-large-v2

Open-source: 1024 dims, MTEB leaderboard top performer — prefix queries with 'query: '

BAAI/bge-large-en-v1.5

Open-source: 1024 dims, optimized for retrieval — also needs 'query: ' prefix

cosine_similarity(query_vec, doc_vecs)

Compute similarity scores — normalize embeddings first for cosine

pgvector / Qdrant / Weaviate

Vector databases: store embeddings + metadata, run ANN search at scale

chunk_size=512, overlap=50

Typical chunking parameters: 512 tokens with 50-token overlap for context continuity

Implementing Hybrid Search

Item	Description
`BM25 Component`	Use Elasticsearch's built-in BM25 or rank_bm25 Python library for lexical scoring
`Score Fusion`	Combine BM25 and vector scores: weighted_sum = α * bm25_score + (1-α) * vector_score
`Reciprocal Rank Fusion (RRF)`	Rank-based fusion: score = Σ 1/(k + rank_i) — no normalization needed, robust to score ranges
`Fusion Weight Tuning`	α=0.3 favors semantic; α=0.7 favors keyword. Tune on your data: start at 0.5
`Query Classification`	Route queries: short/ambiguous → semantic; precise/technical → lexical; mixed → hybrid
`Metadata Filtering`	Pre-filter by tags, date, author before scoring — reduces candidate set size
`Elasticsearch KNN`	ES 8.x+ supports dense_vector fields with approximate KNN alongside BM25 queries
`OpenSearch Neural Search`	Built-in hybrid search: combine neural query with keyword query in a single request

Re-Ranking for Accuracy

cohere.rerank(query, documents, top_n=5)

Cohere Re-rank API: re-scores top-K documents for accuracy — $0.001/search

cross-encoder/ms-marco-MiniLM-L-6-v2

Open-source cross-encoder: score query-doc pairs — ~10ms per pair on GPU

Two-Stage Pipeline

Stage 1: bi-encoder retrieves top-100 candidates (fast). Stage 2: cross-encoder re-ranks to top-10 (accurate)

Retrieval Depth

Retrieve more candidates than you display: 100→50→10 pipeline gives re-ranker room to work

Multi-Lingual Re-rank

Cohere supports 100+ languages; BGE-reranker-v2-m3 handles multilingual cross-lingual

Re-rank Fusion

Combine multiple re-ranker scores: average cross-encoder + Cohere re-rank scores

Latency Budget

Two-stage: 50ms retrieval + 100ms re-ranking = ~150ms total. Acceptable for most search UIs

Caching

Cache embeddings + re-rank scores for frequent queries — major latency reduction

Pro Tip: Hybrid search (BM25 + embeddings) consistently outperforms either method alone in production. BM25 catches exact matches; embeddings catch semantic meaning. Combine with a re-ranker for best results.