Semantic Search: How Search Stops Matching Letters and Starts Matching Meaning
Keyword search matches letters. Semantic search matches meaning — by turning text into vectors and ranking by geometry. Here’s how embeddings, cosine similarity, HNSW indexes, and reranking fit together, with the interview one-liners.
Explain it like I’m five
Imagine you run a library, and you have two librarians. The first librarian only understands letters. You ask for “a bedtime story about dragons” — she marches to the D shelf and hunts for titles with D-R-A-G-O-N spelled out. If the perfect book is called The Brave Little Lizard Who Breathed Fire, she walks right past it. The letters don’t match, so to her, it isn’t a match.
The second librarian understands what you mean. You describe the story — a kind, scaly creature who breathes fire and guards a cave — and she brings you the lizard book, because a fire-breathing lizard is what a dragon is when you’re five. She didn’t match letters. She matched meaning.
The first librarian is keyword search. The second is semantic search. The trick is a simple one: a model reads every book — and your question — and turns each into a list of numbers. Texts that mean similar things get similar lists of numbers. So instead of comparing letters, the computer compares the numbers, and the numbers understand. This guide is the grown-up version: how text becomes numbers, how “similar numbers” is actually computed, and how you search a million books without comparing all of them.
Intuition: matching letters vs matching meaning
Keyword search is the first librarian with better math. Classical ranking — TF-IDF, and its sharper successor BM25 — scores documents by the query’s literal terms: how often the words appear, how rare they are across the corpus, how long the document is. It’s fast, interpretable, and brutally effective when the words line up. It is also completely blind to meaning.
That blindness has a name in interviews: the car/automobile problem. The document says “automobile,” your query says “car,” and the letters don’t match — zero hits, even though it’s exactly what you wanted. Same story with “affordable laptops” vs “budget notebooks,” “heart attack” vs “myocardial infarction,” or a support ticket saying “my screen is frozen” when the help article says “display unresponsive.” People paraphrase constantly; letters don’t.
Semantic search hires the second librarian. Both the documents and your query are converted into embeddings — fixed-length vectors of numbers, typically a few hundred to a couple thousand dimensions — and results are ranked by how close the vectors are, not by which letters they share. “Car” and “automobile” land near each other in this space, so the query finds the document the letters would have missed. The one-line version for the interview: keyword search answers “which documents contain these terms”; semantic search answers “which documents mean something close to this.” Neither replaces the other — production systems usually run both — but you have to know which problem each one solves.
How it works: meaning as geometry
Embeddings turn text into points in space. An embedding model — usually a transformer trained for the job — maps any piece of text to a vector, say 768 numbers, where similar meanings end up pointing in similar directions. “Meaning” becomes geometry: distance and angle now stand in for relatedness. Two sentences that say the same thing in different words sit next to each other; unrelated sentences sit far apart.
The lineage is worth knowing. The idea that meaning could live in a vector space goes back to word2vec (Mikolov et al., 2013), which showed individual words could be embedded so that relationships became arithmetic on vectors. Sentence-BERT (Reimers & Gurevych, 2019) lifted the trick from words to whole sentences, producing embeddings you could actually compare. DPR — Dense Passage Retrieval (Karpukhin et al., 2020) then trained a pair of encoders specifically for search: one for the query, one for the documents, with a contrastive objective that pulls matching pairs together and pushes the rest apart. Modern embedding models are the descendants of this line: trained so that semantically related text is geometrically near.
Cosine similarity, by hand
“Closeness” almost always means cosine similarity: the cosine of the angle between two vectors. It runs from 1 (same direction) through 0 (perpendicular — unrelated, for the non-negative embeddings most models emit) to −1 (opposite). Two tiny vectors make it concrete:
a = [0.9, 0.1, 0.2] (“car”) and b = [0.8, 0.2, 0.1] (“automobile”): dot product 0.72 + 0.02 + 0.02 = 0.76; lengths |a| ≈ 0.927, |b| ≈ 0.831; cosine = 0.76 / (0.927 × 0.831) ≈ 0.99 — nearly identical direction. Now c = [0.1, 0.9, 0.0] (“pancake”): cosine with a ≈ 0.21 — nearly unrelated. That’s the whole ranking signal: embed the query, score it against every document vector, return the top-k angles.
The retrieval pipeline
Embed the corpus once, the query at runtime. Every document is embedded offline and its vector stored in an index. When a query arrives, it’s embedded by the same model, scored against the stored vectors by cosine similarity (or inner product — identical when vectors are normalized), and the top-k are returned. Two things to tattoo on your notes: the query and the documents must go through the same model, or their vectors live in different spaces and the geometry is meaningless; and “top-k” is the entire output contract — semantic search is a ranking problem, not a filtering one.
Where semantic search actually shows up
Brute force dies at scale — this is where ANN comes in. Scoring one query against a million 768-dimensional vectors costs roughly a million dot products: about 768 million multiply-adds per query. Optimized code does that in a few hundred milliseconds on a CPU core — fine for a demo, hopeless at real query rates. Approximate nearest neighbor (ANN) search trades a little exactness for a lot of speed: it returns almost the top-k, but in milliseconds. The dominant algorithm is HNSW — Hierarchical Navigable Small World (Malkov & Yashunin, 2018). Picture the vectors organized as a graph in layers: the bottom layer links every point to its near neighbors, and each layer above is sparser, with long-range “highway” links across the space. A query starts at the top and greedily hops toward the closest neighbor, descending layer by layer — coarse jumps first, fine steps last. A handful of hops, roughly logarithmic in the collection size, lands near the true nearest neighbors with recall typically in the high nineties. Two knobs matter: efConstruction (how carefully the graph is built — quality at index time) and efSearch (how many candidates are considered per query — the recall-vs-latency dial at serve time).
Vector databases: the index with an address. A raw ANN index is a library; a vector database wraps it in storage, metadata, and filters. FAISS (Meta AI, 2017) is the workhorse library — fast, free, battle-tested, including a solid HNSW implementation — but you own the ops: sharding, backups, serving. The database layer adds payloads and predicates: store the vector plus metadata (author, date, price, category) and ask for “top-k vectors where category = shoes.” Your options: Pinecone (fully managed — no infra, pay per usage), Weaviate, Milvus, and Qdrant (open-source databases you can self-host or run as managed cloud). Pick FAISS for research, prototypes, and maximum control; pick a database or managed service when you need filtered search, replication, and someone else’s pager going off at 3am.
RAG retrieval. The canonical use case: chunk your documents, embed the chunks, and at query time retrieve the top-k chunks to stuff into the LLM’s context. The retriever is the whole quality story of a RAG system — a great generator with a bad retriever just hallucinates confidently.
Product search. “Comfy red shoes for standing all day” has almost no words in common with “ergonomic crimson sneakers, all-day cushioning” — letters fail, meaning wins. E-commerce search is where semantic retrieval earns its keep, usually hybridized with keyword matching for SKUs and brand names.
Customer-support search. Tickets arrive paraphrased a thousand ways (“screen frozen,” “display unresponsive,” “laptop won’t wake”) and should all resolve to the same help article. Semantic search collapses the paraphrase space into one answer.
Code search. “Function that retries with exponential backoff” → the actual retry_with_backoff() implementation, even if its docstring says “re-executes flaky calls with growing delays.” Natural language in, code out — trained on code-text pairs, it bridges the vocabulary gap between humans and codebases.
Recommendations. Embed users and items (or just items) and “similar to this” becomes a nearest-neighbor query: watched this movie → vectors of nearby movies. Same geometry, different catalog.
The retrieval math, by hand
One million documents, 768-dimensional embeddings, 4 bytes per float. That’s the canonical setup — here’s what it costs, and where the standard pipeline pieces earn their place.
Embed, index, and search in Python
The full loop in a few dozen lines: embed documents with a sentence-transformer, build an HNSW index with FAISS, and run a top-k query. (Illustrative — install sentence-transformers and faiss-cpu to run it.)
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np
# 1) Embed the corpus once. 768-dim vectors, L2-normalized so that
# inner product == cosine similarity.
model = SentenceTransformer("all-mpnet-base-v2")
docs = [
"Cozy family-run pizzeria with a wood-fired oven, open late.",
"Used car dealership with weekend servicing and financing.",
"Budget laptop deals: refurbished notebooks under $400.",
]
vectors = model.encode(docs, normalize_embeddings=True).astype("float32")
# 2) Build the HNSW index: 32 links per node (M), tuned build quality.
index = faiss.IndexHNSWFlat(768, 32)
index.hnsw.efConstruction = 200
index.hnsw.efSearch = 64 # recall/latency knob at query time
index.add(vectors)
# 3) Embed the query with the SAME model, take the top-k angles.
query = "affordable italian dinner nearby"
q = model.encode([query], normalize_embeddings=True).astype("float32")
scores, ids = index.search(q, k=2)
for i, s in zip(ids[0], scores[0]):
print(f"{s:.3f} {docs[i]}")
# 0.812 Cozy family-run pizzeria with a wood-fired oven, open late.
# 0.204 Budget laptop deals: refurbished notebooks under $400.
# "pizzeria" won on meaning alone: the query never said the word.
Six questions that test the real understanding
What to say (≈60 sec): “Keyword search — TF-IDF, BM25 — ranks documents by the query’s literal terms: term frequency, rarity, document length. It’s fast and interpretable but blind to meaning — the car/automobile problem: the document says ‘automobile,’ the query says ‘car,’ zero hits. Semantic search embeds documents and the query into vectors and ranks by cosine similarity, so paraphrases and synonyms match. Keyword search answers ‘which documents contain these terms’; semantic search answers ‘which documents mean something close to this.’ In production you run both — hybrid — because keyword search still wins on exact tokens like SKUs and error codes.”
What to say: “An embedding model maps text to a fixed-size vector where similar meanings point in similar directions — meaning becomes geometry, and cosine similarity measures the angle. The lineage: word2vec in 2013 showed words could live in a vector space with analogies as vector arithmetic; Sentence-BERT in 2019 lifted it to whole sentences; DPR in 2020 trained query/document encoders with a contrastive objective — pull matching pairs together, push the rest apart — specifically for retrieval. So ‘capture’ is literal: training optimizes the geometry until related text is near and unrelated text is far.”
Likely follow-up: “Why must the query and documents use the same model?” — because each model defines its own vector space. Embed the query with model A and the docs with model B and the angles are meaningless; the geometry only exists within one model’s space.
What to say: “Brute force is a million 768-dim dot products per query — about 768 million multiply-adds, a few hundred milliseconds on a CPU core. At any real QPS that’s a GPU fleet doing arithmetic. HNSW builds a layered navigable graph: the bottom layer links every vector to its near neighbors, upper layers add sparse long-range highways. A query starts at the top and greedily hops toward the closest neighbor, descending layer by layer — roughly logarithmic hops, a few thousand dot products, single-digit milliseconds, recall in the high nineties. efSearch is the dial: more candidates per query buys recall with latency.”
What to say (≈90 sec): “Four stages. Embed: chunk documents into ~512-token pieces with overlap — the chunk is the unit of retrieval — and embed offline with one model, L2-normalized. Index: build an HNSW index; a million 768-dim float vectors is ~3 GB plus graph overhead, so it fits in RAM on one machine. Retrieve: embed the query with the same model, ANN top-50 in milliseconds, with metadata filters applied for things like category or date. Rerank: run a cross-encoder over the top-50 and keep the top-5, and fuse a BM25 pass with reciprocal rank fusion so exact tokens — SKUs, error codes, names — don’t get missed. Version the embeddings: swapping the model means re-embedding everything, since vectors from two models aren’t comparable.”
What to say: “FAISS is a library, not a database — Meta’s, fast, free, great HNSW, but you own sharding, backups, and serving. It’s the right pick for research, prototypes, and when you want total control. Weaviate, Milvus, and Qdrant are open-source vector databases: vectors plus metadata payloads, filtered top-k, replication — self-host or use their managed cloud. Pinecone is fully managed: no infra at all, pay per usage. So the decision is really about ops: library when you run the machine, database when you need filtered search with durability, managed when you’d rather pay than page.”
What to say: “Four classics. Embedding swaps: change the model and every stored vector is garbage — re-embed the whole corpus, and version your embeddings so old and new never mix. Chunking mistakes: whole-document vectors average to mush, sentence fragments can’t answer alone; ~512 tokens with overlap is the starting point, then measure. Stale indexes: documents change but vectors don’t — you need an update/delete path, not just a build script. Exact-match queries: ‘HTTP 429’ or a part number has no meaningful paraphrase, so pure semantic search underperforms BM25 there — which is why the hybrid pass isn’t optional, it’s the design.”
Key takeaways
- Keyword search matches letters; semantic search matches meaning. Embeddings turn text into vectors where meaning becomes geometry — similar meanings point in similar directions.
- Cosine similarity is the ranking signal: the angle between vectors, 1 for identical direction, 0 for unrelated. The pipeline is fixed — embed the corpus once, embed the query with the same model, rank, return top-k.
- Brute force dies at scale (768M multiply-adds per query at 1M docs × 768 dims). HNSW’s layered graph gets roughly log-time lookups in milliseconds with recall in the high nineties; efSearch trades recall for latency.
- Production shape: chunk into ~512-token pieces with overlap, hybridize BM25 with vectors via reciprocal rank fusion, and rerank the top-50 down to the top-5 with a cross-encoder.
- FAISS is the library you run yourself; Weaviate, Milvus, and Qdrant are vector databases you can self-host; Pinecone is fully managed. All of them store vectors plus metadata and serve filtered top-k.
- The interview arc: embed → index (HNSW) → retrieve top-k → rerank. And the trap to name: swapping the embedding model invalidates the entire index — version your embeddings and re-embed everything.
Sources & further reading
Every claim in this guide traces to one of these — the wording is ours, the ideas are credited.
- Mikolov et al., “Efficient Estimation of Word Representations in Vector Space” (word2vec), 2013 — words as vectors; relationships as vector arithmetic
- Reimers & Gurevych, “Sentence-BERT,” 2019 — sentence-level embeddings you can compare with cosine similarity
- Karpukhin et al., “Dense Passage Retrieval for Open-Domain QA” (DPR), 2020 — bi-encoder trained with a contrastive objective for retrieval
- Malkov & Yashunin, “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” (HNSW), 2018 — the layered-graph ANN algorithm
- Johnson, Douze & Jégou, “Billion-scale similarity search with GPUs” (FAISS), Meta AI, 2017 — the vector-search library
- ANN-Benchmarks (GitHub) — empirical recall/latency comparisons of ANN algorithms including HNSW