How Vector Databases Work: Embeddings, HNSW & Semantic Search

One query. A million stored vectors. Do we compare against every one? That is exact nearest neighbor, and it is correct, and it is linear. At that scale, latency is the product. A point in space is not a search. This lecture is the store those numbers live in, and the hop that finds the nearest neighbor without reading every row.

Here is the picture I want stuck in your head. Meaning becomes a vector. The vector is stored with an id and a chunk of text. A later query becomes a vector too. The system returns the nearest match, not the exact keyword. That retrieve step is what RAG needs before generation. Embeddings built the space. This is how you search it.

The honest algorithm is simple. Embed the query. Cosine with every stored vector. Sort. Take top k. That is exact nearest neighbor, and it is correct. It is also linear in the corpus. A few thousand chunks, fine. A million chunks, you are scanning a million rows on every question. Latency becomes the product. At scale, scanning every vector becomes the latency problem.

People say they have a vector database when they have a numpy array. An array stores vectors. It does not give you filtered search, incremental inserts, or a query path that stays fast as the corpus moves. A production vector store adds queryable vectors, metadata, persistence, updates, and optional ANN indexes. Postgres with pgvector counts. A raw matrix does not, not once the corpus moves.

Each row is three things glued together. The vector, the address in embedding space. An id that points back to the chunk of text. Metadata, the fields you will filter on later, source, date, tenant, permission. Lose the id and you retrieved a point with no document. Lose the metadata and you cannot restrict the search to this customer. The vector is not the knowledge. The chunk is.

The lifecycle is boring on purpose. Ingest a document. Chunk it. Embed each chunk. Insert the row. Build or update the index. Later, embed a query and ask for nearest neighbors. RAG only sees the text that comes back. If insert is stale, query is stale. If the embedding model changes, old vectors are in a different space. You do not mix models in one index and hope.

Distance is a choice. Cosine cares about angle. Inner product cares about angle and magnitude. Euclidean cares about raw distance. If both vectors are L two normalized, cosine similarity and inner product produce the same ranking. If you do not, they do not. Mixing them is a silent bug. The query embedding and the document embeddings must use the same metric the index was built with. Same model, same normalize, same distance.

Approximate nearest neighbor sounds like a downgrade. It is the product. You trade a little recall for a lot of speed. The index is allowed to miss the true nearest if it returns a close one fast. That is the contract. You measure it. Recall at k. If gold is not in the shortlist, no later rerank can save you. A N N is a lossy candidate generator by design.

H N S W is a graph. Hierarchical navigable small world. Layers of neighbors. You enter at a coarse layer, hop toward the query, drop down, hop again. It is a highway system, not a magic box. The graph costs memory. Build is slower. Query is fast. If you heard of H N S W in the embeddings lecture, this is where it lives: inside the database, as the thing you walk instead of scanning the table.

I V F, inverted file, is the other common shape. You cluster the vectors. At query time you pick the nearest clusters, then search only those. It is a first cut. Miss the right cluster and the gold chunk is gone. More clusters probed means more recall and more latency. H N S W and I V F are not competing religions. They are two ways to avoid reading a million rows.

Metadata is not decoration. Tenant id, published date, allowed sources. If you filter after ANN, you might fetch neighbors you then throw away, and recall drops. If you filter before, the graph walk has to respect the predicate, which some indexes do badly. Pre-filter, post-filter, or hybrid. The order is an engineering choice, and it changes who gets retrieved. Permissions belong here, not in the prompt.

Every knob fights. E F search on H N S W explores a larger candidate set. I V F probes search more lists. Higher values find more of the true neighbors, and cost milliseconds. Product people feel latency. Retrieval people feel missed chunks. You do not pick a default from a blog. You measure recall at k on questions you care about, then buy the smallest latency that still hits the gold set.

Semantic search is not a vibe. It is nearest neighbor in embedding space. Refund window and money back policy can sit close even when the tokens differ. That is the win. The failure is the opposite: two chunks that share words and not meaning, or a query that lands near the wrong cluster. The database did what you asked. The embedding space decided what near meant.

Dense retrieval misses exact identifiers. SKU, error code, a proper name the embedder smashed. Lexical search, B M twenty-five, still wins on those. Hybrid means you retrieve from both, then merge. Not because it sounds complete. Because meaning search and token search fail on different questions. A retrieval stack with no lexical path will leak those misses into RAG. Two systems can share that job.

Reranking is a second model on a shortlist. A cross encoder that reads query and chunk together. The reranker runs on a small shortlist, so retrieval must find the right candidates first. The index finds candidates. The reranker only reorders them. It cannot invent a passage the index never returned. If top k missed the gold, rerank shuffles the wrong list. Fix retrieval first. Then rerank.

The policy changed on Tuesday. The vectors still describe Monday. That is not a hallucination. That is stale vector data. You re-chunk, re-embed, upsert. Deletes matter too. A chunk that left the corpus should leave the store. If you only insert, you search a graveyard. RAG will cite a paragraph legal already rewrote. The generator looks confident. The retrieve step is lying.

Query embeddings and document embeddings have to live in one space. Same embedding model version, and a compatible encoding configuration. A newer embedder is a new space. You do not query v two against an index built by v one. The quiet mismatch is a different prompt template for queries versus documents. Some models need that split. Ignore it and near becomes noise.

A good query response is not just text. It is chunk text, id, score, and metadata. Score is a ranking signal, not truth. A high cosine is not a citation. RAG should see the passage and the source. If you drop metadata here, the generator cannot say which file it used. The database did the search. The application still has to keep the receipt.

Put it on the RAG pipeline. Query. Embed. Retrieve. Stuff. Generate. The vector index sits inside retrieval. Not memory. Not the model. Not the whole retrieve step. If retrieve misses, generation still writes. That is why citations are not proof. This lecture is why the miss happens: the wrong metric, the wrong filter, stale rows, or an ANN walk that never reached gold.

You do not have to buy a separate product to have a vector index. P G vector puts ANN next to the rows you already trust. Dedicated stores exist because some teams want that graph outside the database. The engineering choice is the mechanism, not the logo. H N S W, filters, hybrid, upsert. If those words are in the query path, you are in the right neighborhood.

When RAG is wrong, do not start at the prose. Print the retrieved chunks. Is gold in top k. If no, retrieval failed: chunking, the query, embedding mismatch, filters, stale data, or ANN recall. If gold is missing, do not tune the generator first. If yes, stuffing or the generator failed. That split is the whole job. Vector search is measurable. Treat it like a query plan, not like magic meaning.

This is not computer memory, and not a cache you free. A vector database stores embeddings so semantic search can return nearest neighbors.

Embeddings turn meaning into vectors. This lecture stores those vectors and searches them. RAG takes the nearest chunks and writes with evidence in the prompt. Miss the store, and RAG has nothing to fetch. Miss RAG, and the model guesses. The next lesson is that retrieve. Stay in this neighborhood. Do not wander off into tools until the index is honest.

You have the space. You have the index. Next we put retrieved evidence in front of the generator, and we watch every way that still fails. Chunk too big. Chunk too small. Lost in the middle. A miss that still writes. That is RAG. Retrieve, then generate. Citations are not proof. The retrieve step is still lossy. If nearest match is clear, you are ready for retrieve as a pipeline. Embeddings built the coordinates. This lecture stored them. RAG is what happens after the match is injected into the prompt.