What Is a Vector Database? How AI Search and RAG Actually Store Data

A vector database stores data as high-dimensional numerical vectors (called embeddings) and is built to answer a fundamentally different question than a normal database: not "which rows exactly match this value" but "which rows are most similar in meaning to this one." That single capability — similarity search over meaning rather than exact-match search over values — is what makes modern semantic search, recommendation systems, image similarity search, and retrieval-augmented generation (RAG) pipelines for LLMs actually work, and it's specifically why a regular relational or document database, no matter how well indexed, can't do the same job without help.

What "vector" actually means here

An embedding is a list of numbers — typically anywhere from a few hundred to a few thousand dimensions — produced by a machine learning model that has learned to represent the meaning of a piece of content (a sentence, an image, an audio clip, a product) as a point in that high-dimensional space. The key property that makes embeddings useful is that similar content ends up as nearby points: the embeddings for "How do I reset my password" and "I forgot my login credentials" land close together in that space, even though the two sentences share almost no words in common, because the embedding model captured that they mean roughly the same thing. A vector database's entire job is storing millions or billions of these vectors and answering "which stored vectors are closest to this new one" quickly enough to be usable in a live application.

Why a regular database doesn't do this job well

A traditional database is built around exact or range-based matching — an index on a column can tell you every row where a value equals X, or falls between X and Y, extremely quickly. It has no native concept of "closeness in meaning" between two pieces of text or two images; a keyword search for "laptop bag" simply won't return a product listed as "notebook sleeve," even though a human would recognise them as near-synonyms, because there's no shared keyword to match on. Some relational databases can be extended to store and search vectors (more on pgvector below), but the underlying computation — finding the nearest neighbours to a query vector among millions of stored ones — is different enough from normal indexed lookups that it benefits from purpose-built data structures and, at real scale, purpose-built systems.

How similarity search actually works

The core operation is nearest-neighbour search: given a query vector, find the stored vectors closest to it by some distance measure, most commonly cosine similarity (how aligned two vectors' directions are, ignoring magnitude) or Euclidean distance (straight-line distance between the two points). Computing this exactly against every stored vector — a full linear scan — is accurate but becomes too slow once you have millions of vectors and need sub-second responses. Vector databases instead use approximate nearest neighbour (ANN) algorithms, which trade a small amount of accuracy for a large speed gain by organising vectors into structures that let a search skip over most of the dataset entirely.

The indexing structures behind ANN search

• HNSW (Hierarchical Navigable Small World) — builds a multi-layered graph connecting similar vectors, and search "walks" the graph from a coarse layer down to a fine one, honing in on the nearest neighbours without comparing against every stored vector. Widely used because it offers a strong balance of speed, accuracy and reasonable memory use.
• IVF (Inverted File Index) — clusters vectors into groups (via something like k-means) at index time, and a search first identifies the most relevant cluster(s) and only compares against vectors within them, rather than the whole dataset.
• Product Quantization (PQ) — compresses each vector into a much smaller approximate representation to reduce memory footprint, often combined with IVF or HNSW when the dataset is too large to keep every full-precision vector in memory.

Where vector databases fit inside a RAG pipeline

Retrieval-augmented generation (see our RAG explainer for how the overall pattern works) uses a vector database as its retrieval step: documents are split into chunks, each chunk is converted into an embedding and stored, and at query time the user's question is itself embedded and used to search the vector database for the most semantically relevant chunks. Those retrieved chunks are then passed to the LLM as context alongside the original question, so the model answers using the actual retrieved content rather than only what it happened to learn during training. The vector database is specifically the component responsible for finding "which parts of our knowledge base are actually relevant to this question" — the LLM itself has no efficient way to search millions of documents on its own.

Popular vector databases compared

• Pinecone — fully managed, cloud-only, no self-hosting option; popular for teams that want to avoid operating infrastructure and are comfortable with a proprietary managed service.
• Weaviate — open source, available both self-hosted and as a managed cloud offering; supports hybrid search (combining vector similarity with traditional keyword search) out of the box.
• Milvus — open source, built specifically for large-scale vector search with a strong focus on horizontal scalability; commonly self-hosted for large datasets.
• Qdrant — open source, written in Rust, offers both self-hosted and managed options, with a particular focus on rich metadata filtering alongside vector search.
• Chroma — open source and lightweight, popular for prototyping and smaller applications, often embedded directly in a Python application rather than run as a separate service.
• pgvector — an extension for PostgreSQL that adds vector storage and similarity search directly inside a regular Postgres database, rather than requiring a separate system (see below).

pgvector — using Postgres as a vector database

pgvector is a PostgreSQL extension that adds a native vector column type and similarity-search operators directly into Postgres, meaning a team that already runs Postgres for their normal application data can store and search embeddings in the same database rather than standing up and operating an entirely separate vector-specific system. It supports both exact nearest-neighbour search and, more recently, ANN indexing (via HNSW and IVFFlat index types) for larger datasets. The practical appeal is real: one less system to run, one less data-consistency problem between "the source of truth" and "the search index," and the ability to combine a vector similarity search with a normal SQL WHERE clause in a single query. For genuinely massive scale or the most demanding latency requirements, a purpose-built vector database can still out-perform pgvector — but for the majority of applications, especially ones already on Postgres, pgvector is frequently the simplest correct choice rather than a compromise.

Metadata filtering — vector search is rarely used alone

In real applications, a pure similarity search is almost never the whole query — you typically also want to filter by structured metadata alongside it: "find the most similar product to this one, but only in stock and under ₹2,000," or "find the most relevant document chunks, but only from documents this user has access to." Every vector database listed above supports storing metadata alongside each vector and filtering on it as part of the search, though how efficiently that filtering is applied (before, during, or after the similarity search itself) varies meaningfully between systems and can materially affect both result quality and query latency at scale — worth benchmarking with your actual data and filter patterns rather than assuming any two vector databases behave identically here.

Do you actually need a dedicated vector database?

Not always. For a small dataset — tens of thousands of vectors rather than tens of millions — a brute-force linear scan computing exact similarity against every stored vector can be fast enough, especially with modern hardware and a reasonably sized embedding dimension, meaning you might not need approximate search or specialised indexing at all. For a team already running Postgres, MongoDB, Elasticsearch or Redis, several of these now offer a vector-search capability of their own that may be entirely sufficient without introducing a new dedicated system. The genuine case for a purpose-built vector database — or a lightweight extension like pgvector — shows up once dataset size, query latency requirements, or the sophistication of the retrieval logic (hybrid search, complex metadata filtering at scale, multi-tenant isolation) outgrow what a bolted-on capability in an existing system comfortably handles.

Step-by-step: how a document ends up searchable

1. A source document (a PDF, a support article, a product description) is split into smaller chunks — usually a few hundred to a couple thousand characters each, since embedding an entire long document as one vector loses too much specific detail.
2. Each chunk is passed through an embedding model, which outputs a fixed-length vector representing that chunk's meaning.
3. The vector, along with the original chunk text and any relevant metadata (source document, page number, access permissions), is stored in the vector database.
4. At query time, the user's search query or question is passed through the same embedding model to produce a query vector.
5. The vector database performs an approximate nearest-neighbour search to find the stored vectors closest to the query vector, optionally filtered by metadata.
6. The original text chunks behind those closest vectors are returned — either shown directly as search results, or passed to an LLM as retrieved context in a RAG pipeline.

Hybrid search — combining vector and keyword search

Pure semantic similarity search is excellent at understanding intent and synonyms, but it can occasionally underperform a simple keyword match on very specific terms — a product SKU, an exact error code, an uncommon proper noun the embedding model wasn't trained to weight heavily. Hybrid search addresses this by running both a traditional keyword-based search (often using BM25, the same family of ranking algorithm behind classic full-text search) and a vector similarity search, then combining the two result sets with a weighting or re-ranking step. Several vector databases, including Weaviate and Qdrant, support this natively; teams using a pure vector-only system sometimes implement hybrid search themselves by running a separate keyword search alongside the vector query and merging results in application code. For a RAG pipeline answering questions that mix natural language with specific identifiers or codes, hybrid search is frequently the difference between reliably retrieving the right document and missing it because the exact term didn't carry enough semantic weight on its own.

Choosing an embedding model and dimension size

The embedding model you choose determines both the quality of your similarity search and the practical cost of running it — larger embedding dimensions (1536, 3072, or more) generally capture more nuance but cost more to store and search, while smaller dimensions (384, 768) are cheaper and faster but can lose some of that nuance for subtle distinctions. Different embedding models also aren't interchangeable: a vector produced by one model's embedding of a sentence cannot be meaningfully compared against a vector produced by a different model, even for the exact same input text, since each model learns its own internal geometry for representing meaning. This has a very practical consequence — if you ever switch embedding models, every previously stored vector needs to be regenerated with the new model and re-indexed; there's no way to mix embeddings from two different models in the same similarity search and get sensible results.

Evaluating retrieval quality

It's tempting to treat "the search returns something" as success, but a genuinely useful evaluation checks whether the *right* results come back, and in a reasonable order. A common approach is building a small test set of realistic queries paired with the documents or chunks that should ideally be retrieved for each one, then measuring recall (of the results that should have been retrieved, how many actually were) and precision (of the results that were retrieved, how many were actually relevant) against that test set. This matters more than it might seem for a RAG pipeline specifically, because a language model given irrelevant retrieved context doesn't just fail to help — it can confidently generate an answer based on the wrong material, which is often harder to notice than an obviously empty or broken search result would be.

Common mistakes when working with vector databases

• Using inconsistent chunk sizes or splitting strategies between indexing and querying, which can quietly hurt retrieval quality without an obvious error.
• Re-embedding content with a different model version than what's already stored, producing vectors that aren't comparable to the existing ones (embeddings from different models generally can't be meaningfully compared against each other).
• Skipping metadata filtering entirely and relying on similarity alone, even when access control or basic relevance filters (like date range or category) are clearly needed.
• Assuming approximate search always returns identical results to an exact search, then being confused when a tuned index occasionally misses a genuinely relevant but borderline-distant match.
• Choosing a fully managed, proprietary system for a small-scale internal tool where a simple extension like pgvector on an existing Postgres instance would have been far simpler to operate.