Every June, a student brings us a project that "needs a vector database." Usually it's a chatbot that should answer questions from a few hundred PDFs. Sometimes it's a recommendation system for a small shop. And every time, the first question we ask is: do you actually need one? Most of the time, the answer is no. But to understand why, you have to understand what a vector database does, and that starts with embeddings.
What an embedding actually is
An embedding is a list of numbers — typically 384 to 1536 of them — that a machine learning model produces for a piece of text, an image, or an audio clip. The numbers are not meaningful on their own. The meaning lives in the relationships between them.
Take the sentence "The cat sat on the mat." A good embedding model will place this sentence close to "A feline rested on the rug" in the vector space. It will place it far from "The stock market closed higher today." The model learned these relationships by reading enormous amounts of text and adjusting the numbers so that similar meanings end up near each other.
The key insight is that distance in this space corresponds to semantic similarity. If you compute the cosine similarity between two embedding vectors, a value near 1 means the texts are about the same thing. Near 0 means they're unrelated. This is the foundation of everything a vector database does.
We tell students to think of it as a map of meaning. Each point on the map is a piece of content. Nearby points are related. A vector database is just a system that lets you ask: "Which points are closest to this one?" — and get an answer fast.
Why exact search doesn't scale
The naive way to find the nearest neighbors is to compute the distance from your query vector to every stored vector. For a thousand documents, that's a thousand distance calculations. For a million, it's a million. Each calculation involves multiplying and adding a few hundred numbers, so it's not slow in absolute terms — maybe a few milliseconds per query at a million vectors. But it doesn't scale. At ten million vectors, you're looking at tens of milliseconds per query, and the storage server is doing nothing but distance math.
More importantly, the cost grows linearly. Every new vector adds the same amount of work to every query. That's the problem vector databases solve. They don't store vectors in a plain list. They build an index that lets the database skip most of the vectors entirely.
Approximate nearest neighbor search
The trick is that you don't need the exact nearest neighbors. You need the nearest ones, approximately. If a search returns 95% of the truly closest items and misses a few, nobody notices. This is called approximate nearest neighbor search, or ANN.
The most common algorithm in production systems is HNSW — Hierarchical Navigable Small World graphs. The name sounds intimidating, but the idea is simple.
Imagine a graph where each node is a vector, and edges connect vectors that are close to each other. To find the nearest neighbor of a query, you start at any node, look at its neighbors, move to the one closest to the query, and repeat until you can't get closer. This is greedy search on a graph, and it works well because the graph is built so that close vectors are connected.
HNSW adds a twist: it builds multiple layers of the graph. The top layer has few nodes and long edges — think of it as a highway system. The bottom layer has all nodes and short edges — local streets. A search starts on the highway, quickly gets to the right region, then descends to the streets and finds the exact spot. This gives you logarithmic search time. A million vectors might require only a few hundred distance calculations instead of a million.
The trade-off is memory. HNSW keeps the graph in RAM, and the graph can be several times larger than the vectors themselves. A million 768-dimensional vectors take about 3 gigabytes of raw storage. The HNSW index can push that to 10 or 15 gigabytes. That's why vector databases are often memory-bound, not CPU-bound.
A concrete example
Let's make this real. Suppose you have a collection of 50,000 support tickets and you want to find similar past tickets when a new one arrives. Here's the minimal flow in Python, using a small embedding model and a vector database client.
from sentence_transformers import SentenceTransformer
import chromadb
model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.PersistentClient(path="./tickets_db")
collection = client.get_or_create_collection("tickets")
# Embed and store a ticket
ticket_text = "User cannot reset password after update"
embedding = model.encode(ticket_text).tolist()
collection.add(ids=["ticket_123"], embeddings=[embedding], documents=[ticket_text])
# Search for similar tickets
query = "Password reset fails after upgrade"
results = collection.query(query_embeddings=[model.encode(query).tolist()], n_results=3)
print(results["documents"])
That's the whole pattern. Embed the content, store the vectors, query with a vector. The database handles the index, the search, and the storage. You don't need to know how HNSW works to use it, but you need to know it exists, because it explains why the database behaves the way it does.
One thing we see every batch: students forget that the embedding model matters more than the database. If your embeddings are bad, no index will save you. A model trained on general English will do poorly on medical jargon or code. Test your embeddings on a small sample before you build the whole pipeline. We've seen projects where switching from a general model to a domain-specific one improved search quality more than any database tuning.
When you don't need a vector database
This is the part we wish more tutorials covered. A vector database is a tool with a specific job, and it's not always the right one.
If you have fewer than 10,000 items, a brute-force search over numpy arrays is fast enough. We measured it once: 10,000 vectors of 384 dimensions, cosine similarity, took about 15 milliseconds per query on a laptop. That's fine for a prototype or a small internal tool.
If your data is structured — rows with clear columns like price, date, category — a regular relational database with proper indexing will serve you better. Vector search is for unstructured data: free text, images, audio. If you can filter by a column value, do that first and then search the small remaining set.
If you need exact results — say, for a legal or medical application where missing a relevant document is unacceptable — approximate search is risky. You can mitigate it by increasing the number of candidates and re-ranking, but that adds complexity. Ask yourself whether approximate is acceptable before you commit.
And if you're building a chatbot that answers from a few hundred PDFs, you don't need a vector database. Load the embeddings into memory, compute similarities with numpy, and return the top three chunks. It will be fast enough, and you'll save yourself an infrastructure dependency.
What a vector database actually adds
So what do you get when you do need one? Three things.
First, persistence and scaling. The database manages the index on disk, handles incremental updates, and distributes across machines. You don't write your own index maintenance code.
Second, hybrid search. Most vector databases let you combine vector similarity with metadata filters. You can say "find the most similar tickets, but only from the last 30 days, and only with status 'open'." That's genuinely useful and hard to do efficiently on your own.
Third, operational features: replication, backup, access control, monitoring. These matter in production, and they're exactly the things you don't want to build yourself.
The honest answer is that for a prototype, you don't need any of this. For a production system serving thousands of queries per second over millions of vectors, you do. The line is somewhere in between, and it depends on your data size, your query rate, and your tolerance for building infrastructure.
Common questions
How do I choose between different vector databases like Pinecone, Weaviate, and Qdrant?
Start with your deployment constraints. Pinecone is a managed service — you pay and it runs. Weaviate and Qdrant can be self-hosted, which matters if you have data privacy requirements. Beyond that, test with your own data. The indexing algorithms are similar, so performance differences are usually small. Pick the one with the best documentation for your language and the simplest operational story for your team.
Do I need to fine-tune the embedding model for my specific domain?
Not always, but it helps when your domain has unusual vocabulary. A general model handles everyday English well. For legal, medical, or code-specific text, test a domain model first — many exist for free. Fine-tuning is a bigger step; only do it if a domain model doesn't exist or performs poorly. Measure retrieval quality on a small labeled set before and after.
Can I use a vector database with images or audio, or only text?
Yes. Any content that can be embedded can be searched. Image models like CLIP produce vectors that place similar images near each other. Audio models do the same for sound. The database doesn't care what the vector came from. The same search code works; only the embedding model changes.
What happens when I update or delete a vector in an HNSW index?
Updates and deletes are supported, but they're slower than inserts. HNSW is designed for a mostly-append workload. Deleting a node requires removing it from the graph and reconnecting its neighbors, which is more work than adding a new node. If your workload involves frequent updates, test the database's update performance with your data size before committing. Some databases handle this better than others.
Courses this relates to
Come and sit a class before you decide
Reading about it only gets you so far. Pick a day, sit in on a class that is already running, write some code, and talk to the students in it. Free, and nothing to pay afterwards unless you want to join.