A model doesn't know your internal documents. It doesn't know your refund policy, the decision you made in a meeting last March, the schema of your product, or the reason your team stopped using a particular vendor. None of that was in its training data, and none of it is in the context unless you put it there. Retrieval is the discipline of getting the right piece of your own material into the context at the moment it's needed. It's the R in RAG, and it's the part everyone overbuilds.
The problem retrieval actually solves
The failure looks like this: you ask the model a question that depends on a fact only your company knows, and it either makes something up or gives you a generic answer that's true on the internet but wrong for you. The model isn't broken. It simply never saw the document that has the answer.
There are only two ways a model knows anything. Either it learned a pattern during training, or you handed it the information in the current context. Your internal knowledge lives in exactly one of those places, and it's not the training data. So the whole job of retrieval is narrow and unglamorous: find the specific document, or the specific paragraph, that answers the question, and put it in front of the model before it answers. That's it. RAG is not a magic knowledge layer. It's "get the right file in front of the model," done at query time instead of by hand.
Once you frame it that way, the question stops being "which vector database" and becomes "what's the simplest thing that reliably surfaces the right material." Usually the answer is much lighter than the industry wants you to believe.
The spectrum, simplest to heaviest
Retrieval is a spectrum, not a product. Here's the ladder I actually climb, and I stop at the lowest rung that works.
-
Paste the relevant document. If you already know which doc answers the question, paste it into the context. No infrastructure, no indexing, no failure modes. This covers a shocking number of real cases: a policy, a spec, a single reference file. If a human on your team would answer by opening one known document, your AI should too.
-
Give the model a tool to read and search your files directly. When you don't know in advance which document is relevant, don't pre-index everything; give the model the ability to look. A tool that lists a directory, greps for a term, and reads a file by path lets the model navigate your corpus the way you would. This is retrieval as tools, not chat, and it's the rung most teams skip straight past.
-
Keyword or full-text search. When the corpus is too big to hand the model a directory listing, put a real index in front of it: SQLite FTS, Postgres full-text, ripgrep across a repo, whatever fits. Keyword search is fast, cheap, debuggable, and it fails in ways you can read. When it returns the wrong thing, you can see exactly why.
-
Embeddings and vector search. Only when the corpus is genuinely too large to grep, and when meaning-based matching actually matters, do you reach for embeddings. Vector search shines when the user's words won't match the document's words: they ask about "time off" and the policy says "paid leave." That's a real problem, and vectors solve it. But it's a specific problem, not the default one.
The mistake is starting at rung four. Teams stand up an embeddings pipeline, a vector store, a chunking strategy, and a re-ranking step before they've confirmed that a direct file read wouldn't have answered the question. You've now got infrastructure to maintain, embeddings to keep in sync, and a new class of bugs, in exchange for solving a problem you might not have had.
Why "start by pasting the file" usually wins
Most internal corpora are smaller and more structured than people admit. My agents work against a real one: an Obsidian vault backed by git, with project documentation organized by stack and a Johnny Decimal filing scheme. When an agent needs the CRM contact schema or the gotchas file for a given project, the correct retrieval is a direct file read at a known path, or a scoped search inside one folder. There's no vector store anywhere in that fleet, and there doesn't need to be. The structure is the index.
This is the general case, not a quirk of my setup. If your knowledge lives in a repo, a wiki, or a folder tree that a person can navigate, your AI can navigate it the same way, and it will be more accurate for it. A direct read gives the model the whole document, in order, with its headings and its context intact. A vector search gives it three chunks that scored well on cosine similarity and may have been sliced mid-sentence. When you already know where the answer lives, similarity math is a downgrade, not an upgrade.
I have built a genuine multi-tenant RAG system with embeddings and vector search, and there the complexity was justified: a large corpus, per-tenant isolation, and queries where the user's phrasing rarely matched the source text. That system earns its keep. The point isn't that vector search is wrong. It's that it's a specific tool for a specific scale, and you should be able to say out loud why you've reached that scale before you build for it.
The failure modes are context failures
Retrieval doesn't remove the hard part of working with a model. It relocates it. Whatever you retrieve becomes context, and it inherits every way context can go wrong. I wrote about that at length in give your AI the context it needs; retrieval is just an automated way of filling that context, which means it can fill it badly.
-
Fetching the wrong chunk. Vector search returns the nearest neighbors, not the correct answer. Nearest is not the same as right. A chunk about last year's pricing scores just as well as this year's, and the model, having been handed it confidently, answers from it confidently.
-
Stale documents. Retrieval is only as current as your index. If the source changed and the embeddings didn't, you're serving the model a fact that was true last quarter. Keyword and direct-read approaches sidestep this because they read the live file; embedding pipelines have to be re-synced, and someone has to own that.
-
Retrieving too much and drowning the signal. The tempting fix for "it missed the answer" is to retrieve more: top-twenty chunks instead of top-three, the whole folder instead of the one file. Now the answer is in there somewhere, buried under noise, and the model's attention is spread thin across a wall of marginally relevant text. More retrieved context is not more accuracy. Past a point it's less.
Every one of these is a context-quality problem wearing a retrieval costume. The fix is the same discipline you'd apply to anything you feed a model by hand.
Treat retrieved context like any other context
The bar for automatically retrieved material is exactly the bar for material you paste yourself: relevant, current, and not bloated. Retrieval doesn't lower that bar because a machine did the fetching. If anything it raises it, because you're no longer eyeballing what goes in.
So apply the same three checks. Relevant: does the retrieved passage actually bear on the question, or did it just score well? Narrow the query, scope the search to the right folder, prefer a precise read over a fuzzy match. Current: is this the live version of the document? Prefer reading the source of truth over a copy that has to be kept in sync. Not bloated: retrieve the least that fully answers the question. If a one-page spec answers it, don't also pull in five loosely related notes because your top-k was set to six.
A related habit that pays off: have the model tell you what it retrieved and why, ideally as structured output rather than buried in prose. When an answer is wrong, the first question is always "what did it actually have in front of it," and a retrieval step that reports its sources turns a mystery into a five-minute fix. If the model cited the stale doc, you have your answer. If it cited nothing relevant, your retrieval missed, and you know which layer to look at.
The honest version
Retrieval is not a capability you bolt on. It's the plumbing that gets your own material in front of a model that would otherwise be guessing, and the best plumbing is the least plumbing that works. Start by pasting the file. Give the model a tool to read your files when you don't know which one. Add keyword search when the corpus outgrows a directory listing. Reach for embeddings when, and only when, scale and meaning-based matching make the case for you.
The teams that get burned aren't the ones who kept it simple too long. They're the ones who built the vector pipeline first and spent the next six months debugging why it kept handing the model the wrong paragraph. Get the right document in front of the model. Everything else is an optimization you should have to justify.