A retrieval-augmented generation pipeline is four decisions in a trench coat: how you cut documents into chunks, which embedding model turns those chunks into vectors, how you retrieve the right chunks for a question, and how much of the retrieved context you actually hand to the language model. Get any one wrong and the system answers confidently from the wrong paragraph. This guide works each decision with the vendors' own published figures, so you can price and size a pipeline before you build it.
Who this is for, and who should stop here
This playbook is for an operator with a corpus worth searching: more than 10,000 pages of documentation, support tickets, contracts, or product manuals, where the questions are specific and the answers live inside your own text. If your corpus is under 1,000 pages and the questions are generic, do not build this. A hosted search product or a plain language-model call over pasted context will answer faster and cheaper, and the pipeline you build will cost more to maintain than the problem is worth. If your documents are regulated — medical, legal, financial — the pipeline below still applies, but retrieval must never be the only control: access rules belong in the database, not in the prompt.
The four stages, and what each one costs you
| Stage | The decision | The wrong default | Authority |
|---|---|---|---|
| Chunking | Chunk size, overlap, boundaries | One fixed size for every document type | Operator practice; Qdrant hybrid-search docs |
| Embedding | Model, dimensions, re-embedding policy | The biggest model, always | OpenAI embeddings guide |
| Retrieval | Dense only, hybrid, or hybrid plus re-rank | Dense-only search on keyword-heavy questions | Qdrant hybrid-search docs |
| Context | How many chunks reach the model | Stuffing the whole window | Anthropic context-window docs |
Chunking: where quality is won or lost
A chunk is the unit the model sees. Too large and the embedding averages away the specific sentence that answers the question; too small and the chunk lacks the context to be understood on its own. Three strategies cover nearly every corpus.
| Strategy | How it cuts | Best for | Failure mode |
|---|---|---|---|
| Fixed-size | Every N tokens, regardless of structure | Uniform prose, logs, transcripts | Splits answers across a boundary |
| Recursive | Splits on paragraphs, then sentences, then words | Mixed documents with headings | Uneven chunk sizes complicate budgeting |
| Semantic | Splits where meaning shifts | Research papers, contracts, long reports | Expensive to compute; boundaries drift |
Start with recursive chunking at roughly 500 tokens per chunk with an overlap of 10–20% of the chunk size — 50 to 100 tokens carried from the previous chunk. The overlap is the cheap insurance against a boundary split: an answer that straddles two chunks is still retrievable from either side. A worked example: a 2,000-token article at 500-token chunks with 100-token overlap produces 5 chunks, not 4. That 25% embedding overhead is the correct trade — retrieval misses cost far more than embedding tokens.
Metadata is the second half of chunking. Store the source document, the section, the publication date, and the access level with every chunk. Retrieval can then filter by date or permission before ranking.
Embedding models: the price table that decides the budget
OpenAI's embeddings guide publishes the comparison that matters. Its table prices models in pages per dollar, assuming roughly 800 tokens per page, and scores them on the MTEB evaluation.
| Model | Dimensions | MTEB score | Pages per dollar | Max input |
|---|---|---|---|---|
| text-embedding-3-small | 1,536 | 62.3% | 62,500 | 8,192 tokens |
| text-embedding-3-large | 3,072 | 64.6% | 9,615 | 8,192 tokens |
| text-embedding-ada-002 | 1,536 | 61.0% | 12,500 | 8,192 tokens |
The authority column is the OpenAI embeddings guide, read 19 November 2026. Two facts in that table do most of the deciding. First, the large model scores 2.3 points higher on MTEB and costs roughly 6.5 times more per page. Second, ada-002 is dominated: it scores lower than the small model and costs five times more. There is no case left for starting a new pipeline on ada-002.
The pages-per-dollar figures convert directly into a per-token price, and the conversion is worth showing because every other cost in this guide hangs off it:
For text-embedding-3-small: 62,500 pages per dollar times 800 tokens per page is 50,000,000 tokens per dollar, or $0.02 per 1M tokens. For text-embedding-3-large: 9,615 times 800 is 7,692,000 tokens per dollar, or $0.13 per 1M tokens. The worked corpus math follows. A corpus of 100,000 documents at 800 tokens each is 80,000,000 tokens: $1.60 to embed once on the small model, $10.40 on the large model. Embedding is the cheapest stage of the pipeline by two orders of magnitude. Anyone who tells you the embedding bill is the problem has not done this arithmetic.
One rule has no exception: vectors from different models are not comparable. If you change embedding models, you re-embed the entire corpus in one pass and swap the index atomically. A mixed index returns rankings that mean nothing, and the failure looks like bad retrieval rather than what it is, a corrupted index.
Retrieval: dense, keyword, and the fusion between them
An embedding is a vector of floating-point numbers, and the distance between two vectors measures relatedness. OpenAI recommends cosine similarity, and notes that its embeddings are normalized to length 1, which makes cosine similarity computable as a plain dot product. That is the dense half of retrieval. It is excellent at meaning and weak at exact terms: ask for "Section 409A" and dense search may return the paragraph about deferred compensation generally while missing the one that prints the section number.
Keyword search has the opposite shape. BM25 matching finds the exact term and misses the synonym. The production answer is hybrid: run both, then fuse the rankings. Qdrant's hybrid-search documentation describes the pattern precisely — a prefetch runs a BM25 query and a dense vector query, each returning its top 20 candidates, and the main query fuses them into a final top 10. The fusion most operators use is reciprocal rank fusion, and Qdrant publishes its formula with the constant k set to 2 by default and zero-based rank positions:
A worked example makes the formula concrete. A document ranked first by dense search (rank 0) and third by BM25 (rank 2) scores 1 / (2 + 0) + 1 / (2 + 2) = 0.50 + 0.25 = 0.75.
A document ranked first by only one method scores 0.50. The fusion rewards agreement between the two methods, which is exactly the property you want when one method is lexical and the other is semantic. Qdrant also offers distribution-based score fusion as an alternative; reciprocal rank fusion is the default most teams should start with because it needs no score calibration between methods that produce incomparable numbers.
Implementation notes: the retrieval layer is usually a REST API over HTTP that returns JSON, so the client is a thin SDK call rather than a bespoke integration. The vector store behind it typically indexes with HNSW graphs, and bulk embedding runs on GPU-backed workers; a SQL store such as pgvector keeps vectors beside the relational data when the corpus is small.
Set the prefetch limit generously and the final limit tightly. Fetching 20 candidates from each method and returning 10 fused results is the shape in Qdrant's own example: the expensive model call only ever sees the final 10, while the cheap retrieval stages had room to disagree.
Re-ranking: the cheapest accuracy you can buy
Retrieval returns candidates; re-ranking re-scores them. Cohere's Rerank models sort the retrieved documents by semantic relevance to the query, and the current generation — rerank-v5.0-pro and rerank-v5.0-fast — carries a 32,768-token context length per document, returns calibrated relevance scores between 0 and 1, and offers Context Focus, which returns the most relevant passages inside each document so you send only those passages to the language model. The older rerank-v3.5 generation is limited to a 4,096-token context. The practical rule: retrieve 20 to 50 candidates, re-rank them, and send the top 5 to 8 to the model. Re-ranking costs a fraction of a generation call and routinely moves the answering chunk from position 12 to position 2, which is where the model's attention actually is.
One honest gap in the pricing picture: Cohere's pricing page, read 19 November 2026, currently lists enterprise and contact-sales pricing with no self-serve per-search rate published for Rerank. Do not invent a per-search price for your budget. Carry re-ranking as a quoted line item, get the number in writing from the vendor, and size the rest of the pipeline on the embedding and generation figures below, which are published.
Context budgeting: the window is not the budget
Modern models carry enormous windows — Anthropic documents 1M-token context windows on its current Claude models, with 200k tokens on older ones — and that number misleads operators into thinking retrieval can be sloppy. Anthropic's own documentation names the failure: as token count grows, accuracy and recall degrade, a phenomenon it calls context rot. More context is not automatically better. The budget formula is:
A worked example on a 200,000-token window: a 2,000-token system prompt and a 4,000-token output reserve leave 194,000 usable tokens, which at 500 tokens per chunk is 388 chunks. The window is not the constraint — the model's attention is. Send the top 5 to 8 re-ranked chunks, roughly 2,500 to 4,000 tokens of context, and spend the remaining window on nothing. If the answer is not in those chunks, the fix is better retrieval, not a bigger window. When a conversation must run long, Anthropic's documented answer is server-side compaction — summarizing earlier turns on the server — not stuffing the full history into the window.
What a query actually costs
Generation dominates the bill, and the arithmetic is short. OpenAI's pricing page, read 19 November 2026, lists gpt-6.2-sol at $2.00 per 1M input tokens and $10.00 per 1M output tokens. A query that sends 8 chunks of 500 tokens plus a 500-token system prompt is 4,500 input tokens, or $0.009. A 300-token answer is $0.003. The query embedding is roughly 50 tokens at $0.02 per 1M tokens, or $0.000001. Total: about $0.012 per query, or $12.00 per 1,000 queries before re-ranking. At 100,000 queries a month, generation costs $1,200 and embedding $0.10. The corpus is nearly free to embed and the query is cheap; the only number that moves the budget is how many tokens you send to the model per query. Cut chunks, not corners.
Run that arithmetic through our ROI calculator before you commit to a build: the pipeline pays for itself only when the queries it answers would otherwise consume staff time, and the calculator turns the per-query cost above into a monthly break-even headcount figure. Our vector database architecture guide covers the storage layer this pipeline sits on — Qdrant, Pinecone, and pgvector on cost, latency, and operations — and our self-hosting LLMs cost guide prices the alternative of running the models yourself instead of paying per token.
Founder & CEO Insight — Enow A. Jovial
Failure modes worth naming before launch
- Stale index. Documents change; embeddings do not update themselves. Re-embed on every document change and version the index with the corpus.
- Chunk-boundary answers. The answering sentence sits across two chunks and neither ranks high enough alone. Overlap and heading-prefixed chunks are the mitigation.
- Model-change corruption. Swapping embedding models without a full re-embed — the silent killer, because the system keeps answering.
- Unmeasured quality. A pipeline with no evaluation set drifts for months. Build the eval set before launch: 50 to 100 real questions with the document that answers each one. Measure recall at k — the share of questions whose answering document appears in the top k retrieved chunks — and a faithfulness check that every claim in the answer traces to a retrieved chunk. The worked numbers in this guide are arithmetic from vendor-published prices, not lab results; your recall numbers must come from your own eval set.
Build checklist
- Inventory the corpus. Count documents, tokens, and update frequency before choosing anything. A corpus that changes daily needs an incremental embedding pipeline; a static corpus does not.
- Pick the chunking strategy. Recursive chunking at roughly 500 tokens with 10–20% overlap, headings prefixed, metadata attached.
- Pick the embedding model. Start with text-embedding-3-small at $0.02 per 1M tokens; move to the large model only when the eval set shows a retrieval gap the small model cannot close.
- Build retrieval hybrid. BM25 plus dense, fused with reciprocal rank fusion at k = 2, prefetch 20 per method, final top 10.
- Add re-ranking. Retrieve 20 to 50, re-rank with a current-generation model, send the top 5 to 8 to the language model.
- Budget the context. Apply the usable-context formula, cap the sent chunks at 8, and reserve output tokens before counting.
- Ship the eval set. Fifty to 100 questions with known answers, recall at k and faithfulness measured on every deploy, with a rollback when either regresses.