A retrieval system that answers well in a notebook usually breaks in production for unglamorous reasons: the index goes stale, a chunk boundary cuts an answer in half, retrieval takes 900 milliseconds instead of 90, or the bill arrives ten times the estimate. None of those failures is a model problem — each is an operations problem, with a number attached. This playbook covers that operating side: chunking and embedding refresh, index build and re-index cadence, latency budgets, evaluation gates in CI, fallback behaviour, and cost math from vendor prices read on November 19, 2026.
It is written for the team with a working RAG prototype that must now run it for real users: a corpus that changes daily, wrong answers that cost money or trust, volume about to grow tenfold. A few hundred unchanging documents need a script and a local index, not this playbook.
The production shape
A production RAG system has six stages — ingest, chunk, embed, index, retrieve, generate — each with its own owner and failure mode. Ingest versions documents; chunking assigns stable identifiers; embedding records model name, version, and dimensions; indexing writes vectors with the filters the application uses; retrieval narrows candidates; generation logs answers with chunk identifiers cited.
The rule for all six: never let two stages change at once. Change the chunk size without re-embedding, and new chunks carry vectors from the old segmentation. Change the embedding model without rebuilding the index, and queries embed into a space the stored vectors do not share. Every change to chunking, embedding model, or distance metric is a full rebuild — the cost section prices it, and it is small enough that there is no excuse for skipping it.
Chunking and embedding refresh
Chunking decides what retrieval can ever find: a chunk mixing two subjects matches either weakly, and a chunk cut mid-sentence hands the generator a fragment. The tooling defaults are a starting point: LlamaIndex documents its SentenceSplitter with an example of chunk_size 1024 and chunk_overlap 20, and LangChain's text-splitters package ships a RecursiveCharacterTextSplitter that tries separators in order.
Pick the chunk size from the shape of your documents. Long prose takes 800–1,000 tokens with 10–15% overlap; short records take one record per chunk with no overlap; code splits on structural separators; tables stay intact or split by row group, because a split table returns a number without its header, which is how wrong figures reach users.
Overlap is insurance, not free: a 20% overlap on a million chunks adds roughly 20% to the embedding and storage bill. If removing it does not move recall on your golden set, remove it.
Embedding refresh has three triggers, and only three: the source document changes, the embedding model or its version changes, or the chunking rules change. A calendar schedule is a backstop, not a trigger — 90 days for fast-moving corpora, 180 days for slow ones. Keep a manifest: document and version, chunk identifiers, embedding model and version, last embed timestamp. When a user reports a stale answer, the manifest tells you in one lookup whether the document was re-embedded after its last edit.
The embedding model choice is a cost and dimension decision first. OpenAI's pricing page, read on November 19, 2026, lists text-embedding-3-small at $0.02 per 1 million tokens, text-embedding-3-large at $0.13, and text-embedding-ada-002 at $0.10. The embeddings guide gives the default dimensions: 1,536 for the small model, 3,072 for the large model, with a parameter to reduce dimensions. The large model costs 6.5 times the small model per token and stores twice the bytes. Unless your golden set shows the large model retrieving answers the small model misses, use the small model.
Index build and re-index cadence
The index is where retrieval speed is decided. Qdrant's indexing documentation describes HNSW as a graph-based index and payload indexes as the mechanism that speeds up filtering: index only fields used in filtering, and build the HNSW index after the payload indexes exist so the graph can use filter-aware edges. pgvector documents the same knobs with different defaults — HNSW m of 16, ef_construction of 64 — and states that a higher ef_construction gives better recall at the cost of build time and insert speed.
Rebuild on a schedule tied to corpus change: more than 5% of chunks changing per week means a weekly rebuild; 1–5% monthly; below 1% quarterly — with upserts between. Any change of embedding model, chunking rule, or distance metric forces a rebuild out of cycle.
Incremental upserts keep the index fresh between rebuilds, but do not fix drift. Deleted documents leave vectors behind unless deletion is an explicit step, and updated documents leave the old chunk beside the new one unless the old identifier is removed in the same write. Every upsert job must log inserted, updated, and deleted counts. A week with zero deletions in a corpus that edits daily is a pipeline fault, not a quiet week.
Quantization cuts storage and speeds search. Scalar quantization compresses each component from a 32-bit float to an 8-bit integer — a 4x reduction, error usually under 1%. Binary quantization reduces each component to a single bit, a 32x reduction with up to a 40x speedup — but only for high-dimensional vectors, and it needs rescoring. Product quantization reaches up to 64x when memory binds. Qdrant's 2024 sweep found binary quantization with 3x oversampling and rescoring at 0.9847 recall at 100 for text-embedding-3-small and 0.9966 for text-embedding-3-large.
| Quantization | Compression | Cost | When to use it | Authority |
|---|---|---|---|---|
| None, float32 | 1x | Full memory | Small corpora | Qdrant docs |
| Scalar, uint8 | 4x | Error usually under 1% | Default once recall is baselined | Qdrant docs |
| Binary, 1 bit | 32x | Needs rescoring | Large corpora | Qdrant docs |
| Product | Up to 64x | Accuracy and speed trade-offs | Memory is the top priority | Qdrant docs |
One silent trap: pgvector's vector type supports up to 2,000 dimensions and halfvec up to 4,000, while text-embedding-3-large produces 3,072 by default — too many for vector. Moving to the large model means switching to halfvec or reducing dimensions, and rebuilding the index either way.
Retrieval latency budgets
Latency is a budget, not a hope: set a total p95 budget, divide it across the stages, and alert on the stage that breaches its share. The numbers below are a starting budget — adjust the split to yours, but keep it written.
| Stage | p95 budget | What consumes it |
|---|---|---|
| Query embedding | 100 ms | The embedding API round trip |
| Vector search | 150 ms | HNSW traversal, filtering, oversampling |
| Reranking | 200 ms | Scoring the candidate set against the query |
| First generated token | 350 ms | Model call setup, first token |
| Total | 800 ms | Retrieval plus first token |
One formula enforces the budget per stage:
A stage over its share on more than 5% of queries in a rolling hour is a breach, and the response is a named action: vector search over budget means raising ef or adding quantization; reranking over budget means shrinking the candidate set; the embedding call over budget means checking whether the query is embedded twice. Log latency per stage on every query — averages hide the breach; p95 shows it.
Evaluation gates in continuous integration
Evaluation is the gate that stops a bad index from reaching users, and it belongs in continuous integration, not in a notebook run. The metric definitions live in the RAG evaluation metrics guide on Ragas, hallucination, and precision; this playbook covers where those metrics sit in the pipeline and what blocks a deploy.
The gate needs three artifacts, all versioned: a golden set of 50 to 200 question-and-answer pairs drawn from real user questions, the expected source chunk for each, and the threshold each metric must clear. Set thresholds from your own golden set and write them down before the first run, so a failing gate cannot be redefined as a pass.
Four checks run on every change. Recall runs on pull requests touching chunking, embedding, or index configuration, blocking below the written threshold or when a golden question's chunk leaves the top 10. Faithfulness runs on pull requests touching the prompt or generator, blocking when an answer cites a chunk that does not support it. Latency and cost checks run on every staging deploy, blocking on a p95 breach or a cost above the written ceiling.
Fallback when retrieval fails
- Empty result set. Retry once with filters relaxed, then once with the query rewritten to its key terms. If both return nothing, say so and offer the closest available topic — never answer from no sources.
- Timeout. Return the best partial result set if one exists, marked as retrieved under a timeout. If none exists, fail with a retryable error.
- Low-confidence results. Widen the candidate set and rerank, then answer only from chunks above the threshold; if none clears it, refuse and state why.
- Index unavailable. Serve from the last known-good snapshot if one is configured, and alert immediately. If none exists, say the retrieval path is down rather than answering without retrieval.
Log every fallback with its cause, query, and chunks used. A fallback rate above 2% of queries in a rolling day is an operations incident — the fix belongs in the pipeline, not in the fallback code.
Founder & CEO Insight — Enow A. Jovial
The cost math, reproduced
Prices below were read from the vendors' own pages on November 19, 2026. The worked example uses a stated planning corpus: 1,000,000 chunks averaging 400 tokens, embedded with text-embedding-3-small.
One full re-embed costs $8.00 at the small model's price; the same corpus on text-embedding-3-large costs $400 \times \$0.13 = \$52.00$.
Storing the vectors:
Pinecone lists serverless storage at $0.33 per GB per month, so raw vector storage costs $6.144 \times \$0.33 = \$2.03$ per month. Scalar quantization divides that by four, to about 1.54 GB and $0.51 per month; binary by 32, to about 0.19 GB and $0.06 per month. Payloads, metadata, and the index sit on top — treat these as the floor of the storage bill.
Pinecone lists serverless write units from $4 per million and read units from $16 per million on the Standard tier — $24–$27 per million for reads on Enterprise — both varying by cloud and region. The free Starter tier includes up to 2 GB of storage, 2 million write units, and 1 million read units a month. The page does not publish a unit-per-query mapping in the material opened this cycle, so the model below counts one read unit per query and one write unit per vector written: a planning assumption to confirm against your own Pinecone usage dashboard.
At 100,000 queries a month, each query about 50 tokens:
At 1,000,000 queries a month the same arithmetic gives $17.00 per million queries — the same $0.017 per 1,000, because both meters are linear.
A full re-index costs the re-embed plus the rewrite:
Twelve dollars per full rebuild is why the rebuild rule is affordable: the expensive mistake is serving stale vectors for a month to save $12.00. Price your own volume with the ROI calculator, compare managed against self-hosted in the self-hosting LLMs and vector databases cost guide, and read store trade-offs in the vector database comparison of Qdrant, Pinecone, and pgvector.
Managed serverless wins while volume is uncertain; self-hosted wins when corpus and query rate are known.
| Monthly line | Amount | Authority |
|---|---|---|
| Query embeddings, 5M tokens | $0.10 | OpenAI pricing, $0.02 per 1M tokens |
| Retrieval, 100,000 read units | $1.60 | Pinecone pricing, read units from $16 per 1M |
| Vector storage, 6.144 GB raw | $2.03 | Pinecone pricing, $0.33 per GB per month |
| Total | $3.73 | Sum of the three lines above |
The launch checklist
- [x] Chunking rules versioned: size, overlap, separators recorded.
- [x] Embedding model pinned: name, version, dimensions.
- [x] Payload indexes before the HNSW build: only filtered fields.
- [x] Quantization after a recall baseline: scalar by default; binary only with rescoring.
- [x] Latency budget written per stage: p95 targets, with alerts.
- [x] Evaluation gate in CI: golden set, thresholds, snapshot.
- [x] Fallback ladder implemented and logged: all four failure modes planned.
- [x] Cost model at current volume: priced from vendor pages.
- [x] Re-index runbook tested: one full rebuild, timed and priced.
- [x] Rollback restores code and index together: last passing snapshot recorded.
A prototype proves retrieval can work. Production proves it keeps working when documents change, when volume grows, and when the index fails at 2 a.m. The difference is not a better model: versioned chunks, a priced rebuild, a gated deploy, a fallback that tells the truth.

