IBRAVRA Media Network
AI Tools • 12 min read

RAG in Production: The Deployment Playbook for Chunking, Indexing, Latency, and Cost

A working prototype is not a production system. This playbook runs the RAG operation end to end — chunking and embedding refresh, index builds and re-index cadence, latency budgets, CI evaluation gates, fallbacks, and reproducible cost math at stated query volumes.

By Enow A. Jovial • Published 2026-11-19

A retrieval system that answers well in a notebook usually breaks in production for unglamorous reasons: the index goes stale, a chunk boundary cuts an answer in half, retrieval takes 900 milliseconds instead of 90, or the bill arrives ten times the estimate. None of those failures is a model problem — each is an operations problem, with a number attached. This playbook covers that operating side: chunking and embedding refresh, index build and re-index cadence, latency budgets, evaluation gates in CI, fallback behaviour, and cost math from vendor prices read on November 19, 2026.

It is written for the team with a working RAG prototype that must now run it for real users: a corpus that changes daily, wrong answers that cost money or trust, volume about to grow tenfold. A few hundred unchanging documents need a script and a local index, not this playbook.

The production shape

A production RAG system has six stages — ingest, chunk, embed, index, retrieve, generate — each with its own owner and failure mode. Ingest versions documents; chunking assigns stable identifiers; embedding records model name, version, and dimensions; indexing writes vectors with the filters the application uses; retrieval narrows candidates; generation logs answers with chunk identifiers cited.

The rule for all six: never let two stages change at once. Change the chunk size without re-embedding, and new chunks carry vectors from the old segmentation. Change the embedding model without rebuilding the index, and queries embed into a space the stored vectors do not share. Every change to chunking, embedding model, or distance metric is a full rebuild — the cost section prices it, and it is small enough that there is no excuse for skipping it.

A RAG system fails at the seams between stages, not inside the model. Version every stage, change one at a time, and treat a chunking or embedding change as a full rebuild.

Chunking and embedding refresh

Chunking decides what retrieval can ever find: a chunk mixing two subjects matches either weakly, and a chunk cut mid-sentence hands the generator a fragment. The tooling defaults are a starting point: LlamaIndex documents its SentenceSplitter with an example of chunk_size 1024 and chunk_overlap 20, and LangChain's text-splitters package ships a RecursiveCharacterTextSplitter that tries separators in order.

Pick the chunk size from the shape of your documents. Long prose takes 800–1,000 tokens with 10–15% overlap; short records take one record per chunk with no overlap; code splits on structural separators; tables stay intact or split by row group, because a split table returns a number without its header, which is how wrong figures reach users.

Overlap is insurance, not free: a 20% overlap on a million chunks adds roughly 20% to the embedding and storage bill. If removing it does not move recall on your golden set, remove it.

Embedding refresh has three triggers, and only three: the source document changes, the embedding model or its version changes, or the chunking rules change. A calendar schedule is a backstop, not a trigger — 90 days for fast-moving corpora, 180 days for slow ones. Keep a manifest: document and version, chunk identifiers, embedding model and version, last embed timestamp. When a user reports a stale answer, the manifest tells you in one lookup whether the document was re-embedded after its last edit.

The embedding model choice is a cost and dimension decision first. OpenAI's pricing page, read on November 19, 2026, lists text-embedding-3-small at $0.02 per 1 million tokens, text-embedding-3-large at $0.13, and text-embedding-ada-002 at $0.10. The embeddings guide gives the default dimensions: 1,536 for the small model, 3,072 for the large model, with a parameter to reduce dimensions. The large model costs 6.5 times the small model per token and stores twice the bytes. Unless your golden set shows the large model retrieving answers the small model misses, use the small model.

Store the embedding model name and version on every chunk, not just in a deployment note. When a model version changes silently behind an API, the per-chunk record is the only proof of staleness.

Index build and re-index cadence

The index is where retrieval speed is decided. Qdrant's indexing documentation describes HNSW as a graph-based index and payload indexes as the mechanism that speeds up filtering: index only fields used in filtering, and build the HNSW index after the payload indexes exist so the graph can use filter-aware edges. pgvector documents the same knobs with different defaults — HNSW m of 16, ef_construction of 64 — and states that a higher ef_construction gives better recall at the cost of build time and insert speed.

Rebuild on a schedule tied to corpus change: more than 5% of chunks changing per week means a weekly rebuild; 1–5% monthly; below 1% quarterly — with upserts between. Any change of embedding model, chunking rule, or distance metric forces a rebuild out of cycle.

Incremental upserts keep the index fresh between rebuilds, but do not fix drift. Deleted documents leave vectors behind unless deletion is an explicit step, and updated documents leave the old chunk beside the new one unless the old identifier is removed in the same write. Every upsert job must log inserted, updated, and deleted counts. A week with zero deletions in a corpus that edits daily is a pipeline fault, not a quiet week.

Quantization cuts storage and speeds search. Scalar quantization compresses each component from a 32-bit float to an 8-bit integer — a 4x reduction, error usually under 1%. Binary quantization reduces each component to a single bit, a 32x reduction with up to a 40x speedup — but only for high-dimensional vectors, and it needs rescoring. Product quantization reaches up to 64x when memory binds. Qdrant's 2024 sweep found binary quantization with 3x oversampling and rescoring at 0.9847 recall at 100 for text-embedding-3-small and 0.9966 for text-embedding-3-large.

Qdrant quantization documentation showing scalar quantization at 4x and binary quantization recall figures
Qdrant quantization documentation showing scalar quantization at 4x and binary quantization recall figures
QuantizationCompressionCostWhen to use itAuthority
None, float321xFull memorySmall corporaQdrant docs
Scalar, uint84xError usually under 1%Default once recall is baselinedQdrant docs
Binary, 1 bit32xNeeds rescoringLarge corporaQdrant docs
ProductUp to 64xAccuracy and speed trade-offsMemory is the top priorityQdrant docs

One silent trap: pgvector's vector type supports up to 2,000 dimensions and halfvec up to 4,000, while text-embedding-3-large produces 3,072 by default — too many for vector. Moving to the large model means switching to halfvec or reducing dimensions, and rebuilding the index either way.

Retrieval latency budgets

Latency is a budget, not a hope: set a total p95 budget, divide it across the stages, and alert on the stage that breaches its share. The numbers below are a starting budget — adjust the split to yours, but keep it written.

Stagep95 budgetWhat consumes it
Query embedding100 msThe embedding API round trip
Vector search150 msHNSW traversal, filtering, oversampling
Reranking200 msScoring the candidate set against the query
First generated token350 msModel call setup, first token
Total800 msRetrieval plus first token

One formula enforces the budget per stage:

Stage share=Stage p95 latencyTotal p95 budget×100\text{Stage share} = \frac{\text{Stage p95 latency}}{\text{Total p95 budget}} \times 100

A stage over its share on more than 5% of queries in a rolling hour is a breach, and the response is a named action: vector search over budget means raising ef or adding quantization; reranking over budget means shrinking the candidate set; the embedding call over budget means checking whether the query is embedded twice. Log latency per stage on every query — averages hide the breach; p95 shows it.

A retrieval path with no per-stage latency log cannot be fixed, only argued about. If you cannot say which stage consumed the 800 milliseconds, you have a measurement problem — fix the instrumentation first.

Evaluation gates in continuous integration

Evaluation is the gate that stops a bad index from reaching users, and it belongs in continuous integration, not in a notebook run. The metric definitions live in the RAG evaluation metrics guide on Ragas, hallucination, and precision; this playbook covers where those metrics sit in the pipeline and what blocks a deploy.

The gate needs three artifacts, all versioned: a golden set of 50 to 200 question-and-answer pairs drawn from real user questions, the expected source chunk for each, and the threshold each metric must clear. Set thresholds from your own golden set and write them down before the first run, so a failing gate cannot be redefined as a pass.

Four checks run on every change. Recall runs on pull requests touching chunking, embedding, or index configuration, blocking below the written threshold or when a golden question's chunk leaves the top 10. Faithfulness runs on pull requests touching the prompt or generator, blocking when an answer cites a chunk that does not support it. Latency and cost checks run on every staging deploy, blocking on a p95 breach or a cost above the written ceiling.

Fallback when retrieval fails

  1. Empty result set. Retry once with filters relaxed, then once with the query rewritten to its key terms. If both return nothing, say so and offer the closest available topic — never answer from no sources.
  2. Timeout. Return the best partial result set if one exists, marked as retrieved under a timeout. If none exists, fail with a retryable error.
  3. Low-confidence results. Widen the candidate set and rerank, then answer only from chunks above the threshold; if none clears it, refuse and state why.
  4. Index unavailable. Serve from the last known-good snapshot if one is configured, and alert immediately. If none exists, say the retrieval path is down rather than answering without retrieval.

Log every fallback with its cause, query, and chunks used. A fallback rate above 2% of queries in a rolling day is an operations incident — the fix belongs in the pipeline, not in the fallback code.

Founder & CEO Insight — Enow A. Jovial

"A RAG system is a promise that the answer comes from your documents. The moment it answers from nothing and sounds confident doing it, that promise is broken — and no latency budget or cost saving repairs a broken promise. Build the fallback first, then the feature."
— Enow A. Jovial, Founder & Chief Executive Officer

The cost math, reproduced

Prices below were read from the vendors' own pages on November 19, 2026. The worked example uses a stated planning corpus: 1,000,000 chunks averaging 400 tokens, embedded with text-embedding-3-small.

Corpus tokens=1,000,000×400=400,000,000\text{Corpus tokens} = 1{,}000{,}000 \times 400 = 400{,}000{,}000
Embedding cost=400,000,0001,000,000×$0.02=$8.00\text{Embedding cost} = \frac{400{,}000{,}000}{1{,}000{,}000} \times \$0.02 = \$8.00

One full re-embed costs $8.00 at the small model's price; the same corpus on text-embedding-3-large costs $400 \times \$0.13 = \$52.00$.

Storing the vectors:

Vector bytes=1,000,000×1,536×4=6,144,000,000 bytes=6.144 GB\text{Vector bytes} = 1{,}000{,}000 \times 1{,}536 \times 4 = 6{,}144{,}000{,}000 \text{ bytes} = 6.144 \text{ GB}

Pinecone lists serverless storage at $0.33 per GB per month, so raw vector storage costs $6.144 \times \$0.33 = \$2.03$ per month. Scalar quantization divides that by four, to about 1.54 GB and $0.51 per month; binary by 32, to about 0.19 GB and $0.06 per month. Payloads, metadata, and the index sit on top — treat these as the floor of the storage bill.

Pinecone lists serverless write units from $4 per million and read units from $16 per million on the Standard tier — $24–$27 per million for reads on Enterprise — both varying by cloud and region. The free Starter tier includes up to 2 GB of storage, 2 million write units, and 1 million read units a month. The page does not publish a unit-per-query mapping in the material opened this cycle, so the model below counts one read unit per query and one write unit per vector written: a planning assumption to confirm against your own Pinecone usage dashboard.

At 100,000 queries a month, each query about 50 tokens:

Query embedding cost=100,000×501,000,000×$0.02=$0.10 per month\text{Query embedding cost} = \frac{100{,}000 \times 50}{1{,}000{,}000} \times \$0.02 = \$0.10 \text{ per month}
Retrieval cost=100,0001,000,000×$16=$1.60 per month\text{Retrieval cost} = \frac{100{,}000}{1{,}000{,}000} \times \$16 = \$1.60 \text{ per month}
Cost per 1,000 queries=$0.10+$1.60100=$0.017\text{Cost per 1,000 queries} = \frac{\$0.10 + \$1.60}{100} = \$0.017

At 1,000,000 queries a month the same arithmetic gives $17.00 per million queries — the same $0.017 per 1,000, because both meters are linear.

A full re-index costs the re-embed plus the rewrite:

Re-index cost=$8.00 (re-embed)+1,000,0001,000,000×$4=$12.00\text{Re-index cost} = \$8.00 \text{ (re-embed)} + \frac{1{,}000{,}000}{1{,}000{,}000} \times \$4 = \$12.00

Twelve dollars per full rebuild is why the rebuild rule is affordable: the expensive mistake is serving stale vectors for a month to save $12.00. Price your own volume with the ROI calculator, compare managed against self-hosted in the self-hosting LLMs and vector databases cost guide, and read store trade-offs in the vector database comparison of Qdrant, Pinecone, and pgvector.

Managed serverless wins while volume is uncertain; self-hosted wins when corpus and query rate are known.

Pinecone serverless pricing showing storage at $0.33 per GB per month and read units from $16 per million
Pinecone serverless pricing showing storage at $0.33 per GB per month and read units from $16 per million
Monthly lineAmountAuthority
Query embeddings, 5M tokens$0.10OpenAI pricing, $0.02 per 1M tokens
Retrieval, 100,000 read units$1.60Pinecone pricing, read units from $16 per 1M
Vector storage, 6.144 GB raw$2.03Pinecone pricing, $0.33 per GB per month
Total$3.73Sum of the three lines above

The launch checklist

  • [x] Chunking rules versioned: size, overlap, separators recorded.
  • [x] Embedding model pinned: name, version, dimensions.
  • [x] Payload indexes before the HNSW build: only filtered fields.
  • [x] Quantization after a recall baseline: scalar by default; binary only with rescoring.
  • [x] Latency budget written per stage: p95 targets, with alerts.
  • [x] Evaluation gate in CI: golden set, thresholds, snapshot.
  • [x] Fallback ladder implemented and logged: all four failure modes planned.
  • [x] Cost model at current volume: priced from vendor pages.
  • [x] Re-index runbook tested: one full rebuild, timed and priced.
  • [x] Rollback restores code and index together: last passing snapshot recorded.

A prototype proves retrieval can work. Production proves it keeps working when documents change, when volume grows, and when the index fails at 2 a.m. The difference is not a better model: versioned chunks, a priced rebuild, a gated deploy, a fallback that tells the truth.