AI CostBase

RAG pipeline cost

Three separate bills in one system: index once (embedding your corpus), answer forever (LLM + retrieved context on every query), and keep it fresh (re-embedding churn + the vector DB).

The formula

corpus_tok = pages × words × 1.33 initial = corpus_tok/1M × E_in (one-time) refresh = initial × refresh% (monthly) per_query_in = chunks × chunkTok + query_tok llm = queries × (per_query_in/1M × P_in + ans/1M × P_out) × 30 monthly = llm + refresh + vectorDB

Token prices as of . Re-ranking adds a second small model call per query — not modeled by default. Chunk overlap inflates indexed tokens ~10–15%; the refresh% input is where you absorb that.

Three bills that behave nothing alike

A RAG system is not one expense. It is three, and they scale on different axes — which is why a single "cost per query" figure is always misleading.

BillPaidScales with
Initial indexingOnceCorpus size (one-time, then stops)
Answering queriesEvery query, foreverTraffic × retrieved context × model tier
Refresh and re-embeddingMonthlyChange rate of the corpus

The first bill is the one that shocks people. Compute the one-time index of a large corpus and it can look like a project in itself — but it is also the only bill that stops growing. The one that runs away is the middle row, and it runs away quietly because each individual query looks negligible.

The retrieved context is the bill, not the question

With the defaults on this page, five retrieved chunks at 400 tokens each supply about 2,000 tokens of context against a 150-token question. Roughly 93% of the input on every query is material you chose to retrieve, not text the user typed.

That single ratio is the most useful diagnostic in a RAG cost review. If retrieval context dominates your input, then tuning top-k, chunk size and re-ranking order will move your bill by multiples, while upgrading or downgrading the answering model will move it by a percentage. Teams routinely do the second thing because it is a one-line config change, and ignore the first because it requires thinking about retrieval quality.

Where teams get this wrong

Cutting the bill

Chunking is a cost decision, not only a quality decision

Chunk size sets the price of every query. Larger chunks mean fewer retrieval units are needed to cover a question, but each one carries more tokens into the model. Smaller chunks retrieve more precisely and then require a larger top-k to assemble enough context — which usually costs more than the precision saved.

Overlap compounds it. Ten to fifteen percent of indexed tokens are typically duplicated by overlap, and when overlapping chunks are returned together they inflate the per-query context as well. Retuning chunk size is a twenty-minute change that can move the query line by a multiple — which makes it the highest-leverage maintenance task in a retrieval system, and the one most often left sitting at its framework default.