What a RAG chatbot costs per month

Por Rohit Athaide · Publicado em 4 de nov. de 2026

Short answer: a retrieval-augmented generation (RAG) chatbot answers questions using your own documents. It looks up relevant passages, adds them to the prompt and asks a model to answer. For most chatbots, the model’s token charges are the largest and fastest-growing part. The vector store is a mostly fixed monthly cost, and ingestion is occasional.

Key points

• Parts: ingestion, vector store, query embedding, generation, hosting, logging.
• Generation tokens usually dominate and grow with use.
• Retrieved passages make prompts long.
• Caching and smaller models cut cost.
• Self-hosting only pays at high, steady volume.

What are the parts?

What does a worked example look like?

Made-up round numbers to show the method, not real prices. A support chatbot handles 30,000 questions a month. Each prompt holds 300 tokens of instructions, 100 of question and 5 passages of 400 tokens. Each answer is 250 tokens.

1. Input tokens per question: 300 + 100 + 5 x 400 = 2,400.
2. Monthly input: 30,000 x 2,400 = 72 million tokens.
3. Monthly output: 30,000 x 250 = 7.5 million tokens.

Generation is 328.50 of 459, or 72 percent. Now retrieve 3 passages instead of 5: input falls to 1,600 tokens per question, input cost to 144.00, and the total to about 387. A second change, caching the 300-token instructions at a cached rate of say a tenth, saves a little more. Passage count is the lever, since the question and answer are small.

Which part usually dominates?

For most chatbots, generation is the largest part and grows with use. Ingestion is a one-time or occasional cost. The vector store is mostly a fixed monthly cost. Retrieved passages are what make prompts long.

How do I estimate it?

1. Questions per month.
2. Input tokens per question: instructions, history, retrieved text and the question.
3. Output tokens per answer.
4. Multiply by the model’s per-million rates.
5. Add the fixed parts.

Use the AI cost estimator, which has presets for tasks such as support chats and shows the formula for each figure.

How can I reduce it?

• Retrieve fewer, better passages. Better search beats longer context.
• Cache repeated prompts. Instructions repeated every call are eligible for cached rates on many models. See the discounts page.
• Use a smaller model for easy questions and a larger one for hard ones.
• Limit conversation history sent each time.
• Cap output length.

Which mistakes are common?

• Sending the full chat history on every turn.
• Retrieving ten passages “to be safe”.
• Re-embedding the whole library after every small edit.
• Logging every prompt in full forever.

Decision checklist

• How many tokens go in per question?
• Are instructions cached?
• Is a smaller model enough for most questions?
• Is the vector store sized for the library?

Next steps

If volume is very high, running your own model may beat per-token pricing. See self-hosting an LLM versus paying per token. To compare platforms for the same model, read Azure OpenAI vs Bedrock vs Vertex AI.