Why RAG assistants fail in month three

Most retrieval-augmented assistants are judged on the demo. Someone asks ten questions the team already knows the answers to, the answers look good, and the project ships. Three months later the support team says the assistant has "got worse". It usually has not. The documents changed, the questions changed, and nobody was measuring either.

A RAG system is two systems chained together: a retriever that finds passages and a model that writes an answer from them. When quality drops, you need to know which half broke. That is only possible if you evaluate them separately, on a fixed set of questions, on a schedule.

Build the evaluation set from real questions

Start with 80 to 150 questions pulled from real usage: support tickets, search logs, sales emails, internal chat. Do not write them yourself, because you will write questions your documents answer well. For each question, a subject expert records the expected answer in one or two sentences and the document sections that support it.

Keep a deliberate share of hard cases. About one in five questions should be ones the assistant should refuse or escalate: out-of-scope topics, questions that need data it does not have, and questions where two policies conflict. A system that answers everything confidently is a liability, not a feature.

{
  "id": "refund-014",
  "question": "Can a customer get a refund after 45 days on an annual plan?",
  "expected": "Yes, pro-rated, if requested within 60 days of renewal. Needs manager approval.",
  "sources": ["billing-policy#annual-refunds", "support-playbook#approvals"],
  "should_refuse": false,
  "tags": ["billing", "policy-conflict"]
}

Three metrics, two layers

We track three numbers every week, and each one points at a different part of the system.

• Retrieval hit rate: for what share of questions did the top five passages include at least one expected source? If this drops, the problem is chunking, indexing or new content, not the model.
• Groundedness: does every claim in the answer appear in the retrieved passages? We score this with a second model call plus a human spot check of 20 answers. If this drops while hit rate holds, the prompt or the model changed.
• Resolution accuracy: would the expert accept this answer as-is? This is the number the business cares about, and it is the only one that needs human judgement every cycle.

Add one guardrail metric: the refusal rate on the questions that should be refused. If it falls, the assistant is guessing.

Why month three is the cliff

Three things tend to land at about the same time. First, the source documents drift: pricing pages are rewritten, a policy is replaced, and the old PDF is still in the index, so the retriever returns both. Second, users learn what the assistant is good at and start asking harder, more specific questions. Third, someone upgrades the model or edits the system prompt to fix one complaint and quietly breaks five other behaviours.

An assistant does not decay on its own. Its inputs change while its tests stay frozen at launch.

The weekly review that keeps it honest

The routine is small enough that teams actually keep doing it. Every week the evaluation set runs automatically against the live configuration, and the three metrics are compared with the previous run. Anything that moves more than three points gets a ten-minute look. Every week, one person reads 20 real conversations, with no sampling for flattering ones, and adds any failure to the evaluation set as a new case. Whenever content, prompt or model changes, the full set runs before release, and a regression blocks the deploy.

After six months the set typically has 250 or more cases and reads like a history of everything that ever went wrong. That history is the real asset. It is what lets you swap in a cheaper model or a better retriever and know within an hour whether you made things better.

Start before you launch

If you take one thing from this, build the evaluation set during development, not after the first complaint. It doubles as your acceptance test, your regression suite and the basis for any conversation about whether the assistant is good enough to go live. An assistant with no numbers is a demo, and demos do not survive month three.