A practical RAG evaluation setup you'll actually maintain

Most RAG evaluation setups I've inherited were built once, for a demo, by someone who has since moved on. They score well on a handful of hand-picked questions and tell you nothing about what happens when a real user asks something slightly outside that set. The goal of an eval system isn't to prove your RAG pipeline works — it's to catch the day it stops working.

I track two kinds of metrics, and I don't conflate them. Retrieval metrics — precision and recall against a labeled set of query-to-document pairs — tell you whether you're finding the right context at all. Generation metrics — groundedness and faithfulness, whether the answer is actually supported by what was retrieved — tell you whether the model is using that context honestly. A pipeline can fail at either layer, and the fix is completely different depending on which one broke.

The test set is the part people skip, and it's the part that matters most. I keep it small — 50 to 100 examples, not thousands — pulled from real user queries rather than invented ones, and I refresh it on a schedule rather than letting it calcify around whatever the product looked like on day one. A stale test set gives you false confidence, which is worse than no test set at all.

What makes this maintainable is a dashboard that runs the eval on every deploy and plots the trend, not just the latest number. Embedding model swaps, prompt tweaks, chunking changes — they all show up as a dip on a graph before they show up as a support ticket, which is the entire point.

The bar for success isn't a perfect score. It's catching regressions cheaply, before your users do, with a setup simple enough that you'll actually keep running it six months from now.