Written by: Karol Sobieraj, Founder & CEO, Digital Colliers
You shipped an AI feature three months ago. You tweaked the prompt last week. Did it get better or worse? Most teams genuinely do not know. They wait for a user complaint or an angry Slack message. By then the damage is done.
This is fixable. Two days of upfront work buys you the ability to catch regressions before they hit production. The work is building an evaluation set and wiring it into your deploy pipeline. Here is how to do it.
Why evaluation sets are not optional anymore
More than 80% of AI projects fail, roughly twice the failure rate of conventional IT projects. A large part of that gap is teams shipping changes blind. You cannot improve what you cannot measure. You cannot catch regressions if you have no baseline.
The pattern I keep seeing is teams that treat AI features like traditional code. They write unit tests for the plumbing. They check that the API call succeeds. But they do not test whether the answer is still good. So they merge a prompt change, redeploy, and discover two weeks later that the feature now hallucinates product names or misses edge cases it used to handle.
The operators who ship AI features that last build evaluation sets before they write the first line of production code. It is not perfectionism. It is survival.
What goes into a good evaluation set
An evaluation set is a curated list of inputs and expected outputs. For a production AI feature, you want 50 to 200 examples. Anything less and you will miss edge cases. Anything more and the set becomes expensive to maintain.
Start with real user inputs. Pull logs from the last month. Look for the queries that broke, the ones that returned garbage, and the ones that worked perfectly. You want diversity. Include short queries and long ones. Include typos. Include ambiguous requests that could go two ways.
For each input, write down what a good output looks like. This does not need to be a character-perfect string. Often it is a rubric. Does the response include the correct product? Does it avoid recommending out-of-stock items? Does it refuse gracefully when the user asks for something illegal?
Version the set in git alongside your code. Treat it like a schema migration. When you discover a new failure mode in production, add it to the set. The evaluation set grows with your understanding of what good looks like.
How to build it in two days
Day one is collection. Export your logs. If you have been live for a month, you probably have hundreds of real queries. Sample widely. Grab 30 that worked well, 10 that failed, and 10 edge cases your team has been nervous about.
Day two is annotation. For each input, write the rubric. This is the slow part. You need a human who understands the domain to label what good looks like. If your feature generates SQL, the rubric might be