My AI judge disagreed with itself. What that taught me about measuring AI.
If you can't measure an AI feature honestly, you can't manage it. Four lessons from evaluating an assistant against data it had never seen.
Program managers are used to asking “how do we know it works?” For AI features the question is harder, because the output changes with every wording. While building a regulatory assistant, I treated quality like a release gate and learned four things.
1. Report the number you didn't tune on
I split my questions into a set for tuning and a set I never touched. Search found the right article for 84% of the tuning questions but 68% of the untouched ones. The 68% is the honest number. Reporting the 84% would have been optimism dressed up as data.
2. Read the failures, not just the score
My delivery-risk model looked reasonable until I printed what it had learned. Its strongest signals were jargon from the open-source projects it trained on, not delivery behaviour. Good scores can hide the wrong reasons, so the next version uses signals that transfer: stuck tickets, carry-over and dependencies.
3. Your measuring instrument needs measuring too
I used a second AI to grade whether answers stuck to their sources. It gave different grades to identical answers on different runs. An automated judge is useful for speed, but its score is directional until you check it against human grading. That check is now a go-live criterion.
4. Every improvement moves a trade-off
Letting the assistant give partial answers cut wrong refusals to zero and lifted correct citations to 100%. It also made answers longer, and the faithfulness grade fell. Neither number alone tells the story. The job is to show both to the decision-makers and agree which risk to accept.
The program-management lesson
AI quality is managed exactly like delivery risk: define what good looks like, measure it on data that wasn't used to tune it, make the weakest metric visible, and gate release on it. Teams that skip this ship demos. Teams that do it ship products.