“It’s Hard to Eval” Is a Product Smell

When developers find it difficult to create evaluations for their AI products, it is often a sign of a fundamental product design flaw. If a user cannot easily verify the output of an AI, the product is likely failing to provide the necessary context or evidence required for trust. Designing for verification should be a priority before building automated evaluation systems.

In practice, this means moving away from providing only a final answer and instead offering checkable artifacts. For an AI data agent, this involves showing the underlying SQL queries, metric definitions, and intermediate calculations. For a lesson plan generator, it means anchoring outputs to existing, vetted plans and highlighting specific edits. For complex tasks like medical reporting, the tool should function as a research assistant that surfaces evidence and contradictions, allowing the user to review facts before a final document is generated.

These design patterns improve the user experience by reducing the cognitive load required to validate work. By breaking down complex outputs into smaller, verifiable units, developers also make automated evaluation more tractable. Ultimately, the goal is to provide clear provenance for every AI-generated claim, using techniques like progressive disclosure to show details only when needed. This approach ensures that the product is not only easier to test but also more reliable and transparent for the end user.