[{"data":1,"prerenderedAt":337},["ShallowReactive",2],{"blog-\u002Fblog\u002Fai-evaluation-and-regression-testing":3,"blog-surround-\u002Fblog\u002Fai-evaluation-and-regression-testing":320,"blog-related-\u002Fblog\u002Fai-evaluation-and-regression-testing":328},{"id":4,"title":5,"audience":6,"body":10,"cluster":281,"contentRole":282,"conversionGoal":282,"cornerstone":283,"date":284,"description":285,"draft":283,"extension":286,"factCheckedAt":282,"faq":287,"featured":283,"language":282,"meta":297,"navigation":298,"order":299,"originalAsset":282,"path":300,"pillar":301,"primaryKeyword":302,"publicationOrder":303,"relatedProject":282,"releaseScope":282,"reviewCycle":304,"reviewMethod":305,"reviewStatus":306,"reviewedAt":284,"reviewedBy":307,"searchIntent":308,"seo":309,"sources":282,"stem":310,"tags":311,"type":318,"updated":282,"__hash__":319},"blog\u002Fblog\u002Fai-evaluation-and-regression-testing.md","AI Evaluation & Regression Testing",[7,8,9],"solo-founder","ai-product-builder","ai-engineer",{"type":11,"value":12,"toc":271},"minimark",[13,32,37,44,47,51,54,116,123,129,133,139,155,167,171,178,181,214,225,229,237,241,244,247],[14,15,16,20,21,25,26,31],"p",{},[17,18,19],"strong",{},"\"It feels better\" is how AI features quietly get worse."," You tweak a prompt, try two or three examples, they look good, you ship — and you've just changed the behaviour for every input you ",[22,23,24],"em",{},"didn't"," try, some of which you made worse. This is the core reliability problem of building with models: the output space is huge, and eyeballing a handful of cases tells you almost nothing about the rest. Evals are the answer, and they're the practice that most separates an AI product you can improve confidently from one you're scared to touch. ",[27,28,30],"a",{"href":29},"\u002Fblog\u002Fprompt-and-model-versioning","Prompt and model versioning"," gives you the safe way to change things; evals are how you know a change was actually an improvement. This is how I build them.",[33,34,36],"h2",{"id":35},"evals-are-a-test-suite-for-a-fuzzy-function","Evals are a test suite for a fuzzy function",[14,38,39,40,43],{},"The reframe that makes evals click: an AI feature is a function with a huge, fuzzy output space, and an eval set is its ",[17,41,42],{},"test suite",". Same idea as code tests — known inputs, expected results, run on every change — adapted to the fact that \"expected\" is sometimes a rule and sometimes a judgement.",[14,45,46],{},"Once you hold that framing, the whole practice follows: build a representative set, define how to score it, run it on every change, and gate on the result. The payoff is the same as tests give code — the confidence to change things fast without breaking what worked.",[33,48,50],{"id":49},"building-the-set-cover-three-kinds-of-case","Building the set: cover three kinds of case",[14,52,53],{},"An eval set is only as good as its coverage. Three kinds of case earn their place:",[55,56,57,73],"table",{},[58,59,60],"thead",{},[61,62,63,67,70],"tr",{},[64,65,66],"th",{},"Case type",[64,68,69],{},"What it is",[64,71,72],{},"Why it matters",[74,75,76,90,103],"tbody",{},[61,77,78,84,87],{},[79,80,81],"td",{},[17,82,83],{},"Normal",[79,85,86],{},"The typical, happy-path inputs",[79,88,89],{},"The bulk of real usage — must stay good",[61,91,92,97,100],{},[79,93,94],{},[17,95,96],{},"Edge",[79,98,99],{},"Boundary, unusual-but-valid, tricky inputs",[79,101,102],{},"Where quality quietly degrades first",[61,104,105,110,113],{},[79,106,107],{},[17,108,109],{},"Regression",[79,111,112],{},"The exact inputs that have failed before",[79,114,115],{},"So a fixed bug can never silently come back",[14,117,118,119,122],{},"The regression cases are the ones people skip and the ones that pay off most: ",[17,120,121],{},"every production failure becomes a new eval case."," A bad output a user reports isn't just fixed — it's added to the set, so any future change that would reintroduce it fails the eval. Over time the set becomes a memory of every way the feature has ever been wrong, and a guarantee it won't be that way again silently.",[124,125,126],"pull-quote",{},[14,127,128],{},"Turn every production failure into an eval case. Do that consistently and your eval set stops being a test suite and becomes institutional memory — a growing proof that the feature can't regress in any way it's failed before.",[33,130,132],{"id":131},"scoring-rule-based-where-you-can-judgement-where-you-must","Scoring: rule-based where you can, judgement where you must",[14,134,135,136],{},"Scoring is where evals live or die, and the rule is simple: ",[17,137,138],{},"make as much rule-checkable as the task allows.",[140,141,142,149],"ul",{},[143,144,145,148],"li",{},[17,146,147],{},"Exact \u002F rule-based"," — for structured or right-answer outputs (a field, a classification, valid JSON against a schema). Cheap, deterministic, unarguable. This is the scoring you can fully trust.",[143,150,151,154],{},[17,152,153],{},"Criteria \u002F rubric"," — for open-ended outputs, a defined rubric of what \"good\" means, applied by a human on a sample or by a model-as-judge (another model scoring against your rubric). More coverage, but more subjective — treat its numbers as directional, not absolute.",[14,156,157,158,161,162,166],{},"The design move that makes evals more reliable is ",[22,159,160],{},"structuring the output so more of it is rule-checkable"," — the same instinct as an ",[27,163,165],{"href":164},"\u002Fblog\u002Fai-output-contracts-and-failure-handling","output contract",". If part of the response is a structured field, you can score that part exactly and reserve the fuzzy scoring for the genuinely open part. Deterministic scoring is the bedrock; build as much of your eval on it as the task permits.",[33,168,170],{"id":169},"the-regression-gate-run-on-every-change","The regression gate: run on every change",[14,172,173,174,177],{},"An eval set that isn't run automatically is a good intention. The discipline is a ",[17,175,176],{},"gate",": every change — prompt, settings, model version — runs the set against a known-good baseline, and a change that drops the score on cases that used to pass is a regression to investigate, not a release to ship.",[14,179,180],{},"This is exactly a code regression suite, applied to the model layer:",[182,183,184,190,196,202,208],"ol",{},[143,185,186,189],{},[17,187,188],{},"Baseline."," The current eval score is the bar.",[143,191,192,195],{},[17,193,194],{},"Change."," Edit the prompt, or adopt a new model version.",[143,197,198,201],{},[17,199,200],{},"Run."," Score the whole set.",[143,203,204,207],{},[17,205,206],{},"Compare."," Improved, held, or regressed?",[143,209,210,213],{},[17,211,212],{},"Gate."," A regression doesn't ship until you understand it.",[14,215,216,217,220,221,224],{},"Step 5 is the point. Without the gate, a \"better\" prompt that broke three edge cases reaches users before you notice — and they become your regression suite, reporting failures as churn. This is also what makes adopting a new model version ",[22,218,219],{},"safe",": you don't trust \"it's newer, it's better,\" you run your evals and see whether it's better on ",[22,222,223],{},"your"," tasks.",[33,226,228],{"id":227},"keep-it-proportionate-for-a-solo-product","Keep it proportionate for a solo product",[14,230,231,232,236],{},"None of this needs a platform. For a solo build, an eval set is a folder of cases and a script that scores them — start with ten cases that matter and grow it every time something breaks. This is the same ",[27,233,235],{"href":234},"\u002Fblog\u002Fhow-i-use-analytics-as-a-solo-builder","analytics-as-a-solo-builder"," instinct: measure the few things that change a decision. The mistake isn't a small eval set; it's having none and shipping prompt changes on vibes. Ten real cases scored on every change beats a hundred you never run.",[33,238,240],{"id":239},"what-i-would-decide","What I would decide",[14,242,243],{},"Build the eval set from the start, small, and grow it from real failures — every bug a user finds becomes a permanent case. Score as much as possible with rules, and structure outputs so you can; reserve subjective scoring for the genuinely open parts and treat its numbers as directional. Gate every change on the set, and never let a regression ship unexamined — that gate is the whole value, because it's what lets you change the feature fast without your users discovering the breakage first. Evals turn \"I think this is better\" into \"this scored better,\" and that single shift is what makes an AI product something you can improve with confidence instead of fear.",[245,246],"hr",{},[14,248,249],{},[22,250,251,252,256,257,260,261,265,266,270],{},"Part of the ",[27,253,255],{"href":254},"\u002Ftopics\u002Fbuilding-ai-products","Building AI Products guide",". See also ",[27,258,259],{"href":29},"prompt & model versioning"," and ",[27,262,264],{"href":263},"\u002Fblog\u002Fproduction-ai-observability","production AI observability",". The ",[27,267,269],{"href":268},"\u002Fnewsletter","newsletter"," sends one finance-systems pattern, product decision or build lesson every two weeks.",{"title":272,"searchDepth":273,"depth":273,"links":274},"",2,[275,276,277,278,279,280],{"id":35,"depth":273,"text":36},{"id":49,"depth":273,"text":50},{"id":131,"depth":273,"text":132},{"id":169,"depth":273,"text":170},{"id":227,"depth":273,"text":228},{"id":239,"depth":273,"text":240},"engineering",null,false,"2026-07-28","How I build eval datasets for AI features — the cases, the scoring, and the regression gate — so a prompt or model change is judged on evidence, not on trying three examples and hoping.","md",[288,291,294],{"question":289,"answer":290},"What is an AI eval dataset?","An AI eval dataset is a curated set of representative inputs, each paired with an expected output or a scoring rule, that you run a prompt or model against to measure quality objectively — the AI equivalent of a test suite. Instead of eyeballing a few examples after a change, you run the whole set and get a score, so 'it feels better' becomes 'it scored higher.' A good eval set covers the normal cases, the edge cases, and the specific failures that have burned you before — every production bug becomes a new eval case, so it can never silently recur. It's the single practice that most separates an AI product you can change confidently from one you're afraid to touch.",{"question":292,"answer":293},"How do you score AI outputs in an eval?","By the strictest method the task allows. Exact-match or rule-based scoring is best where the output is structured or has a right answer (a field value, a classification, valid JSON) — it's cheap, deterministic and unarguable. For open-ended outputs, you fall back to criteria-based scoring: a rubric of what 'good' means, applied either by a human on a sample or by a model-as-judge (another model scoring against your rubric), each with trade-offs. The rule of thumb is to make as much of the output rule-checkable as you can — structure it so parts can be scored exactly — and reserve subjective scoring for the genuinely open parts, because deterministic scoring is the part of an eval you can actually trust without second-guessing.",{"question":295,"answer":296},"What is regression testing for AI?","Regression testing for AI is running your eval set on every change — a prompt edit, a settings change, a new model version — and comparing the score to the previous baseline, so a change that quietly breaks something is caught before it ships. It works exactly like code regression tests: you have a known-good baseline, and a change that drops the score on cases that used to pass is a regression to investigate, not a release to push. The gate is the discipline: a change that regresses the eval doesn't ship until you understand why. Without it, your users are your regression suite, and a 'better' prompt that broke three edge cases reaches them before you notice.",{},true,9.3,"\u002Fblog\u002Fai-evaluation-and-regression-testing","building-ai-products","AI evaluation regression testing",180,"annual","editorial","reviewed","Tan Gravam","informational",{"title":5,"description":285},"blog\u002Fai-evaluation-and-regression-testing",[312,313,314,315,316,317],"ai-products","evals","evaluation","regression-testing","quality","solo-saas","text","tgyWpUdCXWxSPBDASkob47uT6EpEDZa4nk0eQI2csiQ",[321,325],{"title":322,"path":323,"stem":324,"type":318,"language":282,"draft":283,"children":-1},"Agile vs Waterfall for Finance-Systems Delivery","\u002Fblog\u002Fagile-vs-waterfall-for-finance-systems-delivery","blog\u002Fagile-vs-waterfall-for-finance-systems-delivery",{"title":326,"path":164,"stem":327,"type":318,"language":282,"draft":283,"children":-1},"AI Output Contracts & Failure Handling","blog\u002Fai-output-contracts-and-failure-handling",[329,332,335],{"path":29,"title":330,"description":331},"Prompt & Model Versioning: Production Governance for AI Products","A prompt is production code and a model is a dependency you don't control. How I version prompts, handle model changes, run eval datasets, and keep outputs from silently regressing.",{"path":263,"title":333,"description":334},"Production AI Observability: Watching What the Model Does","You can't fix what you can't see. How I think about observing AI features in production — quality, cost, latency, drift and failures — so problems show up as data, not churn.",{"path":164,"title":326,"description":336},"A model will eventually return garbage, the wrong shape, or a confident lie. How I design output contracts, validation, fallbacks and graceful degradation so the product fails safely.",1785267479573]