How to Build an Evaluation System from Zero

What gets measured gets managed. — a maxim widely circulated in management literature; authorship disputed

On Monday, the system classifies a refund as "can be auto-processed." On Tuesday, an engineer swaps out one line of the prompt, and the same order gets routed to a human instead. The business owner asks: "did this get better, or worse?" If yesterday's sample wasn't kept, there's no way to tell — the team is left arguing on gut feeling.

"Did changing the prompt make quality regress?" — a system that can't answer this question doesn't belong in production.

That sounds severe, but take it apart and it's all common sense: without an eval set, every change is a gamble — the bet placed on an engineer's gut. Every customer complaint turns into a he-said-she-said — "it got worse," "no it didn't, you just got pickier" — and nobody can convince anybody. Evaluation isn't a testing step tacked on before launch. It's the floor a production system stands on.

This chapter covers how to build one from zero to production-ready, in four steps: first build a Golden Set, then a Regression Set, then automated judging, and finally wire it into the change process.

I. Crossing from 70% to 98%

Chapter 7's case again: on day one in the real environment, the score came in around 70%; the contract promised above 98%. What closes those 28 points?

Not by tweaking the prompt — that's the fine-tuning at the very end. It's a loop: evaluation surfaces a failure pattern, you fix it specifically, regression testing verifies it, and evaluation surfaces the next one. The insurance practitioner from before calls this the evaluation loop, and the quality bar is the same testable standard from Chapter 8 — right 995 times out of 1,000 in a week¹.

The same climbing curve has a fully documented public version too. A customer-service software company's first-generation product resolved only 23% of conversations on average; swapping the underlying model pushed that to 51%; deep customization for specific customers pushed the ceiling up to 86%. The vendor itself deployed the same product for its own customer service, tuned it continuously for two years after launch, and pushed the resolution rate to roughly 79%, resolving about 560,000 conversations a month.

What separates those three numbers isn't three model upgrades — the stretch from 51% to 86% was ground out one round at a time against an evaluation set. Another software company walked the same road, tuning its own customer-service agent for a full year before pushing the "couldn't answer" rate down from roughly 30% to under 10%.

Why can evaluation drive all of this? Because the quality of an AI system is continuous, probabilistic, and context-dependent — the same answer that dazzles in a demo can be a disaster in a specific business context, and the authority to define "good" sits with the business side, not the engineering side². Iterating without an evaluation metric is running blind: two weeks of tuning go by, and nobody can say whether quality went up or down. An evaluation system is exactly what turns the phrase "what counts as good" sitting inside a business expert's head into a scoring standard the system can run every single day.

II. Step One: The Golden Set — Standards Grown from Real Cases

A Golden Set is a batch of real cases carrying explicit judging criteria: what the input is, what the correct output is, why it counts as correct. It's the foundation of the whole evaluation system, and it only has two sources: real cases accumulated during the Minimum Viable Deployment (MVD) (things the system has already answered, that users have actually used) and disputed cases flagged by front-line users (answers that looked questionable, or drew a complaint). You can't use cases an engineer invented at their desk — a desk can't invent the mess of the field, and it can't invent the exact line a business expert cares about.

How many sample cases does a cold start need? Generally, fifty to a hundred usable ones are enough to get going. Chasing several hundred in one go is a common detour — more cases don't beat a precise standard.

More critical than the count is who does the judging. On the same output, a front-line user might call it a pass, an FDE might call it good, and the customer's business owner might call it a fail — all three standards are correct, because the three people are looking at different things: the front-line user cares whether it's easy to use, the FDE cares whether it's technically right, the business owner cares whether the company can afford it if it's wrong. The judge has to be pinned down at the moment the eval set gets built, and written into every sample: a Golden Set isn't a pile of test questions — it's a pile of questions a specific role has already judged against a standard. This is also where the "who judges" field in Chapter 8's acceptance sheet comes from — the acceptance standard starts taking shape the day the eval set gets built.

The best example of a cold start comes from a financial scenario: hundreds of implicit business rules at one institution — how a loan gets approved, which field anomaly needs to be escalated — lived only in the memory of a few veteran employees, never written down. The approach was to walk those employees through several rounds of real-case simulation: take one real piece of business, have them work through it while explaining why they judged it that way, record it as a rule item on the spot, and feed it to the model as an eval set. The first round is bound to miss things — fill the gaps, run it again, and a few cycles later it settles down. Once it's stable, the veteran employee spot-checks the system's judgments, and it only counts once they sign off³. This whole exercise lays the essence of an eval set bare: a Golden Set is a standard grown out of someone's head — taking a veteran's implicit judgment and making it explicit, one executable question at a time.

III. Step Two: The Regression Set — A Mistake Log That Only Grows

Once the Golden Set is standing, the evaluation system starts growing on its own: every real failure becomes a new sample added to the set.

The system gets a case wrong — it goes in. A user complains about an output — it goes in. An online audit spot-check turns up a problem — it goes in. The set grows the longer the system runs; it's really a mistake log — every entry is a real, costly error, and the point of recording it is to make sure it never happens again. The Regression Set only grows, never shrinks — the one exception being a standard change (say, the judging criteria get revised, and old samples get rejudged against the new standard instead of just getting thrown out).

Two disciplines guard against contamination. Demo data doesn't get in: the clean data from a demo environment would dull the eval set's ability to discriminate against the mess of production. Samples with labeling disagreement get archived separately: a case where two people can't agree isn't a bad sample — it's actually the most informative one. The disagreement itself is a signal that the standard isn't aligned; set it aside and revisit the standard on a regular cadence.

At this point the division of labor between the Golden Set and the Regression Set is clear: the Golden Set is the foundation (the standard gets defined here, small and precise), the Regression Set is the safety net (it keeps growing, large and messy). The first guarantees that "good" has a definition; the second guarantees that a past mistake never repeats.

There's an even more effective way to make the Regression Set "meaner": train a dedicated model on historical data — in a voice agent, for instance, train a small model on historical call recordings to specifically simulate the speech patterns a real deployment runs into, an impatient customer, a patient who slurs their words⁴. This kind of "adversarial test set" hugs the edge cases production will actually hit much more closely than a standard QA set does, and it's also a technique for evolving the Regression Set from "recording errors that already happened" to "actively manufacturing edge cases."

IV. Step Three: Automated Judging — Three Tiers, Plus a Human Calibration Pass

With samples in hand, you can't have a human rejudge every one of them after every change. Automated judging splits into three tiers, ascending by cost:

Tier one, exact match and rules. Where an answer can be compared exactly — classification, extraction, structured output — write a rule-based judge: near-zero cost, fully deterministic results. Whatever a rule can judge, never hand to a model.

Tier two, LLM-as-Judge. For open-ended tasks with no single correct answer — is a summary well written, is a reply appropriately toned — a model acts as the judge, scoring against a rubric written into its prompt. This is the mainstream approach in current engineering practice, but there's a discipline that's easy to overlook: the judge's rubric is itself a "prompt change." Change it, and the judging standard changes right along with it. So the judge's rubric also has to go under regression management — changing the judge and changing the system go through the same release gate.

Tier three, human spot-checking. Periodically pull a small batch of samples for a human to rejudge, and compare against the model judge's conclusions — not because automation can't be trusted, but because automation drifts: a judge model can develop a systematic preference for a certain writing style, or a systematic blind spot for a certain class of error, and only human spot-checking catches that kind of drift. A judge that's never recalibrated is just wrong, automatically, forever.

Cost awareness runs through all three tiers: not every sample needs the most expensive judge. Whatever a rule can judge goes to a rule; the LLM judge only handles open-ended tasks; humans only spot-check — the evaluation system's own running cost is still a cost (Chapter 14 puts it on the books). These mechanisms have already settled into standard components in open-source engineering practice — offline evaluation to guard against regression, online auditing to test against reality, and a cost-aware change gate⁵.

V. Step Four: The Release Gate — No Deployment Without Passing the Regression Set

Everything built up over the first three steps cashes out at this step: any prompt change, model swap, or tool modification must pass the Regression Set before it can be deployed.

This turns "I don't think it regressed" into "regression pass rate: 99.2%." Don't underestimate that shift in wording — the first is a feeling, and it can't enter any decision; the second is a number, and it can walk into an acceptance meeting, a change order, a customer's trust ledger. The line planted back in Chapter 7 pays off here: the one thing an MVD absolutely must build from day one is the eval set — because it's the release gate for every change, and change is the normal state of production.

The gate is also the interface between evaluation and Chapter 8's acceptance sheet: where do the launch-acceptance metrics come from? From the eval set's standard. The baseline is the historical level the eval set produced; the target is the commitment both sides agreed to on the evaluation standard — every number on the acceptance sheet has a reproducible evaluation process standing behind it. Any project where the two documents don't line up has one side making it up as they go.

VI. Online and Offline: The Loop Is the Evaluation Cycle

The gate handles the moment of change; a system already running needs a second set of eyes. Evaluation splits into two tracks:

Offline evaluation guards against regression — it runs against the Regression Set before a change ships, holding the line at "no worse than before." Online evaluation tests reality — production stays under continuous sampled audit, and user feedback (upvotes, corrections, complaints) flows back into the Regression Set. Put the two tracks together and you get the full version of the loop this chapter opened with: offline blocks a regressing change from shipping, online discovers a new failure pattern, the new failure becomes a new sample, and the sample makes the next round of offline evaluation stricter. Once it's turning, this is the evaluation loop that practitioner described earlier¹ (see Figure 12-1).

Figure 12-1: The eval set grows out of real-world cases, passes through the regression gate to production, and brings new failures back into the samples.

This methodology has a fully documented public counterpart worth using as a mirror for this chapter. A leading team's practice settled into three steps: the evaluation set grows out of real cases (never invented by engineers); the business side sits as judge (evaluation is shaped so business experts can actually take part — side-by-side output comparisons, simple better/worse labeling, regular review meetings; as the evaluation set gets more accurate, the business side watches the system get better on its own cases round after round, and trust accumulates from watching it happen with their own eyes); and evaluation gets wired into the production loop (launch isn't the finish line — continuous sampling, with an alert the moment scores dip). This book's four steps don't conflict with those three: the three steps describe direction (real, business-owned, closed-loop), while this chapter's four steps are the operating sequence for a cold start (Golden Set first, then Regression Set, then judging, then the release gate).

The most complete public example of evaluation-first is in agriculture. A farm equipment maker set out to solve herbicide waste: traditional sprayers blanket the whole field, while the new system uses 36 cameras and machine vision to spray only the weeds while moving. But what farmers wanted wasn't technology — it was advice they could trust. The AI team's engineers flew out to the farms and followed the agronomists into the field, first reviewing hundreds of real operating cases together with the experts to build a custom evaluation system, and only then iterating on the model — all of it racing against the planting season. In the end, chemical usage dropped by as much as 70%, and farmer engagement frequency went up sixfold². Those two numbers only exist because "good" had already been defined by an evaluation system.

VII. The Eval Set Is an Asset the FDE Can Take Along

One last piece of arithmetic, pulling the lens back from a single project to the whole team.

When a project ends, the code might get thrown away, the data has to go back to the customer, and the integration solution won't carry over to the next customer — so what actually accumulates? The eval set, and the process knowledge that grew out of building it. One practitioner put it clearly: data is the customer's moat, so it can never simultaneously be the vendor's moat too. What actually accumulates is the tacit knowledge and the evaluation loop that each deployment leaves behind¹.

On the next deployment, the model might be a whole generation newer, and the integration solution gets rebuilt from scratch — but the failure patterns, judging standards, and judge criteria accumulated at the last customer carry straight over as the starting point for the next project's Regression Set. The eval system is one of the rare engineering assets that doesn't lose value when the model gets upgraded — the more the model changes generations, the more you need a ruler that doesn't change with it. How this compounding value cashes out at the organizational level is the subject of Chapter 22 (turning field experience into product capability); here, the job is just to build the asset itself.

Three items on a negative checklist, to self-audit against: building the evaluation platform before the eval set — that's the order backwards; the platform exists to serve the eval set, and a platform with no eval set is just spinning its wheels. Running the eval set only once, right before launch — that's not an evaluation system, it's a launch ceremony. A judge that's never recalibrated — with nobody watching for automation drift, automation just becomes an amplifier for its own errors.

Evaluation establishes "provable results." But there's a moment in production that evaluation alone can't reach: the customer points at yesterday's result on the screen and asks "where did this number come from, why was it judged that way at the time" — what that calls for isn't a score. It's evidence.

---

Footnotes

1. "Inside an Applied AI Company" (Pace long-form post) ↩ ↩² ↩³

2. Fan Bing, Forward Deployed Engineer (FDE) (XDash open-source book) ↩ ↩²

3. "100 Questions About FDE" (open-source ebook) ↩

4. Tencent Research Institute, AI Lens Roundtable, Episode 6: "FDE Non-Consensus Views and a Field Guide from Silicon Valley Founders" ↩

5. enterprise_agent_platform (open-source enterprise agent platform engineering project) ↩