Evals are the unit tests of LLM features

Every team building with LLMs eventually has the same week. Someone tweaks a prompt to fix one complaint, a model version gets upgraded, and three other behaviors quietly get worse. Nobody notices until a customer does.

Traditional software catches this with tests. LLM features need the same thing, adapted to outputs that are probabilistic and rarely identical twice. That's what evals are: a repeatable way to answer "did this change make the feature better or worse?"

Start with real cases, not a benchmark

Public benchmarks tell you how a model does on someone else's problem. Your eval set should describe yours.

I start with 30 to 50 cases pulled from real usage (anonymized), chosen deliberately:

• Typical requests that represent most of the traffic.
• Known failures: every bug report about the feature becomes a case.
• Edge cases: ambiguous questions, missing information, requests the feature should refuse.
• Adversarial inputs: prompt injection attempts, content that tries to change the instructions.

Fifty well-chosen cases catch more regressions than five thousand generated ones. The set grows over time as production shows you new failure modes.

Score properties, not exact strings

LLM outputs rarely match a reference word for word, so exact comparison is mostly useless. Instead, each case checks properties that a correct answer must have. There are three kinds of checks, from most to least reliable:

1. Deterministic assertions. The output parses as valid JSON against a schema. It cites at least one retrieved source. It doesn't mention a competitor. It stays under 120 words. These are cheap, fast, and unambiguous, so use them wherever you can.
2. Reference comparisons. The extracted invoice total equals the expected value. The classification label matches. Structured extraction tasks often fit here.
3. Model-graded rubrics. Another model judges qualities like tone or helpfulness against a written rubric. These are useful but noisy, so calibrate them: score a sample by hand and check that the grader agrees with you before trusting it.

A harness can be very small

You don't need a platform to start. A typed list of cases and a loop is enough:

interface EvalCase {
  id: string
  input: string
  expect: {
    mustCite?: boolean
    contains?: string[]
    maxWords?: number
  }
}

interface Answer {
  text: string
  citations: string[]
}

export async function runEvals(
  cases: EvalCase[],
  answer: (input: string) => Promise<Answer>
) {
  const results = await Promise.all(
    cases.map(async ({ id, input, expect }) => {
      const output = await answer(input)
      const failures: string[] = []

      if (expect.mustCite && output.citations.length === 0) {
        failures.push("no citation")
      }

      for (const phrase of expect.contains ?? []) {
        if (!output.text.includes(phrase)) {
          failures.push(`missing "${phrase}"`)
        }
      }

      if (expect.maxWords && output.text.split(/\s+/).length > expect.maxWords) {
        failures.push("too long")
      }

      return { id, passed: failures.length === 0, failures }
    })
  )

  const passRate = results.filter((result) => result.passed).length / results.length

  return { passRate, results }
}

Run it in CI on every change to prompts, retrieval, or model configuration. Fail the build when the pass rate drops below the current baseline by more than a small margin, and print the failing cases so the regression is obvious.

Compare against a baseline, not a target

An absolute goal like "95% pass rate" invites arguments about whether the number means anything. A relative one is easier to act on: this change moved the pass rate from 88% to 84%, and here are the four cases that broke.

It also helps to break scores down by category. A change that improves average quality while breaking every refusal case is not an improvement.

Close the loop with production

Evals measure what you thought to test. Production shows you what you didn't. The most valuable habit is a steady pipeline from one to the other:

• Log inputs, outputs, retrieved context, and latency for every request.
• Watch signals like fallback rate, user thumbs-down, and escalations to a human.
• Review a sample of failures each week and add the interesting ones to the eval set.

Over a few months, the eval set becomes the most precise description of what the feature is supposed to do, which is exactly what a good test suite should be.

A short checklist

• Every bug report about the feature becomes an eval case.
• Deterministic checks first, model-graded checks only where needed and calibrated.
• Evals run in CI on prompt, retrieval, and model changes.
• Scores are compared against the previous baseline, per category.
• Production failures feed back into the set every week.

None of this is glamorous. It's the same discipline that made regular software reliable, applied to a component that happens to be a model.