~/rena
← blog
EvalsAI EngineeringTestingReliability

You Don't Have a Prompt Problem, You Have an Eval Problem

Every team stuck in a prompt-tweaking loop is stuck for the same reason: they can't measure whether a change helped. Here's how I build eval infrastructure, and why I now do it first.

RRena Thomas·

Here's a conversation I've had, with minor variations, on five different engagements:

"The agent keeps getting this wrong." "How often?" "Pretty often." "Did the last prompt change help?" "I think so? It seemed better."

That team doesn't have a prompt problem. They have a measurement problem, and every hour they spend tweaking prompts is a random walk.

I now build eval infrastructure before I build the agent. Not because I'm disciplined — because I've watched the alternative and it's slower.

Why prompt-tweaking feels productive and isn't

LLM behavior is high-variance. Run the same prompt on the same input ten times and you'll get outputs that differ in ways that matter. So when you change a prompt and the next test looks better, you have learned almost nothing: you've drawn one sample from a distribution you haven't characterized.

Worse, prompt changes are non-local. Adding "always confirm the appointment time before booking" fixes the case in front of you and quietly breaks three others where confirmation was already handled upstream and the agent now double-confirms and annoys the caller. Without a regression set you find out from a customer.

This is the whole argument. It isn't about rigor for its own sake. It's that in a high-variance, non-local system, unmeasured iteration doesn't converge.

Start with 30 cases, not 3,000

The reason teams skip evals is that "build an eval suite" sounds like a quarter of work. It isn't. The first useful version is an afternoon.

Go to your logs. Pull the 30 most recent real failures. That's your v1. Not synthetic cases, not edge cases you imagined — actual production failures, which are already weighted by real distribution.

interface EvalCase {
  id: string
  input: SessionInput
  state?: SessionState       // agents need starting state, not just a prompt
  assertions: Assertion[]
  tags: string[]             // 'scheduling' | 'escalation' | 'regression:PROD-412'
}

Two things I got wrong initially and would tell anyone to fix on day one.

Include state. For a conversational agent, "input" isn't a string. It's a conversation partway through, with facts established and tools already called. An eval harness that only tests turn one tests the easiest 10% of your system.

Tag against incidents. Every production bug becomes a tagged case. This is the part that compounds — six months in, your suite is a precise map of every way your system has actually failed, and no fix silently reintroduces an old bug.

Three kinds of assertion, in order of preference

1. Deterministic checks. Did it call scheduleAppointment? Was the date within the allowed window? Did it avoid mentioning a competitor? Is the JSON valid? These are cheap, fast, and never wrong. Push as much as you can into this bucket — far more fits than people expect.

2. Model-graded checks. For things you can't assert mechanically: was the response actually responsive? Was the tone appropriate? Did it avoid making a promise the company can't keep?

These work, with conditions. A grader needs a rubric specific enough that two humans reading it would agree, and it needs to output a structured verdict with a reason:

const GRADER_RUBRIC = `
You are grading a scheduling agent's response.

PASS if all hold:
- Acknowledges the caller's stated constraint
- Proposes a specific time or asks exactly one clarifying question
- Makes no commitment about pricing or insurance coverage

FAIL otherwise. Output: { verdict: "pass" | "fail", reason: string }
`

Then — and this is the step people skip — validate your grader against human labels. Hand-label 50 cases, run the grader, measure agreement. Below about 85% and your grader is noise you'll mistake for signal. I've seen a team optimize for three weeks against a grader that turned out to be measuring response length.

3. Human review. Expensive, so spend it where it's worth it: calibrating graders, and sampling the cases where automated checks disagree with each other.

Replay, not just simulation

For the multi-provider voice system I mentioned earlier, the highest-value piece of infrastructure we built wasn't the abstraction layer. It was the replay harness.

We recorded real production sessions — every model call, tool invocation, provider event, and state transition — and could replay them deterministically against a modified system. Change a prompt, replay 500 real sessions, diff the outcomes.

That's a fundamentally different quality of signal than synthetic evals. Synthetic cases test what you thought of. Replay tests what users actually did, including the 3% of sessions that are so strange you'd never have invented them.

Two hard-won implementation notes: record at the boundary — every external call in and out, with full payloads — because anything you don't record you can't replay. And expect drift; providers change behavior underneath you, so a replay that passed last month failing today is sometimes information about your vendor rather than your code.

Wire it into CI, or it will rot

An eval suite you run manually is an eval suite you run when you already suspect a problem — which is exactly when it's least useful.

What's worked:

  • Fast deterministic tier on every commit. Under two minutes, blocks the merge.
  • Full suite including model-graded checks nightly, with results posted to the team channel.
  • Replay against the last N production sessions before each deploy, comparing against the current production build rather than an absolute bar.

That last comparison matters. Absolute pass rates on LLM systems are noisy and demoralizing. "Did this change make things better or worse than what's live right now?" is the question you actually need answered, and it's robust to variance in a way absolute thresholds aren't.

What this buys you

Once measurement exists, everything downstream gets unlocked:

  • You can route to cheaper models because you can see what it costs in quality.
  • You can upgrade to a new model version in a day instead of a fortnight of vibes-based testing. This one has paid for itself repeatedly — model releases come fast now, and teams without evals are structurally unable to adopt them quickly.
  • Junior engineers can safely change prompts, because the suite catches what their intuition doesn't.
  • You can tell a stakeholder how good the system is with a number instead of an adjective.

Prompt engineering isn't the skill. Measuring prompt engineering is the skill. The prompts are downstream of that, and they're a lot easier to write once you can see what they do.

If you build one piece of AI infrastructure this quarter, build this one. Thirty cases and a CI job. It's less work than the week you're about to spend guessing.

← all posts