An eval harness is the regression suite for AI — datasets, scorers, and reports I run on every meaningful change.
In my daily workflow
- I add a failing case to the harness before fixing a bug.
- Local runs for dev; CI for merge gates; nightly for deep sets.
- I track pass rate over time, not single green runs.
- I share harness output when reviewing prompt or retrieval PRs.
How it makes me work smarter
Harnesses encode institutional memory of what broke before. They make model upgrades boring — run suite, compare dashboard, ship or rollback. Without them, every provider update is a gamble.
My setup
- Fixture files per feature (assistant, copilot, classifier)
- Schema validators and rubric-based LLM judges
- CLI script wired into package.json
- CI job on pull requests touching AI routes
On this portfolio
Eval gates for the assistant target citation coverage and refusals; copilot evals check registry JSON shape. Job-fit feedback in admin grows the labeled set the harness uses over time.
