Eval gates are the merge blockers for AI features — no ship until structured outputs, retrieval quality, or tone regressions are caught.
In my daily workflow
- I define 10–20 golden cases per feature before expanding scope.
- CI or pre-deploy scripts run evals against staging models and prompts.
- Failures produce a diff: expected vs actual, with trace IDs for debugging.
- Prompt or retrieval changes require eval green before merge.
How it makes me work smarter
Demos lie; evals don't. Gates force me to articulate what 'good' means for the copilot, job-fit classifier, or assistant citations. When a model provider updates weights, I get signal in minutes instead of from user complaints.
My setup
- Golden datasets in repo (JSON or markdown fixtures)
- Scoring: exact match, JSON schema validation, LLM-as-judge with rubric
- Separate smoke evals (fast) and nightly deep evals
- PostHog events tagged when production responses fail guardrails
On this portfolio
The public assistant and copilot graph include structured output schemas; eval harnesses target citation presence, refusal on out-of-scope JDs, and stable tool routing. Admin job-fit analytics surfaces classifier feedback loops back into eval sets.
