Questions about AI-first engineering, MCP servers, RAG, evals, orchestration, and how I ship production AI workflows.
Hybrid index over help articles, resolved tickets, and internal runbooks with structured fields for product area and version. The copilot must cite doc sections and ticket IDs so agents can verify before sending.
Golden questions from real users, tool-call assertions, JSON schema validation, and retrieval hit rates. I run them in CI on prompt or tool changes and block merges on regressions beyond an agreed threshold.
Twenty honest questions beat two hundred synthetic ones. I seed from support logs, sales calls, and the FAQ I am writing anyway, then grow the set every time production surprises us.
LLM-as-judge with rubrics, human spot checks, and structural checks — required fields present, citations included, banned phrases absent. Perfect text match is the wrong bar for conversational outputs.
Human review rate, deflection where applicable, p95 latency per step, retrieval miss rate, and cost per successful task. Sentiment alone is noise without operational counterparts.
Downstream code stops parsing prose. Categories, confidence scores, and citation arrays are validated before any email sends or ticket updates. Failures become schema errors you can alert on.
Structured logs per step, trace IDs across tools, payload redaction, and replay fixtures. I want to reconstruct why an agent chose a tool three Tuesdays ago without guessing from a chat transcript.
Redact at ingestion, hash identifiers where correlation is enough, and separate full-fidelity debug captures behind break-glass access. Agents touch sensitive data — logs should not leak it by default.