Cost-Latency-Accuracy Tradeoff Analysis
Overview
You cannot maximize cost efficiency, latency, and accuracy at once. Product tolerance picks the point on the curve — a real-time assistant optimizes time-to-first-token; an overnight batch job optimizes correctness; a high-volume classifier optimizes cost per successful task.
When to Use
- Choosing between model tiers for a feature
- Designing multi-step pipelines (cheap router + strong writer)
- Finance flags AI spend
- Skip when compliance mandates a specific model regardless of cost
Quick Reference
| Product signal | Prioritize |
|---|
| Interactive chat | Latency (p95 TTFT) |
| User waits < 2s | Smaller / faster model if eval passes |
| Offline reports | Accuracy |
| Thin margin, high volume | Cost per successful request |
| Safety-critical output | Accuracy + validation, accept latency |
Implementation
- Write tolerances explicitly — e.g. p95 < 2s, < $0.02/request, > 85% eval pass rate.
- Sample real inputs — not synthetic happy path only.
- Benchmark 2–3 tiers per step on the same eval set.
- Pick the cheapest tier that clears the bar — not the flagship model by default.
- Split roles — fast model for classify/route, strong model for final generation.
- Re-benchmark quarterly — provider pricing and model tiers shift often.
On This Portfolio
Assistant streams for perceived speed; retrieval and validation run before tokens flow. Admin model settings let me swap copilot tiers without redeploying hardcoded strings — the curve is a config change, not a rewrite.
Common Mistakes
- Paying for Opus-class models on tasks a small model passes
- Optimizing accuracy while users abandon slow streams
- Measuring cost without tying it to successful task completion
Related Skills
- LLM selection by task
- Token cost estimation & budgeting
- Latency benchmarking across providers