---
name: cost-latency-accuracy-tradeoff
description: Use when a model or pipeline choice must balance cost, response speed, and output quality against each other.
---

# Cost-Latency-Accuracy Tradeoff Analysis

## Overview

You cannot maximize cost efficiency, latency, and accuracy at once. **Product tolerance** picks the point on the curve — a real-time assistant optimizes time-to-first-token; an overnight batch job optimizes correctness; a high-volume classifier optimizes cost per successful task.

## When to Use

- Choosing between model tiers for a feature
- Designing multi-step pipelines (cheap router + strong writer)
- Finance flags AI spend
- Skip when compliance mandates a specific model regardless of cost

## Quick Reference

| Product signal | Prioritize |
|---|---|
| Interactive chat | Latency (p95 TTFT) |
| User waits < 2s | Smaller / faster model if eval passes |
| Offline reports | Accuracy |
| Thin margin, high volume | Cost per successful request |
| Safety-critical output | Accuracy + validation, accept latency |

## Implementation

1. **Write tolerances explicitly** — e.g. p95 < 2s, < $0.02/request, > 85% eval pass rate.
2. **Sample real inputs** — not synthetic happy path only.
3. **Benchmark 2–3 tiers** per step on the same eval set.
4. **Pick the cheapest tier that clears the bar** — not the flagship model by default.
5. **Split roles** — fast model for classify/route, strong model for final generation.
6. **Re-benchmark quarterly** — provider pricing and model tiers shift often.

## On This Portfolio

Assistant streams for perceived speed; retrieval and validation run before tokens flow. Admin model settings let me swap copilot tiers without redeploying hardcoded strings — the curve is a config change, not a rewrite.

## Common Mistakes

- Paying for Opus-class models on tasks a small model passes
- Optimizing accuracy while users abandon slow streams
- Measuring cost without tying it to successful task completion

## Related Skills

- LLM selection by task
- Token cost estimation & budgeting
- Latency benchmarking across providers