Inference choices — streaming, batching, model size — directly shape UX and bill on a public assistant.
In my daily workflow
- I stream tokens for conversational UI; batch for offline evals.
- I pick the smallest model that passes eval gates for each step.
- I set max tokens and timeouts per route.
- I cache deterministic sub-results where safe.
How it makes me work smarter
Users perceive speed from time-to-first-token, not total generation time. Splitting router (fast) and writer (strong) nodes optimizes both cost and feel. Batch inference runs at night for embeddings; interactive paths stay lean.
My setup
- Streaming responses on assistant UI
- Role-based model resolution (fast vs strong)
- Token limits on input and output
- Edge vs Node runtime decisions per route
On this portfolio
The assistant streams replies while retrieval and validation complete upfront. Embeddings for knowledge content batch through server jobs after publish — different inference profile, same provider stack.
