AI Ticket Triage with Confidence-Gated Draft Replies

The expensive part of support isn't answering hard questions — it's the twenty minutes of reading, tagging, and routing that happens before anyone answers anything. That part is almost pure classification, and classification is exactly what LLMs are safe to automate first.

The workflow

How it works

1. Normalize the inbound: email, chat widget, and form submissions become one ticket shape with sender history attached.
2. Classify in one structured call: category, product area, urgency, sentiment, and language — each with a confidence score, output as strict JSON.
3. Gate by confidence per category: thresholds differ — billing disputes gate stricter than password resets, because misroutes cost differently.
4. High-confidence path: retrieve the top knowledge-base articles, draft a reply that cites them, and attach both draft and sources to the ticket.
5. Low-confidence path: route to the right team's queue anyway, with the classifier's best guess, alternatives considered, and suggested articles — the agent starts warm, not cold.
6. Capture the edit distance: when an agent rewrites a draft heavily, that ticket is flagged for review — either the KB is missing an article or the classifier missed the real intent.
7. Report weekly: misroute rate per category, draft acceptance rate, and the top 'no good KB article existed' clusters — which is your documentation backlog, generated automatically.

Classifier output contract

{
  "category": "billing_dispute",
  "confidence": 0.62,
  "alternatives": ["refund_request"],
  "urgency": "high",
  "product_area": "subscriptions",
  "language": "en",
  "route": "human_queue",
  "reason": "amount mentioned exceeds auto-draft threshold"
}

Stack

• Orchestration — n8n or Temporal-style workflow with retry and dead-letter branches
• Classification & drafting — fast LLM with structured output; separate prompts per stage
• Grounding — existing help-center articles indexed with the ingestion pipeline pattern
• Ticketing — Zendesk / Freshdesk / HelpScout via API, or plain shared inbox
• Feedback store — Postgres table of drafts, edits, and edit distances

The draft doesn't have to be perfect. It has to turn a twenty-minute ticket into a ninety-second review — at every ticket, every day.