An LLM assigns every support ticket a team and an urgency; deterministic rules decide whether it can be routed automatically or must reach a person. Category accuracy 95–98%, urgent recall 13 of 13, and a confidence score rebuilt after the first one turned out to be meaningless.
1 / 2
A support ticket arrives. An LLM picks one of 6 categories and one of 4 priorities. Deterministic code — not the model — then decides whether the ticket routes to a team queue automatically or goes to a human first.
This is a prototype, not a production system. It runs on Groq’s free tier, uses Python standard library only, and the test set is synthetic. I am putting it here because the interesting work is in the evaluation and the guardrails, not the classification.
The model outputs a category. A routing table in code maps that category to a team. It never names a team, which means a re-org changes a lookup table rather than a prompt.
Everything else is layered around it:
| Layer | Control |
|---|---|
| Input | PII redaction — emails, phone numbers, card-shaped numbers, SSNs, IBANs, IPs become placeholders before any model call |
| Input | Ticket text wrapped in <ticket> tags the model is told never to obey; customer-supplied tags are stripped |
| Output | Anything outside the 6 categories or 4 priorities is invalid → human |
| Output | A reply Groq rejects as malformed JSON counts as invalid → human, rather than being dropped as an API error |
| Decision | 4 review rules (below) |
| Audit | Every run stores model, full prompt, review settings and a SHA-256 fingerprint of the dataset |
Only redacted text crosses the network boundary.
The first version asked the model for a single 0–100 confidence. It answered 95 on all 70 tickets — including every one it got wrong. A review rule built on that number would have caught nothing.
Groq does not expose token log-probabilities for these models, so the prompt now asks for a probability on every option, and confidence is the lower of the two chosen probabilities. That separated weak answers from strong ones:
| Confidence | Tickets | Fully correct |
|---|---|---|
| 50–69 | 13 | 38% |
| 70–84 | 27 | 85% |
| 85–100 | 28 | 82% |
A ticket routes automatically only if none of these fire:
Rule 3 exists because of a specific failure. The model rated a data-loss ticket and an API outage as “high” with confidence 85, giving urgent a probability of 0.00 and 0.02. Nothing the model reported could have flagged them. A keyword check caught both and over-flagged only 1 extra ticket.
That rule was written after seeing the misses, which means it only grows from real incidents. That is a limitation, not a feature.
Re-scoring one saved run at different thresholds, with all 4 rules on:
| Threshold | Routed automatically | Right when automatic | Errors caught | Urgent auto-routed |
|---|---|---|---|---|
| 70 | 63% | 83.7% | 10 of 17 | 0 |
| 75 (current) | 49% | 84.8% | 12 of 17 | 0 |
| 85 | 29% | 85.0% | 14 of 17 | 0 |
75 is the lowest threshold where nothing under-prioritized gets through. At 70, a medium ticket routed as low. The 5 errors that do route automatically at 75 all fail in the safe direction — 4 over-urgent, 1 wrong team at the right urgency.
Note what this table is: a product decision with the trade-off priced. Half the tickets handled automatically, or a third with fewer escapes.
| Run | Category | Priority | Both right | Urgent caught |
|---|---|---|---|---|
| qwen3.8-27b, with redaction + injection guard | 95.7% | 91.4% | 90.0% | 13 of 13 |
| gpt-oss-120b, per-option probabilities | 97.1% | 77.9% | 75.0% | 9 of 11 |
3 findings worth keeping:
A confidence number is worthless until you check it against accuracy. The first one looked fine on every dashboard and predicted nothing.
Some failures are invisible to the model. The urgent-terms rule is crude and it caught 2 serious misses that no probability the model reported would have. Model-independent checks earn their place.
Free-tier constraints shaped the design, usefully. 200,000 tokens a day is about 2 full runs, so threshold tuning re-scores saved runs instead of calling the API. That turned out to be the right architecture anyway — it made every policy change reproducible against a fixed set of model outputs.
Redaction catches structured identifiers only — names and free-text health details still reach the model, so the text is pseudonymised, not anonymous. The test set was written and labelled by the same person who wrote the prompt, and the prompt was tuned on it, so differences under about 5 points are noise. Tickets are English, single-issue and well written.
Before real tickets: Zero Data Retention, a data-processing review, 300+ real labelled tickets with two-person agreement on 100 of them, and a held-out set that is never used for tuning.
Keep looking
A generative ad suite that renders the same product across a billboard, a newspaper spread and a social post without the packaging quietly changing between them — solving visual brand drift with image conditioning rather than better prompts.
Turns a PDF, a set of notes or a plain prompt into a teachable slide deck. The interesting part is what happens when every model endpoint is down — it still produces a deck, and tells you it did.
A decision tool that refuses to just agree with you. Submit a dilemma and it returns a weighted comparison matrix, a SWOT, a 10-10-10 analysis and a verdict with a confidence score — then argues against its own recommendation.