Learning path · 5 steps
Build a small, repeatable ticket-routing evaluation
Finish with queue definitions, a labeled test set, saved model responses and a local evaluator. The six supplied tickets make the mechanics manageable, while separate error counts show why a wrong destination and unnecessary triage should not be collapsed into one impressive-looking score.
In this guide 6 sections
These walkthroughs use the downloadable synthetic support-service project. Work in a disposable copy so you can compare the proposed changes with the original defects. Each tool needs separate authorized access; the starter includes no production credentials or incident data.
What you will build
A routing trial you can rerun after a prompt change
Keep the request definition, raw responses and predictions.json together. The local script checks structure and separates exact matches, wrong specialists, unnecessary triage and missed triage; it does not send tickets anywhere.
Included example files
A small support-service project
Synthetic code, logs, a deliberately stale runbook, and six support tickets. The JavaScript uses Node.js 22+ with no packages or network calls. Product accounts and model access are separate.
- routing/tickets.json
- routing/request.json
- routing/predictions.example.json
- routing/evaluate.mjs
Set the queues before seeing predictions
Read the four criteria in request.json and the authored labels in tickets.json. Access, application and network cover distinct evidence; triage catches missing detail or no fitting specialist. Decide whether those definitions match the workflow you want to evaluate.
Check before continuingTicket A is labeled access and D triage before a model is called. The labels are not edited to match a later answer.
Verify the local checker with known data
Run the supplied manual example. It intentionally sends A to triage and D to access. This proves how the local counts distinguish those two mistakes, without pretending that the file contains model output.
Terminal commandnode routing/evaluate.mjs routing/predictions.example.jsonCheck before continuingThe result is total 6, exact 4, unnecessaryTriage 1, missedTriage 1 and wrongSpecialist 0.
Collect actual Choice responses
Use the Jev walkthrough’s copyable request body or matching Playground fields. Supply each ticket as state. Save the raw responses, then record the returned queue and actual model ID in predictions.json using the example file’s structure.
Check before continuingEach A–F ticket has one recorded response; manual-demo is not left as the model name for an actual run.
Compare decisions, then refine the definitions
Run the checker against your predictions. Read a disagreement against the original ticket and criteria. If the definitions overlap, change the request and save a second run separately. Do not silently change the expected labels to improve the count.
Terminal commandnode routing/evaluate.mjs routing/predictions.jsonCheck before continuingEach changed prediction can be traced to a saved request, response and model version.
Design the next evaluation before routing real work
Write evaluation-notes.md describing observed disagreements, remaining ambiguities and the kinds of unseen reports to add next. Compare the effort with a simple rules-based baseline. Leave real ticket execution disconnected until a larger, representative evaluation supports your policy.
Check before continuingThe folder contains request.json, labels, raw responses, predictions.json and evaluation-notes.md. No deployment-accuracy claim is drawn from six examples.
Check the finished project
node routing/evaluate.mjs routing/predictions.jsonThe command should accept one valid result per ticket. Review disagreements individually and preserve the returned model IDs and raw probability distributions. The six-ticket set demonstrates a process, not production accuracy or a safe automation threshold.
Tool guides for this path
An authored walkthrough using the synthetic starter project and linked official tool documentation. Baseline fixture behavior and the local evaluator were checked. The assistants, review service and model calls were not run for this path.
Browse all learning paths