Score a Classifier Without a Judge Model

A router sends each incoming ticket to billing, technical, or abuse. Someone asks whether it is good. The demo looks fluent. Fluency is not the question. The question is how often a billing ticket is sent to abuse, and how often an abuse ticket is missed. Those are counts, if you have labels.

scikit-learn’s model evaluation documentation defines precision and recall from true positives, false positives, and false negatives. You do not need that library to use the definitions. You need a set of items with labels your team accepts, a model that was not tuned on that set, and a table of predicted label against actual label. Accuracy alone hides a rare class. If abuse is one percent of tickets, a router that never predicts abuse is "accurate" and useless. Report the class you care about missing.

A small protocol

1. Write the label set. If two people would disagree, the label is not ready. Resolve the definition before you score the model.
2. Label a set the model and the prompt author do not tune on. Tuning on the test set makes the score a memory.
3. Run the classifier. Build the confusion matrix.
4. Choose the error you fear more. A missed abuse report and a misrouted billing ticket are not the same cost. The threshold follows that choice.
5. When the input is ambiguous, allow an "unsure" label that goes to a person. Forcing a class creates false confidence.
6. Re-run the same set when you change the prompt or the model. Keep the set stable so the comparison is real.

This is not the article on generative benchmarks. Building AI Evaluation Datasets Without Benchmark Leakage is about questions a generator might have seen, and about answers that are not a single label. If your output is a label, do not hire a second model to say whether the first model’s label "seems right." Count it. Use a judge only where you have no label and you are explicit that the judge is a proxy.

Illustrative scenario

This scenario is illustrative. Two hundred tickets are labeled by the support lead and kept out of prompt edits. The router’s abuse recall is what the lead reads first, because a missed abuse ticket is the costly error. Billing precision is second, because a wrong transfer wastes a team. The numbers belong to that set. They are not published here, and they would not transfer to another company’s tickets. The method does. When recall on abuse drops after a prompt change, the change does not ship. No one asks a chatbot whether the router "feels better."

Checklist

• Define labels so two reviewers would agree.
• Hold out the scoring set from tuning.
• Report precision and recall for the class you cannot afford to miss.
• Include an unsure path to a person.
• Recompute on every prompt or model change.
• Do not replace the matrix with a single accuracy percentage.

Questions that usually come next

How large does the labeled set have to be?

Large enough that the rare class appears more than a handful of times, and small enough that you will actually label it with care. A huge set of careless labels is worse than a smaller set with a written definition. There is no universal count that makes a score true.

What if we cannot label?

Then you do not have a classifier score. You have a demo. Spend the time on a definition and a sample, or keep the decision with a person. A judge model without labels compounds uncertainty. It does not remove it.