EvalKit
One CLI to benchmark any model — for accuracy and stability.
The problem
Model evaluation is usually ad-hoc and single-number: you get one accuracy figure and no sense of how much it wobbles from run to run.
What it does
EvalKit is an interactive CLI that evaluates five model families — image classification, object detection, semantic segmentation, speech recognition and text — under repeated random subsampling. For each checkpoint it reports classic metrics (accuracy, mAP, mean IoU, WER) and their stability (variance across 50% data slices), and writes rich reports: confusion matrices, ROC / PR curves, IoU / WER histograms, calibration and t-SNE plots, plus CSV summaries.
Stack
- Python
- PyTorch
- Ultralytics
- Transformers
- scikit-learn
- Matplotlib
- Rich
My role
- Solo developer
- Evaluator architecture
- Metrics & plotting
- Packaging (pip)
Highlights
Five tasks, one tool
classify / detect / seg / speech / text — a shared base class with one evaluator per task, behind a single guided CLI.
Stability, not just accuracy
Repeated random-subsampling reports the variance of each metric, so you see how reliable a score actually is.
Rich auto-reports
Every run drops confusion matrices, PR/ROC curves, histograms, calibration & t-SNE plots and a CSV summary.
Device-aware & pip-installable
Automatic GPU/CPU fallback; install with pip and run `evalkit`.