Computer vision
Open-source evaluation shellDetection Evaluation Shell
Production evaluation infrastructure for an object-detection model: per-class metrics, calibration, drift testing, and latency, on an X-ray contraband benchmark.
An open-source evaluation shell I built around a public X-ray contraband detector: the part that tells you whether a detection model is actually trustworthy, not just what its headline score is.
Problem
The problem.
A detection model's headline accuracy number is the least interesting thing about it. The questions that decide whether you can deploy it are different: where does it fail by class, is its confidence calibrated, how fast does it degrade under noise and occlusion, what is its latency under load. I built the evaluation layer that answers those, reproducing public reference models so the harness has something real to measure. To be precise about the boundary: I built the evaluation shell and reproduced public reference models. I do not claim production detection performance, and the benchmark's headline accuracy is not cited as a performance result.
Approach
How I built it.
- Reproduce public reference detection models (on the PIDray X-ray contraband benchmark and similar public datasets) to known-good baselines, so the evaluation harness is measuring something real.
- Score per segment, not just overall: per-class precision and recall plus confusion matrices stratified by image properties, so failure modes are visible instead of averaged away.
- Map operating points to cost: false-alarm and miss-rate curves across confidence thresholds, so a reviewer can choose a threshold against operational tradeoffs rather than a single number.
- Check calibration: reliability diagrams and temperature scaling, because a model that is confidently wrong is more dangerous than one that signals when it is unsure.
- Stress for drift: inject synthetic corruption (noise, blur, contrast, occlusion) and chart the degradation, alongside p50/p95/p99 latency and throughput under load.
- Emit schema-versioned results plus a model card and data card, with one-command reproduction from pinned environments.
Architecture
What's running, and why.
Reference models
- PyTorch detectors on public data
- YOLO-class reference models reproduced to known-good baselines on the public PIDray X-ray benchmark and similar datasets. Public reference models, not proprietary detectors.
Evaluation harness
- Per-segment metrics
- Per-class precision/recall, confusion matrices stratified by image properties.
- Operating-point + calibration
- False-alarm/miss-rate curves, reliability diagrams, temperature scaling.
- Drift + latency
- Synthetic-corruption degradation curves, latency percentiles, throughput at load.
Reproducibility
- Pinned + carded
- Pinned environments, single config, schema-versioned results.json, model and data cards, one-command reproduction.
Outcome
What shipped.
Public artifact
Open-source on GitHub. An engineering demonstration of evaluation infrastructure, not a benchmark claim.
Per class
Per-class metrics and confusion matrices across the benchmark's object classes
FAR / FRR
Operating-point curves that tie the confidence threshold to false-alarm and miss-rate tradeoffs
Drift + latency
Synthetic-corruption degradation curves, p50/p95/p99 latency, and throughput under load
One command
Reproducible from pinned environments, with model and data cards
Stack
- Python
- PyTorch
- YOLOv8 (Ultralytics)
- NumPy
- Matplotlib
- Jupyter
- CUDA / A100
What this proves
Where this ports.
Buying or building a detection model is the easy part. Knowing whether to trust it in the field is the work, and it is the same work whether the images are X-rays, medical scans, or industrial surfaces: per-class failure analysis, calibrated confidence, drift monitoring, honest latency. This is the evaluation discipline I bring to any model that has to make decisions that matter.
Want the same pattern in your business?
Thirty minutes. Describe what you're working on and I'll tell you if I can ship it.