Trustworthy ML Evaluation Lab (TMEL)

A local, reproducible framework for measuring classifier accuracy, probability calibration, controlled distribution-shift behavior, and confidence-based selective prediction. It tests hypotheses; it does not claim universal robustness.

Setup (macOS / Apple Silicon)

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'
tmel environment-check
pytest

PyTorch selects MPS when available (device: auto), otherwise CPU. Set device: cpu in a config for strict CPU execution.

Quick, auditable run

tmel run --config configs/quick.yaml --output-dir experiments
RESULT="$(ls -t experiments/results_*.json | head -n 1)"
tmel validate "$RESULT"
tmel report "$RESULT" --output-dir reports

Each completed JSON contains config, seeds, timestamp, platform/package versions, runtime, split protocol, model metrics, calibration temperature, shift results, reliability summaries, and selective-prediction points. The report is generated only from a validated result file.

Scope and data

Uses the Breast Cancer Wisconsin Diagnostic (WDBC) dataset distributed by scikit-learn. Its original source is the UCI ML Repository; consult the dataset provider for current provenance/licensing terms. The loader uses sklearn's bundled version for offline reproducibility. Stratified train/validation/test split is fixed per seed; scaling is fit on training data alone.

Synthetic Gaussian feature noise and alternating feature scaling are simulations, not naturally occurring shifts. They preserve labels only as a modeling assumption. The MLP's temperature scaler is fit on validation logits and labels only—never test data.

See docs/ for protocol and limitations. full.yaml is planned, not run by default.