Ml Experimentation

• wandb.init(project="clinical-nlp", job_type="train", config=cfg).
• Log: loss, learning rate, per-entity-type F1, confusion matrix image, sample predictions table.
• Tag runs: baseline, ablation, production-candidate.
• Use W&B Sweeps for HPO; persist the sweep config in ml/training/sweeps/.

Eval harness (the single source of truth)

• ml/training/evaluation/eval.py takes --model-path + --golden-set and outputs a structured JSON + markdown report.
• CI runs it on every PR against a committed "production" model checkpoint pointer (in HF Hub).
• Regression gate: macro-F1 must not drop more than 0.5 points vs. the current production model.

Model registry

• HuggingFace Hub is the registry. One repo per model version (clinical-nlp/ner-v1, -v2).
• Promote via a tag: production. CI deploy reads the production tag.
• Never silently replace a model in serve. Always a new tag, new deploy, rollback-able.

Ablations

• Keep a running log in ml/ablations.md: what was changed, hypothesis, result, conclusion.
• Every production-candidate model has at least one ablation showing the change matters.

What's banned

• Jupyter notebook as the source of truth for a training run. Notebooks are for EDA; training is a script.
• Running training on laptop and claiming it works. Free-tier Colab / Kaggle / HF Spaces GPU acceptable.
• Unversioned data. Data gets a DVC hash or a Hub dataset revision — no exceptions.