Probability Calibration:
A Hands-On, In-Browser Walkthrough

A model that discriminates well can still lie about how confident it is. This walkthrough builds a random-forest mortality model on ICU data, shows how to assess whether its probabilities behave like probabilities — via log-loss, the Brier score, and reliability diagrams — and then corrects them with four calibration methods: Platt scaling, isotonic regression, beta calibration, and spline calibration. We contrast a held-out calibration set against cross-validated calibration, and close with a synthetic-ground-truth study of what each method actually recovers.

Every code listing is live. Run a cell with its Run button or Ctrl/⌘+Enter; prerequisites run automatically. Charts and printed output appear inline.