Concept glossary

2026-10-20

Calibration (of an eval judge). A judge is calibrated when its confidence matches reality: of all traces it scores at 70% likely to fail, about 70% actually fail. Accuracy and calibration are different. A judge can rank traces well yet be badly overconfident, which breaks anything that uses its score as a probability, such as alert thresholds, routing to human review, or expected-cost estimates. You measure it with a reliability diagram or Brier score, and repair it with post-hoc methods like Platt scaling or isotonic regression fit on labeled data. For PMs the rule is simple: if a number is shown to users or used to make automatic decisions, someone must have checked its calibration.