Explore a methodology for measuring probabilistic calibration and dynamically updating scores in anomaly detection systems. Delve into the challenges faced by Microsoft Defender Advanced Threat Protection in monitoring billions of anomaly detectors across diverse data sources. Learn about adaptive mixtures of functional inflated beta-binomial models and their application in identifying and updating scores for cyber indicators of attack. Discover how an automated statistical diagnostics system can infer optimal weighting of anomaly scores through probabilistic inference, addressing the scalability issues of manual detector review. Cover topics such as p-value calibration, model uncertainty, meta-learning, Bayesian approaches, state-space models, filtering, and probability distribution modeling in this comprehensive 1-hour 26-minute talk by Cole Sodja from Microsoft Defender ATP, presented at the Alan Turing Institute.