ML Monitoring ML: Scalable Monitoring of ML Models in Production Environments
Explore scalable monitoring techniques for machine learning models in production environments in this 49-minute conference talk by Ira Cohen, chief data scientist at Anodot. Discover how to detect and address issues affecting model performance, such as staleness, pipeline problems, and potential attacks. Learn about key performance measures for various ML model types and how to use anomaly detection algorithms to identify potential problems. Gain insights into an open-source monitoring agent for Python-based ML frameworks that automatically generates model performance measures, enabling data science teams to track and receive alerts on production issues requiring attention. Understand the importance of monitoring auxiliary measures with stable behavior over time to indicate potential model issues, using examples like churn prediction models and their input feature distributions.