Testing ML Models in Production - Detecting Data and Concept Drift

Testing ML Models in Production - Detecting Data and Concept Drift

Explore a comprehensive 55-minute conference talk on testing machine learning models in production. Learn about core statistical tests and metrics for detecting data and concept drift, preventing models from becoming stale and detrimental to business. Dive deep into implementing robust testing and monitoring frameworks using open-source tools like MLflow, SciPy, and statsmodels. Gain valuable insights from Databricks' customer experiences and discover key tenets for testing model and data validity in production. Walk through a generalizable demo utilizing MLflow to enhance reproducibility. Cover topics including the ML cycle, data monitoring, KS tests, categorical features, one-way chi-squared tests, monitoring tools, MLflow notebooks, ML workflows, data logging, model registry, feature checks, and model staging and migration.