Mandoline - Model Evaluation under Distribution Shift

Mandoline - Model Evaluation under Distribution Shift

Explore a comprehensive framework for evaluating machine learning models under distribution shift in this Stanford University lecture. Dive into the Mandoline approach, which leverages user-defined "slicing functions" to guide importance weighting and improve performance estimation on target distributions. Learn how this method outperforms standard baselines in NLP and vision tasks, and understand its connection to interactive ML systems. Gain insights into the theoretical foundations of the framework, including density ratio estimation and its error scaling. Discover the broader implications for model evaluation, hidden stratification, and iterative model development processes in the context of deploying ML models in real-world scenarios.