Data Quality Approach for Netflix Personalization Systems

Data Quality Approach for Netflix Personalization Systems

Explore an approach to data quality for Netflix personalization systems in this 28-minute conference talk from Databricks. Learn about the challenges of maintaining data quality for machine learning models trained on hundreds of terabytes of data daily. Discover infrastructure and methods used to ensure high data quality, including 'Swimlanes' for defining data boundaries, pipelines for aggregating metrics, visualization tools for observing changes over time, and automated audits for detecting data regressions. Gain insights into optimizing metric computations, using SQL queries for defining metrics, and leveraging Spark for ETL jobs powering visualization and audit tools.