Explore a 21-minute conference talk on scaling machine learning feature engineering in Apache Spark at Facebook. Dive into the implementation of Feature Injection and Feature Reaping techniques, including Spark core/SQL enhancements, indexed/aligned tables, and the new ORC FlatMap encoding. Learn about catalyst optimizations, new ORC physical encodings for feature maps, and the process of writing/committing indexed feature tables. Gain insights into Facebook's approach to improving prediction model quality through efficient data management and processing techniques in Spark.
Scaling Machine Learning Feature Engineering in Apache Spark at Facebook