Explore optimization techniques for complex Apache Spark plans in this 27-minute conference talk from DAIS NA 2021. Dive into Workday's experience building analytics products with Spark, addressing challenges like compiling large-scale DataFrames and handling extensive case statements. Learn about memory-efficient plan logging, common subexpression elimination for redundant subplan removal, and rewriting Spark's constraint propagation mechanism. Discover how these enhancements improve Catalyst performance on production pipelines, and gain valuable tips for managing complex Spark plans in your own projects. The talk covers topics such as data validation, handling large case expressions, optimized constraint propagation, and future improvements in Spark optimization.