Beyond Shuffling - Scaling Apache Spark

YouTube·Scala Days

Explore advanced techniques for scaling Apache Spark in this 43-minute conference talk from Scala Days Berlin 2016. Delve into best practices and code snippets for handling large datasets efficiently. Learn to leverage Spark counters for performance investigation, optimize key-value data operations, and replace groupByKey with memory-efficient alternatives. Discover effective caching and checkpointing strategies to reduce execution time. Gain insights on functional transformations using Spark Datasets, working in noisy cluster environments, and utilizing Spark SQL for improved performance. Master the art of validating Spark jobs with accumulators and explore additional testing resources to enhance your Spark development skills.