Spark 2.0
Explore the evolution of Apache Spark's API in this keynote presentation from Scala Days New York 2016. Dive into the upcoming features of Spark 2.0, including more declarative APIs for automatic optimizations and improved links between Scala data types and binary data formats for efficient processing. Learn about Spark's journey as a large-scale Scala project, its functional API, and its impact on distributed programming. Discover the challenges faced in API design, data representation, and performance optimization. Gain insights into DataFrames, Datasets, and Structured Streaming APIs. Understand Project Tungsten's role in improving space efficiency and runtime code generation. Get a glimpse of Spark's long-term vision and versioning strategy, and find resources to further your Apache Spark knowledge.