Introduction to Spark Datasets
Explore Apache Spark's Dataset API in this 43-minute conference talk from Scala Days Copenhagen 2017. Dive into the basics of working with Spark Datasets, a hybrid approach that combines functional and relational programming concepts. Learn about Spark's components, including machine learning and streaming, and how they're being rewritten to support Dataset-compatible APIs. Discover the performance benefits and space efficiency of Spark SQL, and gain hands-on experience loading JSON data, applying schemas, and performing relational transformations. Understand how the optimizer works and how to mix functional and relational styles effectively. Examine windowed operations and window specifications, and grasp why Datasets are becoming increasingly important in the Spark ecosystem. No prior Spark knowledge is required, but a basic understanding of Scala is recommended.