PySpark in Apache Spark 3.3 and Beyond
Explore the latest advancements in PySpark introduced with Apache Spark 3.3 and get a glimpse of future developments in this 35-minute Databricks conference talk. Dive into the evolution of PySpark since Project Zen's inception in Apache Spark 3.0, including improved error messages, type hints for autocompletion, and visualization implementations. Learn about the popular Pandas API on Spark, introduced in Apache Spark 3.2, which allows running pandas API on Apache Spark. Discover the new features in Apache Spark 3.3, such as expanded API coverage, faster default indexing in Pandas API on Spark, datetime.timedelta support, new PyArrow batch interface, enhanced autocompletion, Python & Pandas UDF profiler, and new error classification. Gain insights into the current efforts and roadmap for PySpark beyond Apache Spark 3.3, covering aspects of functionality, productivity, usability, performance, and feature parity.