Apache Spark on Kubernetes - A Technical Deep Dive
Dive into a technical deep dive conference talk exploring Apache Spark's integration with Kubernetes. Discover the evolution of Spark's scheduler backend, which now includes native Kubernetes support as of version 2.3.0. Explore the technical intricacies of this integration, including the Kubernetes submission client, scheduler backend, and how executors connect to drivers. Learn about dependency management, existing features like client mode and Kerberos support, and the Kubernetes Operator for Spark. Gain insights into the project's history, current capabilities, and future roadmap. Understand why running Spark on Kubernetes is beneficial and how it leverages Kubernetes for scheduling Spark drivers and executors. Get involved in this exciting development at the intersection of big data processing and cloud-native technologies.