Apache Celeborn: Making Spark and Flink Faster, More Stable and More Resilient

Apache Celeborn: Making Spark and Flink Faster, More Stable and More Resilient

Explore Apache Celeborn (Incubating), a high-performance universal Shuffle service for Spark and Flink engines, in this 28-minute conference talk by Zhou Keyong, Aliyun EMR Spark Engine manager and initial author of the project. Learn about Celeborn's core design principles, including high performance and high availability, as well as its unified architecture supporting multiple engines. Discover how this scalable service improves stability, enhances performance, and reduces costs for major companies, including Alibaba, where it handles Shuffle production at dozens of petabytes per day. Gain insights into user stories, future support for additional engines like Tez and MapReduce, and ways to engage with the Apache Celeborn community.