Zeus - Uber's Highly Scalable Distributed Shuffle as a Service
Dive into the architecture and deployment of Zeus, Uber's highly scalable and distributed shuffle as a service powering all data processing at the company. Explore how this ground-up solution supports hundreds of thousands of jobs and millions of containers, shuffling petabytes of data. Learn about Zeus' paradigm-shifting approach to external shuffle, resulting in improved performance for most jobs despite remote data writing. Compare Zeus' performance with different storage systems backed by external shuffle, such as NFS and HDFS. Discover the integration of Zeus with Spark and how it contrasts with Spark's built-in sort-based shuffle mechanism. Gain insights into the future roadmap and plans for Zeus, and understand its impact on addressing hardware failures, reliability, and scalability challenges in one of the largest Spark and Hive clusters in the industry.