Pinterest's ML Evolution: Distributed Training with Ray

Pinterest's ML Evolution: Distributed Training with Ray

Explore Pinterest's journey in optimizing machine learning recommender systems through distributed training with Ray in this 27-minute conference talk from Ray Summit 2024. Dive into the challenges and innovative solutions implemented by Pinterest's ML Platform team to scale data-intensive model training across thousands of jobs. Learn how they leveraged the Ray ecosystem to decompose and orchestrate massive workloads, addressing issues like memory pinning and multi-threaded collate. Discover the significant improvements in training throughput achieved through these optimizations and gain insights into Pinterest's ongoing efforts to enhance their ML infrastructure through internal abstractions and open-source contributions.