A Case for Task Sampling Based Learning for Cluster Job Scheduling
Explore a novel approach to cluster job scheduling in this 16-minute conference talk from NSDI '22. Dive into the challenges of accurately estimating job runtime properties and learn about SLearn, a task sampling-based learning method that outperforms traditional history-based predictors. Discover how SLearn exploits similarities among tasks within the same job to achieve more accurate predictions, even in rapidly changing cluster environments. Examine analytical and experimental analyses of production traces, and understand how SLearn reduces average Job Completion Time (JCT) compared to prior-art methods. Gain insights into SLearn's implementation, design, and potential applications for DAG jobs, as well as its performance in simulation and testbed experiments on Azure using real-world cluster job traces.