Check-N-Run - A Checkpointing System for Training Deep Learning Recommendation Models
Explore a cutting-edge checkpointing system for training large-scale deep learning recommendation models in this NSDI '22 conference talk. Dive into the challenges of checkpointing massive ML models and discover how Check-N-Run addresses size and bandwidth issues. Learn about differential checkpointing techniques that track and save only modified parts of the model, particularly effective for recommendation models with embedding tables. Examine quantization strategies that significantly reduce checkpoint size without compromising training accuracy. Understand how these innovations lead to substantial reductions in required write bandwidth and storage capacity, improving checkpoint capabilities while lowering total ownership costs. Gain insights into the architecture of recommendation models, high-performance training at Meta, and the critical role of checkpointing in failure recovery and continuous learning for online training.