Distributed Training: Hybrid Parallelism and Gradient Optimization - Lecture 20
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Learn advanced distributed training concepts in machine learning through a recorded MIT lecture that explores hybrid parallelism, auto-parallelization techniques, and strategies for overcoming bandwidth and latency bottlenecks. Dive deep into gradient compression methods including gradient pruning for sparse communication, deep gradient compression, and gradient quantization techniques like 1-Bit SGD and TernGrad. Master the implementation of delayed gradient updates while understanding their role in addressing latency challenges in distributed systems. Taught by Professor Song Han, this comprehensive lecture from MIT's 6.5940 course provides essential knowledge for optimizing large-scale machine learning training processes.