BigScience BLOOM - 3D Parallelism Explained - Large Language Models - ML Coding Series
Dive into the fourth video of the Large Language Model series, exploring the BigScience BLOOM model codebase with a focus on understanding 3D parallelism. Learn about pipeline parallelism, model parallelism, and data parallelism - the engineering concepts behind recent scaling efforts and machine learning successes. Follow along as the video walks through the eval script, model construction, sharding techniques, and the forward pass. Gain insights into embedding table sharding, transformer layer sharding, attention layer sharding, and the intricacies of ColumnParallel and RowParallel sharding. Understand how dataset building relates to data parallelism and explore pipeline parallelism communication. Conclude with a comprehensive recap of the 3D parallelism concepts covered in this in-depth, 72-minute tutorial.