Open Pretrained Transformer - ML Coding Series

Open Pretrained Transformer - ML Coding Series

Dive deep into the metaseq codebase behind Meta's large language model OPT-175B in this comprehensive video tutorial. Learn how to set up the code on your machine and explore key concepts of mixed precision training, including loss scaling and unscaling. Follow along as the instructor walks through the training script, constructs dummy tasks and datasets, builds the transformer model, and examines CUDA kernels in C++ code. Gain insights into the training loop, forward pass through a transformer, and crucial aspects of loss handling, scaling, and mixed precision. Perfect for those looking to understand the intricacies of large language model implementation and training.