Neural Nets for NLP 2020 - Efficiency Tricks for Neural Nets
Explore efficiency tricks for neural networks in natural language processing through this lecture from CMU's Neural Networks for NLP course. Delve into tips for training on GPUs, parallel training techniques, and softmax approximations including negative sampling and hierarchical softmax. Learn about the practical aspects of matrix-matrix multiplication, memory considerations, and three types of parallelism: within-operation, operation-wise, and example-wise. Discover how to implement data parallelism using modern libraries like PyTorch DistributedDataParallel. Examine methods for computation across large vocabularies, including noise contrastive estimation, mini-batch based negative sampling, class-based softmax, and binary code prediction. Gain insights into the glamorous life of an AI scientist and explore improvements to binary code prediction techniques.