Neural Nets for NLP 2020 - Language Modeling, Efficiency/Training Tricks
Explore a comprehensive lecture on language modeling and neural network training techniques for natural language processing. Delve into feed-forward neural network language models, methods to prevent overfitting, and mini-batching techniques. Learn about automatic optimization, including automatic minibatching and code-level optimization, as well as various optimizers. Discover how to measure language model performance using accuracy, likelihood, and perplexity metrics. Gain insights into handling unknown words, evaluation strategies, and vocabulary considerations. Examine the limitations of linear models and explore neural language models, including input/output embedding tying. Study different optimization algorithms such as standard SGD, SGD with momentum, Adagrad, and Adam. Understand the importance of shuffling training data for improved model performance.