Neural Nets for NLP 2019 - Attention
Explore a comprehensive lecture on attention mechanisms in neural networks for natural language processing. Delve into the fundamentals of attention, including what to attend to, improvements to attention techniques, and specialized attention varieties. Examine a detailed case study on the "Attention is All You Need" paper. Learn about encoder-decoder models, sentence representations, and the basic idea behind attention as proposed by Bahdanau et al. in 2015. Discover various attention score functions, hierarchical structures, and techniques for handling multiple sources. Address common problems in neural models, such as dropping or repeating information, and explore solutions like incorporating Markov properties and bidirectional training. Gain insights into hard attention, the Transformer architecture, and various attention tricks. Conclude with training techniques, including masking for efficient training operations.