What is Our Loss Function? . We would like to maximize the corpus log likelihood
7
Disconnect Between Samples and Objective
8
VAE Objective . We can create an optimizable objective matching our problem, starting with KL divergence
9
Interpreting the VAE Objective
10
Problem: Straightforward Sampling is Inefficient Current
11
Problem! Sampling Breaks Backprop
12
Solution: Re-parameterization Trick
13
Generating from Language Models
14
Motivation for Latent Variables
15
Difficulties in Training
16
KL Divergence Annealing
17
Weaken the Decoder
18
Discrete Latent Variables?
19
Enumeration
20
Method 2: Sampling
21
Reparameterization (Maddison et al. 2017. Jang et al. 2017)
Description:
Explore latent variable models in neural networks for natural language processing in this comprehensive lecture from CMU's Neural Networks for NLP course. Delve into the differences between generative and discriminative models, as well as deterministic and random variables. Learn about Variational Autoencoders (VAEs) and their applications in NLP, including techniques for handling discrete latent variables. Discover the challenges in training latent variable models and strategies to overcome them, such as KL divergence annealing and weakening the decoder. Examine methods for dealing with discrete latent variables, including enumeration, sampling, and reparameterization techniques. Access accompanying slides and code examples to reinforce your understanding of these advanced concepts in neural network-based NLP.