Language Models • Language models are generative models of text
3
Conditioned Language Models
4
Calculating the Probability of a Sentence
5
Conditional Language Models
6
One Type of Language Model Mikolov et al. 2011
7
How to Pass Hidden State?
8
The Generation Problem
9
Ancestral Sampling
10
Greedy Search
11
Beam Search
12
Ensembling . Combine predictions from multiple models
13
Linear Interpolation • Take a weighted average of the M model probabilities
14
Log-linear Interpolation • Weighted combination of log probabilities, normalize
15
Linear or Log Linear?
16
Parameter Averaging
17
Ensemble Distillation (e.g. Kim et al. 2016)
18
Stacking
19
Still a Difficult Problem!
20
From Speaker/Document Traits (Hoang et al. 2016)
21
From Lists of Traits (Kiddon et al. 2016)
22
From Word Embeddings (Noraset et al. 2017)
23
Basic Evaluation Paradigm
24
Human Evaluation Shared Tasks
25
Embedding-based Metrics
26
Perplexity
27
Which One to Use?
Description:
Learn about conditioned generation in neural networks for natural language processing in this lecture from CMU's Neural Networks for NLP course. Explore encoder-decoder models, conditional generation techniques, and search algorithms like beam search. Examine methods for ensembling multiple models, including linear interpolation and parameter averaging. Discover various types of data that can be used to condition language models, from speaker traits to word embeddings. Gain insights into evaluation paradigms for generative models, covering human evaluation, embedding-based metrics, and perplexity. Understand the strengths and limitations of different evaluation approaches for assessing language model performance.