Главная
Study mode:
on
1
Intro
2
Language Models • Language models are generative models of text
3
Conditioned Language Models
4
Calculating the Probability of a Sentence
5
Conditional Language Models
6
One Type of Language Model Mikolov et al. 2011
7
How to Pass Hidden State?
8
The Generation Problem
9
Ancestral Sampling
10
Greedy Search
11
Beam Search
12
Sentence Representations
13
Calculating Attention (1)
14
A Graphical Example
15
Attention Score Functions (1)
16
Attention is not Alignment! (Koehn and Knowles 2017)
17
Coverage
18
Multi-headed Attention
19
Supervised Training (Liu et al. 2016)
20
Self Attention (Cheng et al. 2016) • Each element in the sentence attends to other
21
Why Self Attention?
22
Transformer Attention Tricks
23
Transformer Training Tricks
24
Masking for Training . We want to perform training in as few operations as possible using big matrix multiplies
25
A Unified View of Sequence- to-sequence Models
26
Code Walk
Description:
Explore machine translation and sequence-to-sequence models in this 44-minute lecture from CMU's Multilingual Natural Language Processing course. Delve into conditional language modeling, simple sequence-to-sequence models, generation methods, attention mechanisms, and self-attention/transformers. Learn about calculating sentence probabilities, hidden state passing, and various generation techniques including ancestral sampling, greedy search, and beam search. Examine sentence representations, attention score functions, and the distinction between attention and alignment. Discover multi-headed attention, supervised training approaches, and the intricacies of self-attention in Transformer models. Gain insights into Transformer training tricks, masking techniques for efficient training, and a unified view of sequence-to-sequence models. Conclude with a code walkthrough to solidify understanding of these advanced NLP concepts.

CMU Multilingual NLP - Machine Translation-Sequence-to-Sequence Models

Graham Neubig
Add to list
0:00 / 0:00