Reminder: Optimizers - SGD: take a step in the direction of the gradient
9
Learning Rate Learning rate is an important parameter
10
Initialization
11
Debugging Minibatching
12
Debugging Decoding
13
Debugging Search
14
Look At Your Data!
15
Quantitative Analysis
16
Symptoms of Overfitting
17
Reminder: Early Stopping, Learning Rate Decay
18
Reminder: Dropout (Srivastava et al. 2014) Neural nets have lots of parameters, and are prone to overfitting • Dropout: randomly zero-out nodes in the hidden layer with probability p at training time…
19
A Stark Example (Koehn and Knowles 2017) • Better search (=better model score) can result in worse BLEU score!
20
Managing Loss Function/ Eval Metric Differences Most principled way: use structured prediction techniques to be discussed in future classes
21
A Simple Method: Early Stopping w/ Eval Metric
22
Reproducing Previous Work
Description:
Explore techniques for debugging neural networks in natural language processing applications. Learn to identify and address common issues such as training time problems, model weakness, optimization challenges, and overfitting. Discover strategies for tuning hyperparameters, initializing weights, and managing minibatches. Examine the importance of data analysis, quantitative evaluation, and the relationship between loss functions and evaluation metrics. Gain insights into early stopping, dropout, and other techniques to improve model performance. Understand the complexities of search and decoding in NLP tasks, and learn how to reproduce previous research results effectively.
Neural Nets for NLP - Debugging Neural Nets for NLP