Главная
Study mode:
on
1
LLM Full fine-tuning with lower VRAM
2
Video Overview
3
Understanding Optimisers
4
Stochastic Gradient Descent SGD
5
AdamW Optimizer and VRAM requirements
6
AdamW 8-bit optimizer
7
Adafactor optimiser and memory requirements
8
GaLore - reducing gradient and optimizer VRAM
9
LoRA versus GaLoRe
10
Better and Faster GaLoRe via Subspace Descent
11
Layerwise gradient updates
12
Training Scripts
13
How gradient checkpointing works to reduce memory
14
AdamW Performance
15
AdamW 8bit Performance
16
Adafactor with manual learning rate and schedule
17
Adafactor with default/auto learning rate
18
Galore AdamW
19
Galore AdamW with Subspace descent
20
Using AdamW8bit and Adafactor with GaLoRe
21
Notebook demo of layerwise gradient updates
22
Running with LoRa
23
Inferencing and Pushing Models to Hub
24
Single GPU Recommendations
25
Multi-GPU Recommendations
26
Resources
Description:
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Explore advanced techniques for full fine-tuning of large language models with limited GPU resources in this comprehensive video tutorial. Dive deep into optimizer strategies, including Stochastic Gradient Descent (SGD), AdamW, and Adafactor, while learning about their VRAM requirements and performance implications. Discover the innovative GaLore method for reducing gradient and optimizer VRAM usage, and compare it with LoRA. Gain insights into layerwise gradient updates, gradient checkpointing, and the implementation of various optimizers. Follow along with practical demonstrations, including a notebook demo of layerwise gradient updates and running models with LoRA. Learn how to inference and push models to the hub, and receive valuable recommendations for both single and multi-GPU setups. Access a wealth of resources, including slides, GitHub repositories, and additional support channels to enhance your understanding of advanced fine-tuning techniques.

Full Fine-tuning LLMs with Lower VRAM: Optimizers, GaLore, and Advanced Techniques

Trelis Research
Add to list
#Computer Science #Machine Learning #Fine-Tuning #Gradient Descent #Deep Learning #Stable Diffusion #LoRA (Low-Rank Adaptation)
1 of 26

LLM Full fine-tuning with lower VRAM