Training Llama 2 in Julia - Scaling Large Language Models

Training Llama 2 in Julia - Scaling Large Language Models

Discover how to train large language models like Llama(2) using Julia in this JuliaCon 2024 conference talk. Learn about scaling neural network training to multiple GPUs simultaneously using Dagger.jl and Flux.jl. Explore the challenges and solutions in implementing a parallel training pipeline for LLMs, including model and data description, job setup, and performance scaling. Gain insights into fine-tuning pre-trained models with techniques like Low Rank Adaptation (LoRA) for specialized tasks. Understand the components required for efficient large-scale GPU workloads in the Julia ecosystem and the potential applications for various model types beyond LLMs.