Главная
Study mode:
on
1
Data preparation for LLMs
2
Downloading the LangChain docs
3
Using LangChain document loaders
4
How much text can we fit in LLMs?
5
Using tiktoken tokenizer to find length of text
6
Initializing the recursive text splitter in Langchain
7
Why we use chunk overlap
8
Chunking with RecursiveCharacterTextSplitter
9
Creating the dataset
10
Saving and loading with JSONL file
11
Data prep is important
Description:
Explore essential data preparation techniques for Large Language Models in this comprehensive tutorial video. Learn how to effectively use LangChain data loaders, tokenize text with tiktoken tokenizers, implement chunking strategies using LangChain text splitters, and store data using Hugging Face datasets. Gain practical insights into preparing text for OpenAI embedding and completion models, with principles applicable to other LLMs like those from Hugging Face and Cohere. Follow along as the instructor demonstrates downloading LangChain documentation, utilizing document loaders, determining optimal text lengths for LLMs, and implementing recursive text splitting with chunk overlap. Discover the importance of proper data preparation and learn how to create, save, and load datasets using JSONL files.

LangChain Data Loaders, Tokenizers, Chunking, and Datasets - Data Prep

James Briggs
Add to list
0:00 / 0:00