LLMOps: OpenVino Toolkit - Quantizing LLama3.1 8B to 4int and CPU Inference

LLMOps: OpenVino Toolkit - Quantizing LLama3.1 8B to 4int and CPU Inference

Explore the process of converting a LLAMA3.1 8B model to OpenVino IR format and quantizing it to 4-bit integer precision in this 19-minute video tutorial. Learn how to perform inference on CPU using the quantized model. Access the accompanying notebook on GitHub to follow along with the step-by-step instructions for model conversion, quantization, and inference using the OpenVino Toolkit. Gain practical insights into LLMOps techniques for optimizing large language models for efficient deployment on CPU hardware.