Finetuning, Serving, and Evaluating Large Language Models in the Wild

Finetuning, Serving, and Evaluating Large Language Models in the Wild

Dive into the world of large language models with this 29-minute conference talk by Dr. Hao Zhang, postdoctoral researcher at UC Berkeley's Sky Lab. Explore hands-on experiences with serving and evaluating over 20 LLM-based chatbots, including Vicuna, within the innovative Chatbot Arena. Gain insights into the open-source chatbot Vicuna, finely tuned from Meta's Llama, and discover the Chatbot Arena platform for real-world model evaluations. Uncover the challenges of serving numerous LLMs, achieving high throughput, and ensuring low latency with limited resources. Learn about key enabling techniques like paged attention (vLLM) and statistical multiplexing with model parallelism (AlpaServe), developed in collaboration with the LMSYS Org team. Understand the efficient memory management for LLM inference, vLLM's memory efficiency, and its open-source adoption.