On Formulating and Evaluating Language Agents
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Explore the world of language agents in this informative talk presented by Shunyu Yao from Princeton University. Delve into the Cognitive Architectures for Language Agents (CoALA) framework, which provides a systematic approach to understanding and evaluating AI systems that utilize large language models (LLMs) for world interaction. Learn about three practical benchmarks - WebShop, InterCode, and Collie - designed to develop and assess language agents using web, code, and grammar respectively. Discover how these scalable benchmarks offer simple yet faithful evaluation metrics without relying on human preference labeling or LLM scoring. Gain insights into the future directions of language agent research and development from Yao, a final year PhD student supported by the Harold W. Dodds Fellowship at Princeton NLP Group.