Accelerating Serverless AI Large Model Inference with Functionalized Scheduling and RDMA
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Explore a conference talk on accelerating serverless AI large model inference through functionalized scheduling and RDMA technology. Dive into the challenges of deploying AI large models on standard serverless inference platforms like KServe, including scheduling inefficiencies and communication bottlenecks. Learn about a highly elastic functionalized scheduling framework developed to achieve second-level scheduling for thousands of serverless AI large model inference task instances. Discover how RDMA technology is leveraged to enable high-speed KV cache migration, overcoming the limitations of traditional network protocol stacks. Gain insights into improving resource utilization, reducing costs, and meeting low-latency and high-throughput demands in AI large model inference deployments.