Decoupling Compute and Storage for Stream Processing Systems
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Explore stream processing system architecture in this technical talk from RisingWave Labs CEO Yingjun Wu at the Dutch Seminar on Data Systems Design. Dive deep into the benefits and challenges of decoupling compute and storage in stream processing systems, with particular focus on addressing data consistency and latency issues when managing large-scale internal states. Learn about an innovative tiered storage mechanism that combines high-performance and low-cost storage tiers to optimize data movement between compute and storage layers. Examine experimental results demonstrating how this approach achieves an optimal balance between performance and cost-efficiency. Drawing from over a decade of experience in stream processing and database systems, including work at Amazon Web Services Redshift team and IBM Almaden Research Center, Wu shares practical insights into building scalable stream processing infrastructure.