Migrating and Optimizing Large-Scale Streaming Applications with Databricks
Explore the process of rearchitecting, redeveloping, and migrating a massive streaming application to Databricks Spark Structured Streaming in this informative conference talk. Discover how a large-scale system processing hundreds of billions of ad events daily at over 5GB/s was transformed to enable real-time analytics, batch reporting, ML-based forecasting, and streaming ad log delivery for programmatic ad campaigns. Learn about the lessons, substantial benefits, and performance enhancements achieved through various optimizations, including memory-related improvements, Kinesis parameter tuning, and parallelizing the output stage within each micro-batch. Gain insights into FreeWheel's programmatic advertising platform, the architecture of their larger data platform, and their robust monitoring and observability solution. Explore how Databricks features, such as the AI assistant, enhanced the development experience throughout the migration process.