Modern ETL Pipelines with Change Data Capture - Building Resilient Data Streams
Explore the development of a modern ETL pipeline using Debezium, Kafka, Spark, and Airflow in this 43-minute conference talk. Learn how GetYourGuide transformed their error-prone legacy system into a robust, schema-change-resilient pipeline capable of multiple daily data lake refreshes. Discover the architecture and implementation steps for building a Change Data Capture layer that streams database changes directly to Kafka. Gain insights into reducing operational time with Databricks and understand the benefits of fresh data for business users. The talk covers the extraction layer, schema service, data landscape, dependency management, transformation layer components, and the importance of testing. Explore special syntax elements, the small file problem, and data warehouse integration. Conclude with a Q&A session addressing how the new pipeline works and its read-write capabilities.