Data Pipelines Evolution from Batch to Streaming
Explore the evolution of data pipelines from batch to streaming systems in this 41-minute conference talk by Confluent. Discover how Apache Flink can bridge the gap between batch and streaming technologies while maintaining the same data pipeline definitions. Begin with an overview of typical batch systems using relational databases, then learn how to transition to streaming solutions using Apache Flink and Apache Kafka with minimal disruption. Examine query-based connectors that mimic batch behavior, and delve into advanced change data capture solutions using Debezium. Address critical topics such as data validation and late event arrival, and explore strategies to mitigate associated risks. Gain valuable insights for organizations considering the migration from batch to streaming systems while minimizing potential disruptions.