Challenges and Solutions for Building Real-time Data Warehousing with Apache Flink, Apache Hive and Apache Iceberg

Challenges and Solutions for Building Real-time Data Warehousing with Apache Flink, Apache Hive and Apache Iceberg

Explore the challenges and solutions for building real-time data warehousing systems using Apache Flink, Apache Hive, and Apache Iceberg in this 36-minute conference talk. Gain insights from Yan Liu, an Apache Hive and Apache Flink contributor with over 10 years of experience in big data. Learn about the latest community developments and architectural designs for migrating batch processing Enterprise Data Warehouses (EDW) to real-time processing EDWs. Discover strategies for handling late events, dirty data routing, and other challenges encountered when transitioning to real-time ETL processing. Understand how to leverage these powerful open-source technologies to create enterprise-level real-time data warehousing solutions.