Apache Linkis Data Processing Practice in Lake-Silo Architecture
Explore Apache Linkis data processing practices in lake-silo architecture through this 15-minute conference talk by Wang Hualei, Deputy Chief Engineer at Postal Savings Bank of China. Gain insights into how a large state-owned bank solves practical problems using Linkis within an integrated lake-warehouse framework. Discover the big data lake-warehouse integrated structure of PSBC and learn about implementation challenges, including complex maintenance of technical components, high technical thresholds for data development, rapid component version upgrades, and metadata communication issues. Examine specific Apache Linkis practices, such as implementing underlying interconnection of computing components, prioritizing pure SQL development, supporting multiple component versions, and utilizing Hive Catalog for unified metadata interface. Explore contributions to the Apache Linkis community, including PostgreSQL support, S3 file storage integration, and containerized deployment practices. Conclude with future technology planning, focusing on strengthening data lake technologies like Iceberg based on Linkis. Read more