Declarative ETL Pipelines with Delta Live Tables - Modern Software Engineering for Data Analysts and Engineers
Explore declarative ETL pipelines with Delta Live Tables in this 51-minute SQLBits conference talk. Learn about modern software engineering and management techniques for ETL, enabling data analysts and engineers to focus on extracting value from data rather than tooling. Discover the concept of Streaming Live Tables based on Spark Structured Streaming, and understand the differences between development and production environments. Gain insights on choosing pipeline boundaries, avoiding pitfalls like hard-coding sources and destinations, and ensuring data quality through Expectations. Delve into using SQL and Python for complex validations and advanced DataFrame operations. Learn best practices for integrating with existing operational tools, automating failure recovery, and modularizing code with configuration. Explore workflow orchestration for triggered DLT pipelines and leveraging Delta for infinite retention of transient sources. Speaker Vuong Nguyen provides valuable knowledge on Azure, Spark, Data Lake, AWS, GCP, and DataBricks, covering topics in Big Data & Data Engineering for developers and data professionals. Read more