Главная
Study mode:
on
1
Intro
2
What is a Streaming Live Table? Based on Spark™ Structured Streaming
3
Development vs Production Fast iteration or enterprise grade reliability
4
Choosing pipeline boundaries Break up pipelines at natural external divisions.
5
Pitfall: hard-code sources & destinations Problem: Hard coding the source & destination makes it impossible to test changes outside of production, breaking CI/CD
6
Ensure correctness with Expectations Expectations are tests that ensure data quality in production
7
Expectations using the power of SQL Use SQL aggregates and joins to perform complex validations
8
Using Python Write advanced DataFrame code and UDFs
9
Installing libraries with pip pip is a package installer for python
10
Best Practice: Integrate using the event log Use the information in the event log with your existing operational tools.
11
DLT Automates Failure Recovery Transient issues are handled by built-in retry logic
12
Modularize your code with configuration Avoid hard coding paths, topic names, and other constants in your code.
13
Workflow Orchestration For Triggered DLT Pipelines
14
Use Delta for infinite retention Delta provides cheap, elastic and governable storage for transient sources
Description:
Explore declarative ETL pipelines with Delta Live Tables in this 51-minute SQLBits conference talk. Learn about modern software engineering and management techniques for ETL, enabling data analysts and engineers to focus on extracting value from data rather than tooling. Discover the concept of Streaming Live Tables based on Spark Structured Streaming, and understand the differences between development and production environments. Gain insights on choosing pipeline boundaries, avoiding pitfalls like hard-coding sources and destinations, and ensuring data quality through Expectations. Delve into using SQL and Python for complex validations and advanced DataFrame operations. Learn best practices for integrating with existing operational tools, automating failure recovery, and modularizing code with configuration. Explore workflow orchestration for triggered DLT pipelines and leveraging Delta for infinite retention of transient sources. Speaker Vuong Nguyen provides valuable knowledge on Azure, Spark, Data Lake, AWS, GCP, and DataBricks, covering topics in Big Data & Data Engineering for developers and data professionals. Read more

Declarative ETL Pipelines with Delta Live Tables - Modern Software Engineering for Data Analysts and Engineers

SQLBits
Add to list
#Data Science #Big Data #Apache Spark #Delta Lake #Delta Live Tables #Programming #Programming Languages #Python #Domain-Specific Languages (DSL) #SQL #Business #Business Intelligence #Data Lakes #Data Engineering #Data Pipelines #ETL
0:00 / 0:00
1 of 14

Intro