Главная
Study mode:
on
1
Intro
2
GoDataDriven
3
Data Build Tool
4
SOL with some Ninja2 sauce
5
DBT as a SOL Runner
6
DBT as a SOL Compiler
7
Next to the SOL there is documentation
8
dbt docs generate dbt docs serve
9
Testing
10
How does DBT communicate with Spark?
11
Switch to incremental ingestion
12
Switch to incremental Delta
13
In practice
14
DBT Macro's
15
Observability is king
16
Very simple Hive UDF
17
Small snippet of Scala
18
Use the UDF in DBT
19
Be proactive
20
Feedback
Description:
Explore a comprehensive 26-minute talk on integrating Data Build Tool (DBT) with Databricks and Delta for efficient data lake management. Learn how this open-source, SQL-first technology enhances data quality and documentation throughout the data lake lifecycle. Discover the basics of DBT and its synergy with Databricks for powerful data processing. Examine how DBT supports Delta to enable SQL-based upserts. Investigate the integration of DBT and Databricks within the Azure cloud environment. Gain insights into emitting pipeline metrics to Azure Monitor for improved observability. Dive into topics such as DBT as a SQL runner and compiler, documentation generation, testing, incremental ingestion, DBT macros, and the use of Hive UDFs. Master the art of maintaining high-quality data pipelines using software engineering best practices.

DBT Using Databricks and Delta

Databricks
Add to list
0:00 / 0:00