Главная
Study mode:
on
1
Intro
2
Who am I
3
Overview
4
dbt
5
sample code
6
dbt run
7
What dbt doesnt have
8
Apache Airflow
9
dbt in Airflow
10
Airflow dag file
11
What is Great Expectations
12
What is Great Expectations Statement
13
Typical Great Expectations Workflow
14
Automatic Profiling
15
Databox
16
Great Expectations Operator
17
Recap
18
Test your data
19
Where do we start
20
Technical pointers
21
Data testing
22
Data validation
23
Putting it all together
24
Airflow dag
25
Source data load validation
26
Running tests during development
27
Test integrity
28
Wrap up
29
QA
Description:
Explore the "dag Stack" - a robust data pipeline solution combining dbt, Airflow, and Great Expectations. Learn how to build a transformation layer with dbt, validate source data and add complex tests using Great Expectations, and orchestrate the entire pipeline with Apache Airflow. Discover practical examples of how these tools complement each other to ensure data quality, prevent "garbage in - garbage out" scenarios, and create comprehensive data documentation. Gain insights into automatic profiling, data testing, and validation techniques. Follow along with sample code demonstrations and technical pointers to implement this powerful stack in your own data engineering projects.

Building a Robust Data Pipeline with the DAG Stack - dbt, Airflow, and Great Expectations

Open Data Science
Add to list