Главная
Study mode:
on
1
Intro
2
Agenda
3
Food Waste
4
Flashfood Data
5
Problem Definition
6
Attempt 1
7
The Declarative Data Pipeline
8
Attempt 3
9
The right amount of automation
10
Why configs?
11
airflow-declarative
12
SyncTablelob
13
Custom Operator
14
Extract
15
Transform
16
Summary
17
Lessons Learned
18
Challenges ahead
19
Spark YAML
20
Keillor's Principles
21
Feedback
Description:
Explore declarative data pipelines in this 28-minute conference talk featuring Anthony Awuley and Carter Kilgour from Flashfood. Learn about food waste reduction efforts, Flashfood's data challenges, and the evolution of their data pipeline solutions. Discover the benefits of declarative approaches, including code reusability and automation. Gain insights into tools like airflow-declarative, SyncTable, and custom operators for extract and transform processes. Understand key lessons learned, future challenges, and the potential of Spark YAML. Conclude with a discussion on Keillor's Principles and the importance of feedback in data engineering.

Code Once, Use Often - Declarative Data Pipelines

Databricks
Add to list