Главная
Study mode:
on
1
Introduction
2
Federicos background
3
Product problem
4
Is it simple
5
Data pipelines
6
Scale
7
Stages
8
Archive
9
Airflow
10
Python
11
Database
12
UI
13
Workflow
14
Operators
15
Airflow UI
16
Airflow scheduling
17
Tracking state
18
Downtime
19
Scrapers
20
Batch IDs
21
Timestamp
22
Formats
23
Redshift copy command
24
JSON path flattening
25
Schema conversion
26
Migration framework
27
Redshift annoyances
28
Futureproof
29
Thank you
Description:
Explore a comprehensive talk from EuroPython 2017 on leveraging Airflow for efficient data pipelines to AWS Redshift. Dive into the fundamentals of Airflow, including its scheduling capabilities and workflow management features. Learn about data pipeline-specific concepts such as backfills and retries, and discover practical examples of integration. Gain insights into structuring data in Redshift, performing basic pre-loading transformations, and managing schemas using SQLAlchemy and Alembic. Follow along as the speaker shares valuable lessons learned and addresses common Redshift challenges. Perfect for data engineers and analysts looking to optimize their ETL processes and harness the power of Airflow in conjunction with AWS Redshift.

Feeding Data to AWS Redshift with Airflow

EuroPython Conference
Add to list
0:00 / 0:00