AGENDA DATA FACTORY INGESTION FRAMEWORK 1 Intro to Data Factory
9
What is Data Factory? ? Data Orchestration - Originally developed for data integration across the cloud where storage and compute are separated
10
Azure Data Factory
11
Data Factory Sources
12
ETL Automation - Traditional
13
ELT Automation - Modern Extract
14
Data Ingestion Components Data Lake Ingestion takes into account the following components...
15
Data Lake Organization RAW ZONE Delta Captures /ServerName
16
Ingestion Conceptual Architecture
17
Data Vault Example
18
Learn Data Vault
19
Loading the Model
20
ADF: Schema Loader
21
ADF: Data Loader
22
Dynamic Linked Services FOREACH ACTIVITY
23
Dynamic SQL in ADF
24
Metadata Model Views
25
Other Features
26
CONTACT
Description:
Explore a comprehensive framework for data lake ingestion using Azure Data Factory in this 52-minute conference talk from PASS Data Community Summit. Dive into data strategy and its impact on framework design, covering the development of a metadata database using Data Vault modeling, metadata collection techniques, and the creation of a fully automated Data Factory pipeline. Learn about data orchestration, ETL/ELT automation, data lake organization, and dynamic SQL implementation in ADF. Gain insights into ingestion conceptual architecture, schema loading, and other key features to enhance your data engineering skills and optimize your Azure-based data solutions.