Главная
Study mode:
on
1
Introduction
2
About Northwestern Mutual
3
Agenda
4
Need for metadata management
5
Ease of use
6
Design
7
Configuration Files
8
Demo
9
CICD
10
Wrap Up
Description:
Explore a 28-minute conference talk on implementing automated metadata management in data lakes using a CI/CD-driven approach. Learn how Northwestern Mutual engineers developed a tool to balance rapid metadata changes with robust validation for downstream system stability. Discover the architecture and design of their centralized git-managed repository for data schemas, utilizing YAML structures and CI/CD capabilities. Gain insights into maintaining information on databases, tables, and views, including schema, ownership, PII, and descriptions. Watch a live demo of creating a new table with CI/CD promotion to production, and understand how this tool can be used effectively by individuals with minimal Spark knowledge.

Automated Metadata Management in Data Lakes - A CI/CD Driven Approach

Databricks
Add to list