Automated Metadata Management in Data Lakes - A CI/CD Driven Approach

Automated Metadata Management in Data Lakes - A CI/CD Driven Approach

Explore a 28-minute conference talk on implementing automated metadata management in data lakes using a CI/CD-driven approach. Learn how Northwestern Mutual engineers developed a tool to balance rapid metadata changes with robust validation for downstream system stability. Discover the architecture and design of their centralized git-managed repository for data schemas, utilizing YAML structures and CI/CD capabilities. Gain insights into maintaining information on databases, tables, and views, including schema, ownership, PII, and descriptions. Watch a live demo of creating a new table with CI/CD promotion to production, and understand how this tool can be used effectively by individuals with minimal Spark knowledge.