Observability for Data Pipelines with OpenLineage
Explore the critical importance of observability in data pipelines through a 24-minute video presentation from Databricks. Dive into the world of OpenLineage, an API standardizing metadata collection across ecosystems, and discover how it enhances data operations, governance, and security. Learn about Marquez, an open-source project that implements OpenLineage API to provide visibility into dependencies across organizations and technologies. Gain insights into how collecting lineage metadata during pipeline execution can improve auditability, reliability, and timeliness of data operations in fast-paced environments. Understand the role of metadata in creating reliable data products and how OpenLineage reduces complexity in lineage information collection. Examine the Spark integration and its purpose in enhancing data pipeline observability.