Dumb-Proofing Data Pipelines: Techniques for Configurable and Maintainable ETL - Databricks

Dumb-Proofing Data Pipelines: Techniques for Configurable and Maintainable ETL - Databricks

Discover techniques to create robust and maintainable data pipelines in this 22-minute Databricks talk. Learn why configurable pipelines are crucial, how to seamlessly promote them across environments, and reconfigure in production without recompiling. Explore the pros and cons of Databricks Notebook widgets, methods to externalize configurations, and leverage Scala features with pure config and typesafe libraries for boilerplate-free code. Gain insights on input validation, preventing data loss and corruption, and ensuring data correctness. Walk away with practical knowledge to enhance your data pipeline development and maintenance processes.