Designing Functional Data Pipelines for Reproducibility and Maintainability
Explore functional programming principles for building scalable, reproducible, and maintainable data pipelines in this EuroPython 2021 conference talk. Learn about the key differences between functional programming and other paradigms, understand how control flow is implemented in functional programming, and discover functional design patterns for data pipeline design in Python. Gain insights into common challenges faced when designing reproducible and maintainable data pipelines at scale, and see how functional programming can address these issues. Through analogies and realistic examples inspired by production environments, examine the possibility of writing purely functional programs in Python. Suitable for data scientists, engineers, and developers with basic understanding of building data pipelines using functions and classes/objects, this talk provides valuable knowledge for creating reliable and extensible data-intensive applications.