Writing and Scaling Collaborative Data Pipelines with Kedro
Explore the world of collaborative data pipelines with Kedro in this EuroPython 2020 conference talk. Learn how QuantumBlack's open-source framework addresses the challenges of maintaining healthy and sustainable data pipelines while facilitating collaboration between data scientists and data engineers. Discover how Kedro balances the need for data volatility and parameterization with stability and performance, enabling teams to scale from single-developer micro-pipes to industrial-sized data processors. Gain insights into writing high-quality, fully featured data pipelines that keep pace with growing business requirements and team expansion. Suitable for Python developers with novice to intermediate knowledge, this talk covers the introduction to data engineering and pipelines, the growth of pipelines, the rationale behind Kedro, its problem-solving capabilities, and includes a practical demo.