Bridging Data Science and Data Infrastructure with Apache Airflow

Bridging Data Science and Data Infrastructure with Apache Airflow

Explore how Apache Airflow can bridge the gap between data science and data infrastructure in this PyCon US talk. Learn how to transform a single-machine Jupyter Notebook into a production-ready model served on Google Cloud Functions. Discover how Airflow's flexible Python interface facilitates collaboration between data engineers and data scientists, allowing for the creation of custom operators and diverse data pipelines. Follow the journey of building a cross-service Spark + Tensorflow pipeline with canary testing and hyper-parameter tuning. Gain insights into how Airflow can connect all layers of a data team to deliver rapid results while maintaining observable and resilient infrastructure.