Developing Elegant Workflows in Python Code with Apache Airflow

Developing Elegant Workflows in Python Code with Apache Airflow

Learn to develop elegant workflows in Python using Apache Airflow in this EuroPython 2017 conference talk. Explore the concept of workflows as directed acyclic graphs (DAGs) and discover how Airflow enables you to bring block diagrams to life. Dive into basic Airflow concepts, including tasks, operators, and sensors. Through practical examples, master the art of defining workflows in Python code, implementing conditional execution, and leveraging Bash commands with templates. Gain insights into extending Airflow's functionality by creating custom task operators, sensors, and plugins. Understand how Airflow's resilient design handles task retries, workflow resumption, and log management through its user-friendly web interface.