Airflow: Save Money Using Deferrable Operators - Introduction and Implementation

Airflow: Save Money Using Deferrable Operators - Introduction and Implementation

Discover how to significantly reduce costs in Apache Airflow 2.2 by leveraging deferrable operators in this informative conference talk. Learn about the concept of deferrable tasks utilizing Python's async feature, and explore how to optimize Airflow sensors and poll-based operators to free up worker slots during polling. Delve into the introduction of deferrable operators, understand when and why to use them, and gain insights on creating custom deferrable operators and sensors. Compare the advantages of deferrable operators over smart sensors and reschedule mode for sensors. Explore various topics including typical operators, async operators, trigger baselines, and available operators through practical examples and demonstrations.