Introduction to GKE Setup of Airflow with Helm and Terraform
Master GKE deployment strategies using Terraform and Helm to efficiently set up and manage Airflow workflows in GitLab, with focus on Pod scheduling and infrastructure automation.
Introduction to Orchestrating ETL Processes with Apache Airflow
Master Apache Airflow's workflow management capabilities for ETL processes through practical examples. Learn core concepts like DAGs, Tasks, Variables, and XComs while building efficient data engineering pipelines.
Automating Airflow Backfills with Marquez
Learn to automate Airflow backfills using Marquez, enhancing data pipeline efficiency and reliability in complex workflows.
Lessons Learned from the Migration to Apache Airflow
Explore Apache Airflow for orchestrating complex workflows, with insights on ETL pipelines, Spark integration, and best practices for managing big data infrastructure.
Airflow: Save Money Using Deferrable Operators - Introduction and Implementation
Learn to optimize Apache Airflow tasks using deferrable operators, saving resources and costs. Explore implementation, advantages, and custom development for efficient workflow management.
MATS Stack for Cross-System Orchestration of Machine Learning Pipelines
Explore the MATS stack for orchestrating ML pipelines across systems, featuring MLFlow, Airflow, Tensorflow, and Spark. Learn how Avast leverages this approach for efficient phishing detection and streamlined ML workflows.
Kubernetes for Modern Data Engineering: An End-to-End Data Engineering Project
Master Kubernetes for data engineering: set up, deploy dashboard, run Apache Airflow with Helm charts. Learn kubectl, manage clusters, and optimize workflows for scalable data processing.
Orchestrating Your Data Pipelines with Apache Airflow
Introduction to Apache Airflow for orchestrating data pipelines, covering key components and demonstrating simple pipeline builds. Includes practical demos and insights on Airflow's evolution and community contributions.
Comparing Argo Workflows and Airflow in Distributed Environments
Explore workflow orchestration tools for distributed environments, focusing on Argo Workflows' advantages in Kubernetes-based pipelines and its scalability across multiple regions and zones.
Bridging Data Science and Data Infrastructure with Apache Airflow
Explore how Apache Airflow bridges data science and infrastructure, enabling collaboration between teams and facilitating the development of flexible, observable, and resilient data pipelines.
In House to Open Source - Stitching the Past to the Future
Explore the transition from legacy code to open-source solutions, focusing on Python's advantages in software migrations and the implementation of robust frameworks like Apache Airflow.
Too Big for DAG Factories?
Learn to scale data workflows using Apache Airflow, covering dynamic DAG generation, automation, and best practices for managing big data pipelines. Suitable for beginners and intermediate developers.
Developing Elegant Workflows in Python Code with Apache Airflow
Learn to orchestrate data processing pipelines using Apache Airflow. Discover how to define, manage, and extend workflows with Python code, creating resilient and efficient data processing systems.
Feeding Data to AWS Redshift with Airflow
Explore Airflow for efficient data pipelines and learn to integrate it with AWS Redshift for effective ETL processes, schema management, and data transformation techniques.
Reproducible and Deployable Data Science with Open-Source Python
Aprenda a criar código de ciência de dados escalável, reproduzível e confiável usando Kedro, Apache Airflow e Great Expectations. Descubra como combinar iteração rápida com manutenibilidade e qualidade de dados consistente.
Orchestrating Python Workflows in Apache Airflow
Explore Apache Airflow for efficient Python workflow orchestration. Learn strengths, weaknesses, key features, and how to create custom operators based on 5 years of production experience.
Building MLOps POCs and Sandbox Environments Using k3s and Argo
Explore building MLOps POCs using k3s and Argo. Learn to reduce implementation time and costs for data pipeline automation and ML workflows, with insights on lightweight Kubernetes and Edge Computing.
Building Mobility Data Platform with EMR and Airflow
Discover how to build and optimize a mobility data platform using AWS EMR and Airflow, focusing on migration strategies, cost optimization, and infrastructure automation with Terraform.
Managed Airflow Service Vane Development Journey
Dive into the development journey of Vane, a managed Airflow SaaS platform, exploring its architecture, implementation challenges, and valuable insights gained from operating data platforms at scale.
On-Premise Kubernetes and Airflow ETL Platform (Haflow) - Introduction and Implementation
Discover SK하이닉스의 클라우드 네이티브 ETL 플랫폼 Haflow의 구축 사례와 아키텍처를 통해 기존 ETL 솔루션의 한계 극복 방안과 리소스 최적화 전략을 파악하세요.
Making Data the Central Feature: From Task-Oriented to Data-Oriented Workflows
Explore the shift from task-oriented to data-oriented workflows in modern data engineering, focusing on tools and approaches that prioritize data assets over process management.