4 Tricks to Leverage Your Airflow Pipelines
Discover four powerful techniques to enhance Airflow pipelines: CSV-based DAG creation, sharable config utilization, entity-based autogeneration, and event-driven implementation.
Four Tricks to Leverage Your Apache Airflow Pipelines
Leverage CSV files, Google Sheets, Airflow Connections, and event-driven techniques to optimize Apache Airflow pipelines and streamline data engineering workflows.
Modern Data Orchestration: Best Practices and Real-World Use Cases
Elevate your data pipeline game with real-world tricks and best practices. Explore patterns, avoid dependencies, and integrate fundamental concepts for efficient data orchestration.
Gatekeeping Data Quality with Apache Iceberg, Toree, and Airflow
Leverage Apache Iceberg, Toree, and Airflow for efficient data quality validation. Customize data audits with Jupyter notebooks and orchestrate workflows at scale using open-source Apache projects.
Creating and Submitting Scala Jobs to Spark Clusters Using Airflow
Build end-to-end data engineering project using Airflow, Docker, Spark, Scala, Python, and Java. Create and submit jobs to Spark clusters, process data, and view real-time results.
Creating and Submitting PySpark Jobs to Spark Clusters
Create and submit PySpark jobs to Spark clusters in an end-to-end data engineering project using Airflow, Docker, and multiple programming languages. Learn job creation, cluster submission, and result visualization.
End-to-End Data Engineering with Apache Airflow, Docker, and Spark Clusters - Using Python, Scala, and Java
Build an end-to-end data engineering project using Apache Airflow, Docker, Spark Clusters, Scala, Python, and Java. Create and submit jobs in multiple languages to a Spark cluster for processing and real-time results.
Football Data Analytics with Tableau - Azure Data Engineering Project Part 3
Explore football data engineering: Extract Wikipedia data, store in Azure, migrate with Data Factory, query with Synapse, and visualize in Tableau. Master end-to-end data processes.
Creating and Submitting Java Spark Jobs to Spark Clusters
Create and submit Java Spark jobs to clusters. Learn end-to-end data engineering with Airflow, Docker, Spark, Scala, Python, and Java. Build and process jobs in multiple languages for live results.
Resilient Predictive Data Pipelines
Explore resilient predictive data pipelines with insights on data security, fault-tolerance, and scalability. Learn about Apache Airflow, AWS Lambda, and elastic stream processing for efficient big data management.
Dynamic Workflow Orchestration with Apache Airflow and CrateDB
Learn to automate and orchestrate complex database processes using Apache Airflow and CrateDB. Discover how to set up dynamic workflows, leverage task mapping, and efficiently manage large-scale data operations.
Using Open Source Tech to Swap Out Components of Your Data Pipeline
Explore open-source technologies for flexible data pipelines, from Apache Kafka to BigQuery, with easy component swapping and cloud deployment options.
Cloud Native Data Pipelines
Explore cloud-native data pipeline architecture, focusing on resilience, cost-efficiency, and operability. Learn about message scoring, Apache Airflow, stream processing, and Avro in building robust data systems.
Applied Data Science & Engineering for Local Weather Forecasts
Explore data science applications in local weather forecasting, covering numerical prediction models, scaling with Apache Airflow, and forecast verification techniques for improved accuracy.
Airflow в кровавом энтерпрайзе
Discover how to effectively deploy and implement Airflow for enterprise automation, with step-by-step guidance on setup, configuration, and best practices for streamlining routine processes.
Airflow Distributed Workloads - ETL Architecture with AWS Lambda and Multi-Threaded Python Scripts
Master distributed workload processing with Airflow, AWS Lambda, and multithreaded Python scripts while exploring ETL architecture for third-party API integration at scale.
Using Frameworks to Test Apache Airflow Data Pipelines
Explore frameworks for testing Apache Airflow data pipelines, covering local development, CI/CD, validation, unit tests, and integration tests. Includes a practical demo and key takeaways.
OpenLineage: An Open Standard for Data Lineage
Explore OpenLineage, an open framework for data lineage metadata collection. Learn its purpose, data model, and integration with tools like Apache Airflow and Spark to enhance pipeline understanding and management.
Demystifying Apache Airflow - Separating Facts from Fiction
Unpack critiques of Apache Airflow, analyze its strengths and weaknesses, debunk myths, and compare with competitors. Make informed decisions about workflow management platforms.
The Silent Symphony: Keeping Airflow's CI/CD and Dev Tools in Tune
Explore Airflow's CI/CD and dev tools ecosystem. Learn how robust pipelines and efficient coding practices contribute to seamless integration, bug fixes, and developer empowerment.
Better ETL with Managed Airflow in Azure Data Factory
Discover Managed Airflow in ADF for complex data workflows. Learn about Apache Airflow, its benefits for ETL orchestration, and see practical demonstrations of its implementation in Azure.