Designing and Implementing a Real-time Data Lake with Dynamically Changing Schema
Learn to build a robust streaming ETL pipeline that handles dynamically changing schemas and new event types without downtime, using Delta Lake and Databricks for real-time data lake implementation.
Modern ETL Pipelines with Change Data Capture - Building Resilient Data Streams
Discover how GetYourGuide built a resilient ETL pipeline using Debezium, Kafka, Spark, and Airflow. Learn to handle schema changes automatically, stream data from SQL databases, and refresh data lakes multiple times daily for fresher insights.
Realtime Stock Market Anomaly Detection Using Machine Learning - End-to-End Data Engineering Project
Develop an end-to-end data engineering project for real-time stock market anomaly detection using Quix Streams, Redpanda, and Docker. Learn to build a complete data pipeline and deploy an advanced ML model.
Delta Live Tables - Building Reliable ETL Pipelines with Azure Databricks
Build fast, reliable, and scalable ETL pipelines using Delta Live Tables on Azure Databricks. Learn declarative approaches and see demos of simple and complex streaming pipelines.
Lessons Learned from the Migration to Apache Airflow
Explore Apache Airflow for orchestrating complex workflows, with insights on ETL pipelines, Spark integration, and best practices for managing big data infrastructure.
Product Recommendation System on Kubernetes Using Argo Workflows
Learn to build a product recommendation system using Argo Workflows on Kubernetes. Explore DataOps, pipelines, and tools like Minio and Apache Spark for efficient data processing and analysis.
Improving Broadcast Joins in Apache Spark SQL
Discover techniques to enhance broadcast joins in Apache Spark SQL, including executor-side broadcasting and memory optimization, for improved performance in large-scale ETL pipelines and complex data processing tasks.
Breakdown New Data Silos in Generative AI
Explore strategies to overcome data silos in generative AI, enhancing model performance through improved data access, discovery, and integration across departments and cloud platforms.
Building a Unified Cancer Immunotherapy Data Library
Explore a unified cancer immunotherapy data library leveraging Datomic for efficient storage, querying, and collaboration. Learn about CANDEL's innovative approach to managing complex research data.
Too Big for DAG Factories?
Learn to scale data workflows using Apache Airflow, covering dynamic DAG generation, automation, and best practices for managing big data pipelines. Suitable for beginners and intermediate developers.
Building Modern Data Pipelines with Spark on Azure HDInsight
Explore common patterns for building end-to-end data analytics pipelines using Apache Spark on Azure HDInsight, covering architecture, integration points, and various pipeline options.
How Bosch Power Tools Leverages User Data and AI for Consumer Feedback and ETL Pipeline Management
Discover how Bosch Power Tools harnesses AI and data analytics to process consumer feedback, maintain language consistency, and optimize ETL pipelines for enhanced customer satisfaction.
Realtime Data Streaming - End-to-End Data Engineering Project
Build a real-time data streaming pipeline using Apache Airflow, Kafka, Spark, and Cassandra. Learn end-to-end data engineering, from ingestion to processing and storage, all containerized with Docker.