Migrating ETL Workflows to Apache Spark at Scale - Pinterest's Experience
Explore Pinterest's journey migrating ETL workflows to Apache Spark, covering challenges, solutions, and performance improvements. Learn about semantic gaps, thrift objects, and innovative Spark profiling techniques.
Delta Live Tables: Modern Software Engineering for ETL Pipelines
Learn how Delta Live Tables simplifies ETL, enhances data pipeline reliability, and automates tasks. Discover modern software engineering practices for efficient data transformation and improved analytics, data science, and ML outcomes.
Re-Imagining Apache Spark Development - Tools for Productivity and Standardization
Exploring next-gen tools for Apache Spark development, focusing on productivity, code standardization, metadata, lineage, and CI/CD agility to overcome limitations of traditional ETL tools.
Different Streaming Methods with Apache Spark and Kafka
Explore real-time analytics using Apache Spark and Kafka. Learn about streaming methods, overcoming technical challenges, and innovative solutions for handling billions of daily events in a scalable, cost-efficient manner.
Orchestration Made Easy with Databricks Workflows
Discover Databricks Workflows: a managed orchestration service for streamlined data pipelines. Learn to create, run, and monitor reliable workflows, implement recovery actions, and explore new features for efficient data management.
Healthcare Claim Reimbursement Using Apache Spark - Optum's System Transformation
Optimizing healthcare claim processing with Apache Spark: Improved performance, reduced costs, and enhanced capabilities for hospitals' revenue recovery through efficient data processing and analysis.
Building the Petcare Data Platform with Delta Lake and Spark ETL Pipeline
Explore Mars Petcare's data platform using Delta Lake and Spark ETL pipeline 'Kyte'. Learn about advantages over Azure Data Factory and leveraging Delta Lake for ETL configurations and data science.
Using Apache Spark for Predicting Degrading and Failing Parts in Aviation
Leveraging Apache Spark for predictive maintenance in aviation, enabling efficient data processing, anomaly detection, and component health scoring to improve fleet readiness and prevent failures.
Faster Data Integration Pipeline Execution Using Spark-Jobserver
Optimize data integration pipelines using Spark-Jobserver for faster execution, context reuse, and efficient resource management. Learn configuration, security, and API integration for improved performance.
Accelerating Spark Workloads in a Mesos Environment with Alluxio
Learn how Alluxio enhances Spark and Mesos performance by providing memory-speed data access across disparate storage systems, optimizing big data processing and enabling new workloads.
Building Modern Data Streaming Apps with Open Source
Discover best practices for building data streaming applications using open-source frameworks. Learn to orchestrate streams, perform ETL, enhance events, and make continuous queries with tools like Apache NiFi, Pulsar, Spark, and Flink SQL.
Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
Learn how Alluxio enhances Spark and Mesos performance by providing memory-speed data access across diverse storage systems, optimizing big data processing and enabling new workloads.
Building Robust Streaming Data Pipelines with Apache Spark
Learn to build robust streaming data pipelines using Apache Spark, Kafka, and Camel. Explore ETL challenges, data management, and lessons from running these technologies in Docker.
Streamlining Data Preparation with Pydantic - A 25-Minute Guide
Explore Pydantic's power in data validation and cleaning for Python projects. Learn to ensure consistency, handle missing values, and streamline data preparation in just 25 minutes.
Power Query in 20 Minutes - How it Works
Explore Power Query's internals and gain a deeper understanding of this increasingly prevalent tool in Power BI and Fabric technologies.
ETL Observability: Azure to Snowflake - Monitoring with Splunk and Power BI
Explore ETL monitoring from Azure to Snowflake using ADF, Splunk, and Power BI. Learn data lineage and observability techniques for efficient pipeline management.
Data Engineering with Databricks - Leveraging the Lakehouse Platform for ETL Pipelines
Leverage Databricks Lakehouse Platform to build ETL pipelines, use Delta Live Tables, and orchestrate tasks with Workflows for efficient data processing.
Implementing Change Data Capture (CDC) and Slowly Changing Dimensions (SCD) Using Spark and Hive
Learn to implement Change Data Capture (CDC) and Slowly Changing Dimension (SCD) patterns in Hive using Spark, with a step-by-step guide for efficient data management and analysis.
How to Build Data Pipelines for ML Projects with Python Code
Learn to build data pipelines for machine learning projects using Python. Explore ETL vs ELT, extraction, transformation, loading, and orchestration. Includes practical example of processing YouTube video transcripts.
Automating Data Pipelines with Python and GitHub Actions - Code Walkthrough
Automate data pipelines using Python and GitHub Actions. Learn to create ETL scripts, set up repositories, configure workflows, and implement ML applications for efficient data processing.
Challenges and Solutions for Building Real-time Data Warehousing with Apache Flink, Apache Hive and Apache Iceberg
Explore solutions for real-time data warehousing using Apache Flink, Hive, and Iceberg. Learn to handle challenges like late events and dirty data routing in enterprise-level ETL migration.