Optimizing Apache Spark and SQL for Improved Performance
Enhance Apache Spark and SQL performance through optimization techniques. Learn strategies for improved efficiency in data processing and analysis.
Working with Data Using Azure Functions
Explore Azure Functions for data interactions with Azure SQL, Cosmos DB, and Event Hubs. Learn to build lightweight, code-first APIs, integrations, ETL, and maintenance routines.
Design Patterns for Data Pipelines Beyond ETL
Discover essential design patterns for managing complex data pipelines in Python, focusing on advanced transformation techniques and Django integration for robust ETL workflows.
Qlik Talend Cloud - AI-Assisted Data Engineering Overview and Demo
Discover how Qlik Talend Cloud revolutionizes data engineering with AI-assisted automation, no-code options, and advanced data integration capabilities for building robust, trustworthy data pipelines.
Introduction to Data Build Tool (DBT) - Features and Project Creation
Master data transformation techniques with DBT (Data Build Tool) to effectively analyze and engineer warehouse data through hands-on project creation and fundamental features.
The Intelligent Future of Data Warehouses and Delta Lake - Structured Streaming with PySPARK
Explore the evolution of data warehousing from traditional ETL to modern Lakehouse architecture, focusing on Delta Lake, Structured Streaming, and PySpark implementation with practical code examples.
DocETL: AI Agents for Complex Document Processing and Data Transformation
Discover how DocETL framework leverages AI agents and LLMs to transform complex document processing, featuring innovative operators and optimization techniques for enhanced data extraction and analysis.
Practical Kimball Data Patterns for Machine Learning Data Warehousing
Discover how dimensional modeling and Kimball patterns can streamline machine learning projects, reduce data preparation time, and improve ML model development through practical warehouse design techniques.
Airflow Distributed Workloads - ETL Architecture with AWS Lambda and Multi-Threaded Python Scripts
Master distributed workload processing with Airflow, AWS Lambda, and multithreaded Python scripts while exploring ETL architecture for third-party API integration at scale.
A Year in Production with Machine Learning on the BEAM
Explore insights on transitioning machine learning pipelines to Elixir, leveraging the Nx ecosystem for streamlined development and swift experimentation in production environments.
Working with BigQuery and DataForm - Streamlining Data Management and Transformation
Streamline data management with BigQuery and DataForm. Learn setup, optimization, and automation for efficient, scalable data pipelines and transformations.
Introduction to Data Engineering - Best Practices and Production-Level Infrastructure
Learn best practices for scaling data science projects from local to production, covering infrastructure, reliability, and real-world applications in data engineering.
Demystifying Apache Airflow - Separating Facts from Fiction
Unpack critiques of Apache Airflow, analyze its strengths and weaknesses, debunk myths, and compare with competitors. Make informed decisions about workflow management platforms.
Azure Data Engineering Tools: Choosing Between Data Factory, Databricks, and Synapse Analytics
Explore Azure data engineering tools: Data Factory, Databricks, and Synapse Analytics. Learn their strengths, use cases, and how to choose the right one for your project.
7 Things I Wish I'd Known Before Building a 60TB Data Warehouse on a Dedicated Pool
Discover key insights and lessons learned from building a 60Tb data warehouse on Azure Dedicated Pool. Explore challenges, solutions, and best practices for large-scale data management.
Deep Skydiving into Data Factory in Microsoft Fabric
Explore advanced Data Factory scenarios in Microsoft Fabric, leveraging Data Pipelines and Dataflows Gen2 for scalable implementations.
Better ETL with Managed Airflow in Azure Data Factory
Discover Managed Airflow in ADF for complex data workflows. Learn about Apache Airflow, its benefits for ETL orchestration, and see practical demonstrations of its implementation in Azure.
Your First Flight with Data Factory in Microsoft Fabric
Explore Data Factory in Microsoft Fabric through a rapid tour, understanding its value for data integration in organizations.
Deep Skydiving into Data Factory in Microsoft Fabric
Explore advanced Data Factory scenarios in Microsoft Fabric, leveraging Data Pipelines and Dataflows Gen2 for scalable implementations.
Star Schema ALL the Things! But Why? - Practical Design Decisions in Data Modeling
Explore practical star schema design, from source to model to report. Learn best practices, test methodologies, and assess performance impacts in data modeling.
From SSIS to Synapse to Fabric - A Data Architect's Case Study
Explore a real-world EDW automation framework migration journey from SSIS to Synapse to Fabric. Learn migration processes, lessons, improvements, and unit testing strategies for data architects.