Introduction to Databricks Delta Live Tables
Accelerate data pipeline development with Databricks Delta Live Tables. Learn setup, ingestion, table creation, pipeline execution, and data validation for faster, more reliable projects.
Delta Live Tables - Building Reliable ETL Pipelines with Azure Databricks
Build fast, reliable, and scalable ETL pipelines using Delta Live Tables on Azure Databricks. Learn declarative approaches and see demos of simple and complex streaming pipelines.
Optimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes
Explore optimal approaches for real-time machine learning using Apache Spark on Kubernetes, covering best practices for efficient model training, deployment, and serving in low-latency, high-scalability environments.
Orchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
Explore integrated use of Apache Toree, YuniKorn, Spark, and Airflow for efficient, scalable data pipelines. Learn to leverage these projects for optimized data processing and workflow orchestration.
How to Pick the Best Data Transformation Strategy
Explore tools for combining data from multiple sources, including Azure Synapse, Spark, and Power BI Data Flows. Learn to select the optimal tool for your data environment.
Declarative ETL Pipelines with Delta Live Tables - Modern Software Engineering for Data Analysts and Engineers
Modernize ETL pipelines using Delta Live Tables for efficient data management. Learn development, production, and best practices for reliable, testable, and scalable data processing.
Obtain, Distribute and Use Temporary Credentials Automatically in Apache Spark and Apache Flink
Learn to securely scale distributed data processing with automatic credential management in Apache Spark and Apache Flink. Explore common authentication challenges and solutions for high-load scenarios.
Building a Fully-Managed OSS-Compatible Lakehouse with BigLake Managed Tables
Discover how BigLake Managed Tables enhances open lakehouse formats by combining BigQuery infrastructure with OSS compatibility for improved data management, access control, and high-throughput streaming.
How Data Hub Flies Above the Clouds - Building a Centralized Social Media Analytics Platform
Discover how to build a centralized data platform using Azure services, integrating social media data through Data Lake, Databricks, and Power BI for comprehensive analytics and reporting.
PySpark Best Practices
Learn best practices for PySpark, including dependency management, code structuring, testing, and performance optimization. Gain insights on writing efficient and scalable Python applications using Spark's distributed computing model.
The Human Factors of Running a Data Driven Company
Insights on building a data-driven culture at QuizUp, focusing on team structure, communication, and leveraging data across the organization to inform product decisions.
DataSource V2 and Cassandra - A New Era of Speed and Flexibility
Explore DataSource V2 in Spark Cassandra Connector: enhanced speed, flexibility, and usability. Learn about Spark's understanding of Cassandra clustering and direct catalogue manipulation for improved data integration and analysis.
From Python to PySpark and Back Again - Unifying Single-Host and Distributed Deep Learning with Maggy
Unifying single-host and distributed deep learning with Maggy framework, enabling seamless transition from laptop to cluster for efficient model development and training using TensorFlow and PySpark.
Architecting for Data Quality in the Lakehouse with Delta Lake and PySpark
Learn techniques for ensuring data quality in lakehouses using Delta Lake and PySpark. Explore data observability, reliability lifecycle, and practical implementation of quality checks across metadata, storage, and query layers.
Sound Data Engineering in Rust - From Bits to DataFrames
Explore Rust for efficient data engineering, from low-level bit manipulation to high-performance DataFrames using Apache Arrow. Learn optimization techniques and benchmarks for improved data processing.
Azure Databricks - Azure + Spark + Machine Learning
Explore Azure Databricks: Combine Azure, Spark, and machine learning. Learn key features through demonstrations and understand its advantages for big data processing and analytics.
Azure Databricks with R - Deep Dive
Explore Azure Databricks using R for big data and machine learning. Learn Spark architecture, APIs, workspaces, and data manipulation techniques for efficient cloud-based analytics.
Azure Databricks - A Brief Introduction
Explore Azure Databricks, a Big Data service based on Apache Spark. Learn about its features, security, and integration with Python, R, SQL, and machine learning frameworks.
Localization with NLP - Global Empire-Building for Fun and Profit
Explore scalable localization techniques using NLP, Docker, and open-source tools. Learn about continuous deployment, live syncing, content generation, and emotional response for global product expansion.
Fast, Cheap and Easy Data Ingestion with AWS Lambda and Delta Lake
Explore event-driven data ingestion using AWS Lambda with Delta Lake. Learn to process, update, and query Delta tables efficiently in Python and Rust for a more responsive data platform.
Data Engineering with Rust and Delta Lake
Explore data engineering with Rust, leveraging Delta Lake, data fusion, and arrow for high-performance pipelines. Learn to ingest, enrich, manage, and export data using Rust's powerful capabilities.