Unit Tests - Overcoming Challenges in Structured Streaming
Discover effective strategies for unit testing streaming jobs, addressing challenges like long-running processes and third-party dependencies to improve data quality and code reliability.
Using Lakehouse for Life-Changing Outcomes in Manufacturing and Life Sciences
Explore how Lifeblood leverages data engineering, IoT, and manufacturing to innovate in clinical blood component processing, featuring scalable solutions, real-time insights, and data governance strategies.
Databricks Streaming - Project Lightspeed Goes Hyperspeed
Explore ultra low-latency data processing advancements, new stateful processing capabilities, and the future of streaming pipelines. Learn how to deliver real-time data for top AI use cases across industries.
Architectural Overview of Atlassian's Next-Generation Data Lakehouse
Discover Atlassian's innovative data lakehouse redesign, exploring account structures, workbench environments, Unity Catalog integration, and declarative approaches for enhanced data management and ML governance.
Practitioners Guide to Leveraging Serverless Compute on Databricks
Simplify workloads with serverless compute on Databricks: eliminate cluster setups, enhance start times, optimize resources, and improve reliability while maximizing cost-efficiency and performance.
Rise of the Medallion Mesh - Innovative Data Architecture Solutions
Explore innovative data architecture solutions combining Medallion and Data Mesh patterns to enhance productivity, democratize data access, and ensure scalable, efficient data management in large organizations.
What's Next for the Upcoming Apache Spark 4.0
Explore key enhancements in Apache Spark 4.0: Spark Connect, Structured Logging, PySpark updates, expanded SQL capabilities, improved connectivity, and real-time data processing advancements for optimized analytics.
Drastically Reducing Processing Costs with Delta Lake
Optimize data pipelines using Delta Lake features on Databricks. Learn techniques to address read-write amplification, improve performance, and reduce costs in large-scale data processing.
Semi-Structured Communication Data with Delta Lake and Microservices
Efficient storage and management of semi-structured communication data for eDiscovery using Delta Lake and microservices, enabling effective querying and analysis for legal processes.
Trek Bikes Accelerates Retail Analytics with Databricks
Trek Bikes optimizes retail analytics using Databricks, achieving near real-time data integration, 80-90% faster analytics, improved data accessibility, and cost reduction through innovative lakehouse architecture.
Udemy's Data and AI Journey - Migrating to Databricks for Enhanced Data Management
Explore Udemy's migration to Databricks, enhancing data democratization with Unity Catalog, DBSQL, and Delta Lake. Learn about cost control, data integration, and the transformative impact on innovation and analytics.
AT&T's Migration of Billions of Events Processing from Hadoop to Databricks
Explore AT&T's strategic shift from Hadoop to Databricks, resulting in 30% cost reduction and halved execution time for network data processing, showcasing cloud-native efficiency.
Optimizing MERGE Performance Using Liquid Clustering and Photon
Explore techniques to enhance MERGE operations using Liquid Clustering, Deletion Vectors, and Predictive I/O for Updates, improving performance and predictability in Delta Lake and Photon.
Lessons Learned from Migrating the Largest Immunization Analytics Platform
Insights from migrating a massive immunization registry to Databricks, covering data understanding, file structure impact, compute requirements, and workflow optimization for healthcare analytics.
Data Platform Modernization with a Data Mesh Architecture - Bradesco Case Study
Modernização da plataforma de dados do Bradesco usando arquitetura Data Mesh com Databricks, unificando ambientes heterogêneos e melhorando governança e eficiência para 75 milhões de clientes e 5 PB de dados.
Speed is Not All You Need for Data Processing
Exploring data processing frameworks beyond Pandas, challenging assumptions about speed, and emphasizing practical considerations for choosing tools that best serve data practitioners.
Fast Cars, Big Data - How Streaming Can Help Formula 1
Explore how data streaming technologies like Kafka and Apache Spark can revolutionize Formula 1 racing, enabling real-time analysis and visualization of car performance data from simulator to pit to HQ.
Deep Dive into Delta Lake and UniForm on Databricks
Explore Delta Lake's architecture, innovations, and use cases for reliable data lake management. Learn how it enhances performance, governance, and quality across various industries.
Mastering Software Operators for Data and AI - PostgreSQL, Apache Spark, and Kubeflow
Explore software operators for PostgreSQL, Apache Spark, and Kubeflow. Learn to simplify complex operations, streamline data processing, and navigate MLOps setups for optimal AI workload performance.
Transaction Log as the Heart of Delta Lake - Understanding Structure and Operations
Dive into the mechanics of Delta Lake's transaction log through hands-on PySpark exercises, exploring its structure, operations, and advantages over traditional data lakes.
From Zero to Data Hero - Take Raw Data to Curated Data in Under 90 Minutes
Discover how to efficiently manage, process, and clean data using low-code tools. Learn to connect Apache Spark, Databricks, and TimeXtender for rapid data curation and confident decision-making.