Dataproc Persistent History Server: Up and Running
Learn to access and troubleshoot Hadoop and Spark application logs using Persistent History Server for ephemeral clusters and Dataproc Serverless, enhancing job monitoring and exploration capabilities.
Productionizing Health Insurance Appeal Generation - AI in Production
Explore fine-tuning models for health insurance appeals, cloud resource utilization, and on-premises Kubernetes deployment. Learn about unexpected challenges in server setup and graphics card installation.
Apache Kafka Real-Time Personalization Architecture
Explore real-time personalization architecture using Apache Kafka and Spark for advanced content recommendation systems in data engineering and AI.
Inside Spark and Kafka: Data Processing and Real-Time Analytics - Episode 1
Explore Charmed Spark and Kafka: Canonical's solutions for Apache Spark and Kafka on Kubernetes. Discover industry trends, real-time applications, and data processing innovations.
Towards Multi-Table Transactions in Delta Lake
Explore managed-commits in Delta Lake: a new protocol enabling multi-table transactions, reliable commit semantics, and enhanced data governance for cloud storage-based data lakes.
State Reader API: The New "Statestore" Data Source for Structured Streaming
Explore Databricks' State Reader API for accessing and analyzing Structured Streaming's internal state data, aiding development and troubleshooting of stateful workloads in Apache Spark 4.0.0.
Delta Lake for Polyglot Data and Machine Learning Workloads
Explore Delta Lake's power for data streams, analysis, and ML workloads. Learn about data quality monitoring, efficient job management, and leveraging Delta for auto-ML models and Spark Structured Streaming.
Delta Tensor: Efficient Vector and Tensor Storage in Delta Lake
Efficient tensor storage solution for machine learning workflows, using chunking and sparse encoding to reduce IO costs and improve storage efficiency in cloud-native Lakehouse environments.
Architecture Analysis for ETL Processing - CPU vs GPU
Explore GPU acceleration for ETL operations, comparing CPU and GPU architectures. Learn about performance gains in database operations and SQL query acceleration using GPUs.
Efficient Near Real-Time Event Ingestion using DLT - Insights and Lessons
Explore Nextdoor's shift to near-real-time streaming with DLT, enabling quick event querying for analysis and monitoring while reducing costs. Learn about challenges, lessons, and optimization techniques.
Shaping the Future of Data in India's Premier Fashion E-Commerce Platform
Optimizing petabyte-scale clickstream data analysis for fashion e-commerce: Migration to Delta Lake enhances scalability, cost-efficiency, and performance in retail analytics.
Data Driven: The Journey to Bring Real-Time ERP Data to a Lakehouse
Michelin's journey to real-time ERP data integration in a lakehouse, leveraging tools like Kafka, Oracle Golden Gate, and Databricks for faster inventory updates and enhanced business analytics.
Delta Kernel - Anatomy of a Delta Lake Connector
Explore Delta Kernel project for simplified Delta Lake connector creation. Learn architectural decisions, engineering considerations, and community usage in this informative session.
Unlocking the Lakehouse with Efficient Data Pipelines - Capital One's Journey
Explore Capital One's data-driven transformation using Databricks Lakehouse, featuring efficient data pipelines, Delta Lake, and best practices for building a centralized data ecosystem.
Accelerating Adoption of Data Lake for Streaming and Machine Learning Use Cases
Discover tools and features for efficient Databricks adoption in ML and streaming, focusing on automation and operational efficiency to accelerate implementation of Delta and Spark compute workloads.
Introducing the New Python Data Source API for Apache Spark
Simplify big data processing with Python in Apache Spark™. Learn key features of the new Data Source API, enabling custom data source implementation without Scala knowledge. Discover benefits for Python developers in Spark ecosystem.
Processing a Trillion Rows Per Day with Delta Lake at Adobe
Scaling data processing at Adobe: managing thousands of Delta tables, handling terabytes daily, and powering marketing scenarios across platforms. Insights on writer scaling, concurrency control, and maintenance.
Streaming Cross-Sectional Data Visualization with Perspective and Spark
Explore real-time data visualization using Perspective and Spark Streaming. Learn to build high-performance, interactive applications for analyzing streaming data in various industries.
Honeywell Intelligrated's IoT Streaming Lakehouse for Warehouse Automation
Explore Honeywell Intelligrated's journey in launching IoT streaming services for warehouse automation, utilizing Databricks Lakehouse for data governance, processing, and AI-driven solutions.
Delta Merge Optimizations with Jodie Helpers
Explore Delta Merge optimization strategies, including data skipping and Change Data Feed implications, to enhance performance and handle failure scenarios in data lakehouse environments.
Power Up Your Lakehouse with Git Semantics and Delta Lake
Enhance data versioning in lakehouses using Delta Lake and lakeFS, enabling git-like semantics for improved time travel, data quality, and operational economics.