Securely Running Databricks on AWS for Public Sector and Regulated Industries
Explore secure Databricks deployment on AWS for regulated industries, covering compliance, encryption, data governance, and workflow orchestration for stringent security requirements.
Dependency Management in Spark Connect - Simple, Isolated, Powerful
Explore Spark Connect's new session-based dependency management system for dynamic updates and isolation in distributed computing, with practical examples for Python and Scala applications.
Unity Catalog Lakeguard - Data Governance for Multi-User Apache Spark Clusters
Explore Databricks' innovations in data governance for lakehouses, enabling efficient multi-user workloads with Shared Clusters and Unity Catalog while reducing costs and operational complexity.
Seamless Flow: Evolving From Batch to Streaming Data Flows Using DLT
Explore how 84.51° leverages Delta Live Tables to evolve data flows from batch to streaming, combining real-time and historical data for enhanced retail insights and customer-centric strategies.
Delivering Portability to Open Data Lakes with Delta Lake UniForm
Explore Delta Lake UniForm's solution for enhancing data lake reliability, performance, and governance across multiple environments. Learn best practices for scaling and optimizing cloud-based data lakes.
Apache Spark Processing with AWS EMR - Data Engineering Project
Master AWS EMR for data processing: set up, submit Spark jobs, visualize on Hadoop UI, and leverage cloud computing for efficient big data handling.
Next-Gen Data Modeling, Integrity, and Governance with YODA - Insights from Yotpo and ZipRecruiter
Explore next-gen data modeling strategies for large-scale systems. Learn about YODA, an internal tool combining developer experience, DBT, Databricks, and more, to address data standardization and governance challenges.
Optimizing Apache Spark and SQL for Improved Performance
Enhance Apache Spark and SQL performance through optimization techniques. Learn strategies for improved efficiency in data processing and analysis.
The Sparkling Solution that Scales to IoT
Exploring Spark and Scala for IoT big data analytics, with demonstrations of infrastructure setup on Apache Mesos and coding techniques for Spark streaming, AKKA, and SparkSQL.
The Evolution of Hadoop at Spotify Through Failures and Pain
Explore Spotify's Hadoop journey, from office closet to massive cluster, through failures and lessons learned. Gain insights on data infrastructure evolution and overcoming obstacles in large-scale systems.
The Data Lakehouse for Streaming Data - A Talk for Everyone Who Loves Data
Explore data lakehouses: combining data lakes and warehouses for reliable, flexible data management. Learn about open-source projects, streaming data support, and see a live Twitter sentiment analysis demo.
The Data Lakehouse - A Tech Talk for Everyone Who Loves Data
Explore data lakehouses: combining data lakes and warehouses for reliability, governance, and flexibility. Learn about open-source projects, streaming data, and future improvements through live demos and code examples.
Apache Spark Beyond Shuffling - Why it isn't Magic - but also where there is some really cool Magic
Apache Spark performance deep dive: scaling challenges, RDD operations, key skew issues, and optimization techniques for distributed data processing. Insights from an IBM Spark engineer.
Speed is Not All You Need for Data Processing
Exploring data processing frameworks beyond Pandas, challenging assumptions about speed, and emphasizing practical considerations for choosing tools that best serve data practitioners.
Open Data Foundations Across Hudi, Iceberg and Delta - OneTable Integration
Explore OneTable's innovative approach to optimizing data across Hudi, Iceberg, and Delta Lake. Learn how it enhances workloads for various query engines and streamlines data analysis.
High Performance Scalable Support for Big Data Stacks with MPI - Leveraging MPI4Spark and MPI4Dask
Discover how MPI4Spark and MPI4Dask enhance big data frameworks by leveraging high-speed networks and HPC systems for improved performance and scalability in distributed computing environments.
Data Governance and Data Lake Implementation with Delta Lake
Discover how to effectively implement Data Lake with Data Governance through Microsoft's key services, featuring real SK case studies and insights on ML Studio and Power BI integration.
The Intelligent Future of Data Warehouses and Delta Lake - Structured Streaming with PySPARK
Explore the evolution of data warehousing from traditional ETL to modern Lakehouse architecture, focusing on Delta Lake, Structured Streaming, and PySpark implementation with practical code examples.
Overview of the Databricks Platform and Data Lakehouse Architecture
Discover how Databricks revolutionizes data processing by combining data warehouse and lake architectures through Delta Lake, Apache Spark, MLFlow, and Koalas frameworks for scalable data engineering and analytics.
Build ML-Enhanced Event Streaming Apps with Python Microservices - Conf42 Python 2023
Develop ML-enhanced event streaming applications using Python microservices, exploring real-time data processing and machine learning integration.
Spark at Scale: Engineering Strategies for Data Science Workflows
Optimize PySpark applications and slash runtimes using engineering strategies for data science workflows, illustrated through a case study on in-store availability reporting.