Introduction to Databricks Lakehouse Platform Architecture and Security
Explore Databricks Lakehouse Platform's architecture, focusing on data reliability, Delta Lake features, and Photon's performance enhancements for efficient data management and analytics.
How to Make Apache Spark on Kubernetes Run Reliably on Spot Instances
Learn to run Apache Spark on Kubernetes using spot instances for cost savings. Discover best practices, code examples, and techniques to handle spot interruptions while maintaining reliability and performance.
Low-Code Machine Learning on Databricks with AutoML
Discover how to build, train, and deploy ML models using Databricks AutoML and low-code tools, empowering users of all expertise levels to create predictive analytics solutions.
Improving Apache Spark Application Processing Time - Configuration and Optimization Techniques
Optimize Spark streaming app performance through configurations, code improvements, and custom data source implementation for processing 700 MB/s of compressed data with strict KPIs.
Implementing End-to-End Demand Forecasting with Databricks and MLflow
Discover how to implement a scalable demand forecasting solution using Databricks and MLflow, improving retail efficiency and reducing food waste through accurate production and delivery planning.
Orchestration Made Easy with Databricks Workflows
Discover Databricks Workflows: a managed orchestration service for streamlined data pipelines. Learn to create, run, and monitor reliable workflows, implement recovery actions, and explore new features for efficient data management.
Administrator Best Practices for Scaling and Securing Databricks Deployments
Optimize Databricks administration for enterprise-scale deployments. Learn best practices, new capabilities, and customer insights for efficient onboarding, security, and workspace management.
Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors
Lessons learned from running massive Spark workloads on Kubernetes at Apple, covering orchestration, resource management, monitoring, and optimization for scalability and performance.
Securing Databricks on AWS Using PrivateLink - Deployment and Automation
Learn to secure Databricks on AWS using PrivateLink, covering architecture, setup, and automation. Gain insights on minimizing public internet data transfers for enhanced security and cost-efficiency.
PySpark in Apache Spark 3.3 and Beyond
Explore PySpark's evolution in Apache Spark 3.3, including improved Pandas API, enhanced productivity features, and future developments. Learn about new functionalities, usability improvements, and performance enhancements.
Data Warehousing on the Lakehouse - Unifying Data, AI, and Analytics
Unify data, AI, and analytics with the lakehouse architecture. Learn success stories, use cases, and best practices for ingesting, storing, and governing data at scale for SQL and BI workloads using Databricks SQL.
SIGCOMM 2020 - Reducto - On-Camera Filtering for Resource-Efficient Real-Time Video Analytics
Explore real-time video analytics optimization with Reducto, a filtering technique for efficient on-camera processing. Learn about system design, performance results, and practical applications.
Apache Arrow and the Future of Data Frames with Wes McKinney
Explore Apache Arrow's columnar data standard and its impact on data frame libraries, analytics systems, and cross-language collaboration in data processing and analysis.
Sketching Big Data
Explore big data sketching techniques with Harvard's Jelani Nelson, covering approximate counting, frequent items, and graph streaming in this insightful lecture.
Migrating Pinterest Apache Spark Clusters from HDFS to S3
Explore Pinterest's migration of Apache Spark clusters from HDFS to S3, addressing technical challenges, optimizing performance, and achieving a smooth transition process.
Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator
Learn to deploy Scala Spark jobs on Kubernetes using Helm and Spark Operator. Live coding demo shows scalable, flexible deployments with minimal custom configurations for stress-free implementation.
User Defined Aggregation in Apache Spark - From Challenges to Improvements
Explore Apache Spark's User Defined Aggregate Functions (UDAFs), their evolution, and improvements in Spark 3.0. Learn to create custom aggregations, enhance performance, and contribute to the Spark community.
Best Practices for Building Robust Data Platforms with Apache Spark and Delta
Learn best practices for building robust data platforms using Apache Spark and Delta. Gain insights on optimizing performance, scaling, security, and cost-effectiveness in big data architectures from real-world experiences.
Patterns and Operational Insights for Large-Scale Delta Lake Workloads
Explore effective Delta Lake patterns, optimization techniques, and operational insights for managing large-scale data workloads on Databricks, including streaming ETL, data enrichment, and analytics.
Delta Lake 0.7.0 and Spark 3.0 - Ask Me Anything Session
Illuminating AMA session with Apache Spark and Delta Lake experts. Explore Spark 3.0 and Delta Lake's latest features, use cases, and development insights. Perfect for data engineers seeking cutting-edge knowledge.
Koalas - Scaling Pandas API on Apache Spark
Learn to scale pandas operations to big data using Koalas, an open-source project implementing the pandas API on Apache Spark. Transition seamlessly from single-machine to distributed environments for large-scale data science.