Apache Spark, Data Science, and Machine Learning Advances - Data + AI Summit 2021 Keynote
Explore AI advancements, data science, MLOps, and Databricks Lakehouse Platform innovations. Featuring industry leaders and Bill Nye, this keynote covers MLFlow updates, AutoML, and practical machine learning solutions.
Boost Delta Lake Performance with Data Skipping and Z-Order
Explore data skipping and Z-ordering techniques to enhance Delta Lake performance, focusing on optimized partitioning strategies for improved read and write efficiency in large-scale data lakes.
How Apache Spark 3.0 and Delta Lake Enhance Data Lake Reliability
Discover performance enhancements in Apache Spark 3.0 and Delta Lake's data reliability features. Learn about Adaptive Query Execution, Dynamic Partition Pruning, and handling skewed queries for improved big data processing.
Engagement Activity Delta Lake for Einstein Analytics and Sales Cloud Einstein
Explore how Salesforce built an engagement activity platform using Delta Lake to support Einstein Analytics and Sales Cloud Einstein, covering data ingestion, incremental reads, and mutation handling.
MLflow Model Serving - From Training to Production
Comprehensive overview of MLflow Model Serving, covering offline and online scoring, deployment options, and integration with Databricks, with practical examples and recent features.
Developing Scalable Machine Learning Pipelines for Gaming Industry
Explore scalable ML pipelines in gaming, leveraging Apache Spark for data-driven product development and improved user acquisition through advanced LTV modeling.
Accelerating MLOps with Apache Spark and SingleStore
Learn to accelerate MLOps using Apache Spark and SingleStore. Discover how to build fast analytical applications, manage supervised learning models, and scale AI/ML operations for data science success.
Designing and Implementing a Real-time Data Lake with Dynamically Changing Schema
Learn to build a robust streaming ETL pipeline that handles dynamically changing schemas and new event types without downtime, using Delta Lake and Databricks for real-time data lake implementation.
Cost Efficiency Strategies for Managed Apache Spark Service with Databricks
Explore cost-efficient strategies for Azure Databricks, including resource management, tier selection, and best practices for optimizing Apache Spark workloads in a managed cloud environment.
Migrating Airflow-Based Apache Spark Jobs to Kubernetes - The Native Way
Learn to migrate Airflow-based Apache Spark jobs to Kubernetes, reducing costs and improving stability. Explore challenges, best practices, and native Airflow integration for seamless transition from AWS EMR to K8s.
Scaling Machine Learning Feature Engineering in Apache Spark at Facebook
Explore advanced techniques for scaling ML feature engineering in Apache Spark, including Feature Injection and Feature Reaping, with optimizations for improved performance and efficiency at Facebook's scale.
Project Zen - Improving Apache Spark for Python Users
Discover improvements in Apache Spark for Python users, including redesigned documentation, type hints, PyPI distribution options, and enhanced PySpark features for a more Pythonic experience.
Getting Started with Apache Spark on Kubernetes
Learn to deploy and optimize Apache Spark on Kubernetes with hands-on demos, covering environment setup, application sizing, performance tuning, and monitoring for efficient data pipeline management.
Delta Lake: Optimizing Merge Operations
Explore Delta Lake merge optimization techniques, including inner and full outer joins, partition pruning, and file pruning. Learn tips for handling large merges efficiently.
Building a Cloud Data Lake with Databricks and AWS - Best Practices and Implementation
Explore building a cloud data lake using Databricks and AWS. Learn about data lake architecture, AWS services, Delta Lake for reliability and performance, and best practices for implementation.
DBT Using Databricks and Delta
Learn to leverage Data Build Tool (DBT) with Databricks and Delta for efficient data lake management, including SQL-first approach, upsert capabilities, and Azure integration for enhanced observability.
Stateful Streaming with Apache Spark - Updating Decision Logic at Runtime
Learn how to implement stateful streaming with Spark, focusing on updating decision logic at runtime for real-time offer personalization in loyalty programs like PAYBACK.
Leveraging Apache Spark and Delta Lake for Efficient Data Encryption at Scale
Efficient data encryption solution for privacy compliance using Apache Spark and Delta Lake. Automates consumer deletion requests, enhances security, and maintains data integrity while ensuring accessibility when needed.
Common Strategies for Improving Performance on Delta Lakehouse
Explore techniques to enhance Delta Lakehouse query performance, including data skipping, optimizing Z-Order, SQL partitions, and Delta Cache. Learn to analyze and tune queries for better results.
Building an MLOps Platform with MLflow for Rapid Model Productionalization
Discover how DataSentics built an MLOps platform around MLflow to streamline model productionalization, reducing deployment time to minutes. Learn about real-life experiences, platform demo, and lessons learned.
Diving into Delta Lake: Understanding the Transaction Log
Explore Delta Lake's transaction log: its structure, functionality, and role in ACID transactions, metadata handling, time travel, and concurrent operations. Gain insights into this core component.