Apache Spark vs Databricks - On-Premise Performance Comparison
Comparison of on-premise Databricks and open-source Spark implementations, showcasing 10x performance gains in cyber workloads and addressing challenges of air-gapped analytics solutions.
DataSketches: A Production Quality Sketching Library for Big Data Analysis
Explore sketching algorithms for efficient big data analysis, enabling fast approximate answers to complex queries and real-time processing of massive datasets in production environments.
Hyperspace - An Indexing Subsystem for Apache Spark
Explore Hyperspace, an indexing subsystem for Apache Spark that enhances query performance and reduces resource costs. Learn about its design, implementation, and benefits for data analytics on large datasets.
Operationalizing Apache Spark MLlib Models with SAS Model Manager
Learn to operationalize Apache Spark MLlib models using SAS Model Manager and Workflow Manager. Discover how to automate and govern the entire model lifecycle, from building to monitoring, accelerating deployment while maintaining control.
Data Microservices in Apache Spark Using Apache Arrow Flight
Explore efficient machine learning pipelines using Apache Arrow Flight in Spark, enabling faster data transport and simplified microservices for improved ML model production.
Data Science Across Data Sources with Apache Arrow - Accelerating Analytics and Interoperability
Explore Apache Arrow's columnar in-memory data representation for real-time analytics across distributed systems. Learn how it accelerates data processing and enables interoperability between various analytical tools and data sources.
AutoML Toolkit Deep Dive - Automating Feature Engineering and Model Optimization
Explore Databricks Labs' AutoML Toolkit for automating feature engineering, model selection, tuning, and deployment. Learn to accelerate machine learning workflows and enhance productivity.
Building a Data Platform for Mission-Critical Analytics on AWS with Databricks
Explore building enterprise data lakes on AWS with Databricks, complementing AWS strategy. Learn from HP's success in transforming business through unified data analytics and overcoming data lake challenges.
Delta from a Data Engineer's Perspective - Creating Robust Big Data Solutions
Explore a data engineer's challenges in creating robust Big Data solutions, covering Delta, Spark streaming, schema enforcement, and time travel functionality in data processing.
Running Apache Spark on Kubernetes - Best Practices and Pitfalls
Learn best practices and pitfalls for running Apache Spark on Kubernetes, covering setup, performance optimization, resource management, security, and future developments in this comprehensive overview.
Portable and Scalable Data Visualization Techniques for Apache Spark and Python Notebooks
Explore portable and scalable data visualization techniques for Apache Spark and Python notebooks, covering strategies, packages, headless rendering, animations, and big data integration.
Preventing Common Pitfalls in Production Streaming Jobs
Explore key strategies for optimizing streaming jobs in production, covering input parameters, stateful streaming, output configurations, and job modifications to ensure performance and fault tolerance.
Composable Data Processing with Apache Spark - Scaling Development and Error Handling
Learn about SIP, an extensible plugin framework for Apache Spark that enhances data processing efficiency, error handling, and scalability in Adobe's Experience Platform.
Re-Imagining Apache Spark Development - Tools for Productivity and Standardization
Exploring next-gen tools for Apache Spark development, focusing on productivity, code standardization, metadata, lineage, and CI/CD agility to overcome limitations of traditional ETL tools.
The Signal and the Noise - Lessons from 20 Years of Data Analysis
Nate Silver shares insights from 20 years of data analysis, focusing on election modeling, forecasting challenges, and lessons learned in interpreting complex data in competitive environments.
Introducing Apache Spark 3.0 - A Decade of Progress and Future Outlook
Apache Spark 3.0: Reflecting on a decade of progress and unveiling new features to enhance data processing accessibility, speed, and compatibility across various sources and environments.
Slowly Changing Dimensions Type 2 in Data Lakes - Tech Chat
Explore Slowly Changing Dimensions Type 2 in data lakes using Delta Lake, focusing on market segmentation for a climbing eCommerce site. Learn to track historical and current data effectively.
Managing the Complete Machine Learning Lifecycle with MLflow - Introduction to MLflow Tracking
Introduction to MLflow: tracking experiments, packaging projects, deploying models, and managing lifecycle. Learn to record, reproduce, and compare ML runs using MLflow's components and UI.
State of the Industry - Machine Learning Platforms and MLOps
Explore the state of MLOps and ML platforms with industry leaders, focusing on challenges, common components, and Databricks' approach to unified data analytics and machine learning.
Predictive Maintenance on IoT Data for Early Fault Detection with Delta Lake
Explore predictive maintenance using IoT data and Delta Lake for early fault detection. Learn about Industry 4.0 applications, data analytics, and machine learning techniques for improved efficiency and cost savings.
How to Build a Cloud Data Platform - Machine Learning and Business Intelligence - Part 4
Learn to leverage Delta Lake for dashboards and advanced analytics, enabling downstream teams with machine learning and business intelligence capabilities on a unified data platform.