Geospatial Data Analysis in Apache Spark - Challenges and Solutions
Explore geospatial data challenges in Apache Spark, comparing libraries, and learning best practices for data ingestion, storage, and retrieval using spatial indexing techniques.
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
Explore Spark SQL's Adaptive Query Execution framework, which optimizes query plans at runtime using real-time statistics for improved performance in data processing and analysis.
The Apache Spark File Format Ecosystem - Optimizing Storage for Performance
Explore key file formats in Spark ecosystem: Parquet, ORC, Avro. Learn core concepts, optimization techniques, and how to select the right format for performant data processing and AI/ML workflows.
Deep Dive into GPU Support in Apache Spark 3.x - Accelerator-Aware Scheduling and RAPIDS Plugin
Explore GPU acceleration in Apache Spark 3.x, covering task scheduling, columnar processing, and the RAPIDS plugin for enhanced ETL, ML, and DL performance without code changes.
Building a Real-Time Feature Store for Machine Learning at iFood
Discover how iFood built a real-time feature store using Databricks and Spark Structured Streaming to process event streams, store data, and develop reliable code for efficient machine learning model predictions.
Using Apache Spark and Differential Privacy for 2020 Census Data Protection
Explore how Apache Spark and differential privacy protect 2020 Census data, balancing accuracy and confidentiality. Learn about the innovative TopDown Algorithm and custom monitoring systems.
Fine-Tuning and Enhancing Performance of Apache Spark Jobs
Explore best practices for optimizing Apache Spark jobs, including resource allocation, parameter tuning, and performance enhancement techniques. Learn to identify bottlenecks and implement solutions for improved efficiency.
Real-Time Forecasting at Scale Using Delta Lake and Delta Caching
Explore real-time forecasting at scale using Delta Lake and Delta Caching. Learn efficient data sampling, storage, and caching techniques for handling massive datasets and achieving rapid forecast response times.
Building a Streaming Microservice Architecture with Apache Spark Structured Streaming
Explore building a scalable, fault-tolerant streaming microservice architecture using Apache Spark, Kafka, gRPC, and Delta Lake for real-time data processing and reliable insights.
Productionizing Machine Learning with Apache Spark, MLflow and ONNX - Cloud Deployment Using SQL Server
Explore productionizing machine learning using Apache Spark, MLflow, and ONNX with SQL Server. Learn model management, lifecycle orchestration, and leveraging SQL Server for ML model storage and deployment.
Building the Petcare Data Platform with Delta Lake and Spark ETL Pipeline
Explore Mars Petcare's data platform using Delta Lake and Spark ETL pipeline 'Kyte'. Learn about advantages over Azure Data Factory and leveraging Delta Lake for ETL configurations and data science.
Using Apache Spark for Predicting Degrading and Failing Parts in Aviation
Leveraging Apache Spark for predictive maintenance in aviation, enabling efficient data processing, anomaly detection, and component health scoring to improve fleet readiness and prevent failures.
Care and Feeding of Catalyst Optimizer - Practical Troubleshooting for Spark SQL
Explore Spark's Catalyst Optimizer challenges, including UDF issues, codegen limitations, and JVM tuning. Learn to diagnose and solve complex query problems for improved Spark SQL performance.
Faster Data Integration Pipeline Execution Using Spark-Jobserver
Optimize data integration pipelines using Spark-Jobserver for faster execution, context reuse, and efficient resource management. Learn configuration, security, and API integration for improved performance.
Encryption and Masking for Sensitive Apache Spark Analytics Addressing CCPA and Governance
Practical methods for balancing data encryption and analytics in Apache Spark, addressing CCPA and governance requirements while maintaining analytical value.
Running Apache Spark Jobs Using Kubernetes
Learn to run Apache Spark jobs on Kubernetes, unifying analytics and data science on a cloud-native architecture. Discover how this approach simplifies infrastructure, enables on-demand deployment, and eliminates big data cluster overhead.
Tackling Scaling Challenges of Apache Spark at LinkedIn - Infrastructure Optimization and User Productivity
Explore LinkedIn's journey scaling Apache Spark, addressing infrastructure challenges, optimizing resource management, and enhancing user productivity through automated analysis and efficiency improvements.
Securing the Spark Fire Hose - 2016
Learn to secure Apache Spark in big data analytics. Explore code and configuration security, data protection strategies, and integration with common technologies to safeguard critical information throughout its lifecycle.
Managing Multi-Cloud Apache Spark on Kubernetes
Learn strategies for managing Apache Spark on Kubernetes across multiple clouds, focusing on auto-scaling, scheduling, and observability techniques for efficient data science infrastructure.
Accelerating Spark Workloads in a Mesos Environment with Alluxio
Learn how Alluxio enhances Spark and Mesos performance by providing memory-speed data access across disparate storage systems, optimizing big data processing and enabling new workloads.
Introducing Kubeflow with TensorFlow and Apache Spark
Explore Kubeflow's role in streamlining machine learning workflows, from model development to production deployment on Kubernetes, with insights on scalability and integration with TensorFlow and Apache Spark.