Healthcare Claim Reimbursement Using Apache Spark - Optum's System Transformation
Optimizing healthcare claim processing with Apache Spark: Improved performance, reduced costs, and enhanced capabilities for hospitals' revenue recovery through efficient data processing and analysis.
Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes
Explore cloud-native Apache Spark scheduling on Kubernetes using YuniKorn. Learn about efficient resource management, quota handling, and auto-scaling for large-scale Spark jobs in cloud environments.
Bringing Satellite and Drone Imagery into Data Science Workflows
Explore satellite and drone imagery analysis in data science workflows using RasterFrames, a Spark SQL extension with Python API for processing spatial image data and integrating with ML frameworks.
Scaling Data and ML with Apache Spark and Feast - Feature Engineering for Production
Explore scaling data and ML with Apache Spark and Feast, focusing on feature engineering challenges and solutions for big data-driven machine learning in production environments.
Accelerating Apache Spark Shuffle for Data Analytics on Cloud with Remote Persistent Memory Pools
Innovative solution for accelerating Apache Spark shuffle operations using remote persistent memory pools, improving performance, scalability, and reliability in cloud-based big data analytics.
Geospatial Data Analysis in Apache Spark - Challenges and Solutions
Explore geospatial data challenges in Apache Spark, comparing libraries, and learning best practices for data ingestion, storage, and retrieval using spatial indexing techniques.
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
Explore Spark SQL's Adaptive Query Execution framework, which optimizes query plans at runtime using real-time statistics for improved performance in data processing and analysis.
The Apache Spark File Format Ecosystem - Optimizing Storage for Performance
Explore key file formats in Spark ecosystem: Parquet, ORC, Avro. Learn core concepts, optimization techniques, and how to select the right format for performant data processing and AI/ML workflows.
Deep Dive into GPU Support in Apache Spark 3.x - Accelerator-Aware Scheduling and RAPIDS Plugin
Explore GPU acceleration in Apache Spark 3.x, covering task scheduling, columnar processing, and the RAPIDS plugin for enhanced ETL, ML, and DL performance without code changes.
Building a Real-Time Feature Store for Machine Learning at iFood
Discover how iFood built a real-time feature store using Databricks and Spark Structured Streaming to process event streams, store data, and develop reliable code for efficient machine learning model predictions.
Using Apache Spark and Differential Privacy for 2020 Census Data Protection
Explore how Apache Spark and differential privacy protect 2020 Census data, balancing accuracy and confidentiality. Learn about the innovative TopDown Algorithm and custom monitoring systems.
Fine-Tuning and Enhancing Performance of Apache Spark Jobs
Explore best practices for optimizing Apache Spark jobs, including resource allocation, parameter tuning, and performance enhancement techniques. Learn to identify bottlenecks and implement solutions for improved efficiency.
Real-Time Forecasting at Scale Using Delta Lake and Delta Caching
Explore real-time forecasting at scale using Delta Lake and Delta Caching. Learn efficient data sampling, storage, and caching techniques for handling massive datasets and achieving rapid forecast response times.
Building a Streaming Microservice Architecture with Apache Spark Structured Streaming
Explore building a scalable, fault-tolerant streaming microservice architecture using Apache Spark, Kafka, gRPC, and Delta Lake for real-time data processing and reliable insights.
Productionizing Machine Learning with Apache Spark, MLflow and ONNX - Cloud Deployment Using SQL Server
Explore productionizing machine learning using Apache Spark, MLflow, and ONNX with SQL Server. Learn model management, lifecycle orchestration, and leveraging SQL Server for ML model storage and deployment.
Building the Petcare Data Platform with Delta Lake and Spark ETL Pipeline
Explore Mars Petcare's data platform using Delta Lake and Spark ETL pipeline 'Kyte'. Learn about advantages over Azure Data Factory and leveraging Delta Lake for ETL configurations and data science.
Using Apache Spark for Predicting Degrading and Failing Parts in Aviation
Leveraging Apache Spark for predictive maintenance in aviation, enabling efficient data processing, anomaly detection, and component health scoring to improve fleet readiness and prevent failures.
Care and Feeding of Catalyst Optimizer - Practical Troubleshooting for Spark SQL
Explore Spark's Catalyst Optimizer challenges, including UDF issues, codegen limitations, and JVM tuning. Learn to diagnose and solve complex query problems for improved Spark SQL performance.
Faster Data Integration Pipeline Execution Using Spark-Jobserver
Optimize data integration pipelines using Spark-Jobserver for faster execution, context reuse, and efficient resource management. Learn configuration, security, and API integration for improved performance.
Encryption and Masking for Sensitive Apache Spark Analytics Addressing CCPA and Governance
Practical methods for balancing data encryption and analytics in Apache Spark, addressing CCPA and governance requirements while maintaining analytical value.
Running Apache Spark Jobs Using Kubernetes
Learn to run Apache Spark jobs on Kubernetes, unifying analytics and data science on a cloud-native architecture. Discover how this approach simplifies infrastructure, enables on-demand deployment, and eliminates big data cluster overhead.