Hybrid Apache Spark Architecture: Optimizing YARN and Kubernetes for Lyft's Workloads
Explore Lyft's hybrid Apache Spark architecture using YARN and Kubernetes, addressing scalability challenges and optimizing workload management for improved reliability and efficiency.
Optimizing Spark SQL Jobs with Parallel and Asynchronous IO
Optimizing Spark SQL jobs through parallel and async IO techniques, including file-level and row group-level parallel read, asynchronous spill, and parquet column family, resulting in 5-30% performance improvement.
Migrating ETL Workflows to Apache Spark at Scale - Pinterest's Experience
Explore Pinterest's journey migrating ETL workflows to Apache Spark, covering challenges, solutions, and performance improvements. Learn about semantic gaps, thrift objects, and innovative Spark profiling techniques.
Delta Live Tables: Modern Software Engineering for ETL Pipelines
Learn how Delta Live Tables simplifies ETL, enhances data pipeline reliability, and automates tasks. Discover modern software engineering practices for efficient data transformation and improved analytics, data science, and ML outcomes.
Hackaday Supercon 2022 - Chris Gammell - The One-Engineer Dev Shop, Then and Now
Techniques, tools, and resources for building efficient one-person development shops, covering design, engineering, and project management aspects of prototype and product creation.
Accelerating Data Processing in Spark SQL with Pandas UDFs - Optimization Techniques
Learn to optimize Spark SQL data processing using Pandas UDFs. Discover techniques to speed up queries, profile jobs efficiently, and leverage custom UDFs for significant performance gains in large-scale data pipelines.
Apache Spark vs Databricks - On-Premise Performance Comparison
Comparison of on-premise Databricks and open-source Spark implementations, showcasing 10x performance gains in cyber workloads and addressing challenges of air-gapped analytics solutions.
DataSketches: A Production Quality Sketching Library for Big Data Analysis
Explore sketching algorithms for efficient big data analysis, enabling fast approximate answers to complex queries and real-time processing of massive datasets in production environments.
Hyperspace - An Indexing Subsystem for Apache Spark
Explore Hyperspace, an indexing subsystem for Apache Spark that enhances query performance and reduces resource costs. Learn about its design, implementation, and benefits for data analytics on large datasets.
Portable and Scalable Data Visualization Techniques for Apache Spark and Python Notebooks
Explore portable and scalable data visualization techniques for Apache Spark and Python notebooks, covering strategies, packages, headless rendering, animations, and big data integration.
Preventing Common Pitfalls in Production Streaming Jobs
Explore key strategies for optimizing streaming jobs in production, covering input parameters, stateful streaming, output configurations, and job modifications to ensure performance and fault tolerance.
Composable Data Processing with Apache Spark - Scaling Development and Error Handling
Learn about SIP, an extensible plugin framework for Apache Spark that enhances data processing efficiency, error handling, and scalability in Adobe's Experience Platform.
PLSE Seminar Series - Alessandra Gorla - Mining Android Apps for Anomalous Behavior
Techniques for detecting suspicious behavior in Android apps by clustering based on advertised functionality and identifying outliers in API usage, with applications in malware detection.
How to Automate Performance Tuning for Apache Spark
Discover techniques to automate Apache Spark performance tuning, addressing common issues and exploring tools for efficient, scalable data pipeline maintenance in production environments.
Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors
Lessons learned from running massive Spark workloads on Kubernetes at Apple, covering orchestration, resource management, monitoring, and optimization for scalability and performance.
Deep Learning Day - Advances in Theory, Systems, and Applications
Comprehensive exploration of deep learning featuring expert talks on scaling, debugging, and advanced techniques. Covers theoretical foundations and practical applications across various domains.
What You See Is Not What You Get - Discovering-Tracking Ad-Driven Social Engineering Attack Campaigns
Explore techniques for uncovering and monitoring social engineering attacks driven by online advertisements, with insights on ad networks, tracking methods, and measurement strategies.
Accelerating Apache Spark Shuffle for Data Analytics on Cloud with Remote Persistent Memory Pools
Innovative solution for accelerating Apache Spark shuffle operations using remote persistent memory pools, improving performance, scalability, and reliability in cloud-based big data analytics.
Care and Feeding of Catalyst Optimizer - Practical Troubleshooting for Spark SQL
Explore Spark's Catalyst Optimizer challenges, including UDF issues, codegen limitations, and JVM tuning. Learn to diagnose and solve complex query problems for improved Spark SQL performance.
Tackling Scaling Challenges of Apache Spark at LinkedIn - Infrastructure Optimization and User Productivity
Explore LinkedIn's journey scaling Apache Spark, addressing infrastructure challenges, optimizing resource management, and enhancing user productivity through automated analysis and efficiency improvements.
Privacy by Design for Agile Development - Uber's Approach
Integrating privacy by design principles in agile development at Uber: challenges, strategies, and practical approaches for ensuring data protection in a fast-paced tech environment.