Let's Play with Charmed Spark - Apache Spark on Kubernetes
Explore Apache Spark on Kubernetes using Charmed Spark. Learn to deploy, configure, and run Spark workloads, integrate with Kafka, and monitor jobs through hands-on demonstrations.
Faster, Safer, Serverless - Empowering Apache Spark Standalone Cluster on Kubernetes
Explore a K8s-native Serverless Spark Service that enhances speed and simplicity, featuring a new K8s operator for cluster creation and job submission, with rapid responsiveness and ML training integration.
Super Reliable Cloud Native Data Processing Using Apache Spark and Cloud Shuffle Manager
Enhance Apache Spark's reliability on Kubernetes with Cloud Shuffle Manager, enabling fault tolerance and cost savings through Spot Instances while maintaining performance for large-scale data processing.
Building an Open Data Lakehouse Using Apache Spark, Apache Iceberg and Dremio
Explore the evolution of big data to data lakehouses, focusing on open architecture development, security, governance, and augmentation using Dremio, Apache Spark, and Apache Iceberg.
The Free Lunch is Over: Dealing with Unstructured Data in the Era of LLMs
Explore unstructured data processing using ML, LLMs, and Apache tools. Learn to handle audio, images, and text with vector embeddings for natural language analysis in data engineering pipelines.
Spark SQL Shuffle Join Optimization at eBay
Optimize Spark SQL shuffle join performance with advanced techniques like unwrapping join conditions, enhancing shuffle exchange reuse, and pushing down partial aggregations.
Databricks Photon and Spark SQL in Advanced Database Systems - Lecture 20
Explore Databricks Photon and Spark SQL's advanced features for efficient data processing and analysis in modern database systems.
Git for Data Lakes: How LakeFS Scales Data Versioning to Billions of Objects
Learn how lakeFS enables Git-like versioning for data lakes, solving challenges of object storage and scaling to billions of objects without compromising performance. Explore branching, committing, and merging for petabyte-scale data management.
Accelerating Apache Spark Workloads with Apache DataFusion Comet
Explore Apache DataFusion Comet's potential to enhance Apache Spark performance, optimizing data processing workflows and improving efficiency in large-scale analytics.
Databricks Photon and Spark SQL in Advanced Database Systems - Lecture 18
Explore Databricks Photon and Spark SQL architectures, optimizations, and performance improvements in modern data processing systems.
Challenges of Spark Application Coexisting with NoSQL Databases
Explore challenges and solutions for integrating Apache Spark with NoSQL databases like MongoDB and Cassandra in high-volume transaction processing systems, focusing on performance optimization and data modeling.
Big Data Transformation Powered by Apache Spark
Revolutionize data processing with Spark. Master transformative functions, aggregation, array manipulation, and advanced joins to unlock actionable insights from big data.
Delta Format and Live Tables - Pioneering Data Innovation
Explore Delta format and Live tables, understanding their role in scalable big data pipelines and modern data engineering innovations.
Data Engineering: A Deep Dive into Databricks
Master Databricks for efficient data processing, focusing on Apache Spark transformations, ingestion methods, storage, orchestration, and best practices to optimize data engineering workflows.
Introduction to Azure Databricks
Explore Azure Databricks: Simplify big data analytics, streamline processing, and enhance data accessibility for informed decision-making.
Data Engineering with Databricks - Leveraging the Lakehouse Platform for ETL Pipelines
Leverage Databricks Lakehouse Platform to build ETL pipelines, use Delta Live Tables, and orchestrate tasks with Workflows for efficient data processing.
Spark Unveiled: Essential Insights for All Developers
Discover Apache Spark's core concepts, architecture, and impact on data processing. Gain essential insights to enhance your development skills and unlock Spark's potential.
Understanding the SQL Aspects of Spark - Spark SQL
Explore Spark SQL's capabilities for distributed processing of structured data, integrating relational operations with Spark's high-performance engine.
Introduction to Databricks and Data Lakehouse Architecture
Explore Data Lakehouse, Databricks platform, Spark architecture, and SQL. Learn advantages and common use cases of Databricks for data processing.
State Storage in Spark Structured Streaming
Learn to configure and manage state storage in Spark Structured Streaming for efficient data processing and analysis.
Implementing Change Data Capture (CDC) and Slowly Changing Dimensions (SCD) Using Spark and Hive
Learn to implement Change Data Capture (CDC) and Slowly Changing Dimension (SCD) patterns in Hive using Spark, with a step-by-step guide for efficient data management and analysis.