Главная
Study mode:
on
1
Intro
2
Accelerator-Aware Scheduling
3
GPU Scheduling Example
4
GPU Discovery Script Example
5
GPU Assignments API
6
GPU Scheduling UI
7
Stage Level Scheduling
8
SOL Columnar Processing
9
Spark 3 with Project Hydrogen
10
Deep Learning Recommendation Machines
11
RAPIDS Accelerator for Apache Spark (Plugin)
12
Is This a Silver Bullet?
13
But It Can Be Amazing
14
Spark SOL & DataFrame Compilation Flow
15
ETL Technology Stack
16
Demo Cluster Setup
17
Databricks Demo Results
18
T4 Cluster Setup
19
RAPIDS Accelerator on AWS
20
Spark Shuffle
21
Accelerated Shuffle Results
22
What's Next
Description:
Dive deep into GPU support in Apache Spark 3.x with this comprehensive 47-minute video from Databricks. Explore accelerator-aware task scheduling, columnar data processing support, fractional scheduling, and stage-level resource scheduling and configuration. Learn about the Apache Spark 3.x RAPIDS plugin, enabling GPU acceleration without code changes. Understand how the Catalyst optimizer physical plan is modified for GPU-aware scheduling and how the plugin leverages RAPIDS libraries. Discover optimizations made to the shuffle plugin using UCX for GPU memory communication. Gain insights into future optimizations involving RDMA and GPU Direct Storage. Examine industry-standard benchmarks and real-world production dataset performance. Cover topics such as GPU scheduling, discovery scripts, assignments API, UI, stage-level scheduling, SOL columnar processing, Project Hydrogen, Deep Learning Recommendation Machines, and accelerated shuffle results.

Deep Dive into GPU Support in Apache Spark 3.x - Accelerator-Aware Scheduling and RAPIDS Plugin

Databricks
Add to list
0:00 / 0:00