Главная
Study mode:
on
1
Introduction
2
Agenda
3
Input Parameters
4
Tuning Max Files per Trigger
5
Tuning State Parameters
6
Performing Aggregates
7
Watermark
8
Help Function
9
State Store Provider
10
State Store Limits
11
Delta Back State
12
Delta Back State Code
13
Performance
14
Output Parameters
15
Small Things to Consider
Description:
Explore critical aspects of running streaming jobs in production environments through this 54-minute conference talk by Databricks. Learn how to prevent common pitfalls that can cause serious issues when productionizing streaming jobs. Dive into four key topics: configuring input parameters to handle unexpected data volume increases, tuning stateful streaming parameters to avoid infinite state accumulation, optimizing Structure Streaming output parameters to prevent small file problems, and modifying streaming jobs in production with checkpoints. Gain practical, hands-on examples of issue manifestation and prevention techniques. Equip yourself with the knowledge to design performant and fault-tolerant streams, ensuring smooth operation in production environments.

Preventing Common Pitfalls in Production Streaming Jobs

Databricks
Add to list
0:00 / 0:00