Play all

Intro

Nearline Applications

Heterogeneous Data Systems

Building the Right Infrastructure

Pluggable Sources & Destinations

Capturing Live Updates

Change Data Capture (CDC)

Streaming Bridge

Mirroring Kalka Data

Brooklin Kalka Mirroring

Application Use Cases

Client makes REST call to create datastream

Datastream is written to Zookeeper

Leader coordinator is notified of new datastream

Leader coordinator calculates work distribution

Leader coordinator writes the assignments to ZK

Zookeeper is used to communicate the assignments

Coordinators hand task assignments to consumers

Consumers start streaming data from the source

Consumers propagate data to producers

Producers write data to the destination

Destinations can be shared by apps

Brooklin Architecture

Brooklin in Production

Brooklin is now open-source!

Description:

Explore a comprehensive overview of Brooklin, LinkedIn's managed data streaming service, in this 41-minute Strange Loop Conference talk. Learn how Brooklin addresses the challenges of scaling up to handle increasing data volume and supporting the proliferation of new data systems at LinkedIn. Discover the architecture and use cases of this centralized, extensible solution for continuously delivering data to nearline applications. Examine Brooklin's support for multiple pluggable sources and destinations, including data stores and messaging systems. Delve into its applications in change data capture (CDC), data propagation between different systems and environments, and Kafka data mirroring. Gain insights into Brooklin's production implementation, its role in replacing Kafka MirrorMaker at LinkedIn, and future development plans for this open-source project.

Dive into Streams with Brooklin

Strange Loop Conference

Add to list

#Conference Talks #Strange Loop Conference #Data Science #Big Data #Data Streaming