Главная
Study mode:
on
1
Intro
2
PROBLEMS
3
WHAT TO DO?
4
OPTIONS?
5
DESIGN PRINCIPLES
6
KNOW YOUR DATA.
7
DON'T REINVENT THE WHEEL.
8
KEEP IT SIMPLE.
9
KNOW YOUR USERS.
10
KNOW YOUR HARDWARE.
11
DATA INGESTION AND STORAGE
12
WHAT TO STORE?
13
HOW TO BE RESILIENT?
14
HOW TO SCALE?
15
COMPACTED TOPICS
16
WINDOWED DATA
17
DATA MODEL
18
IN-MEMORY COMPUTING
19
RAM IS VOLATILE
20
ALGEBRA OF SETS
21
BITMAPS
22
BACK TO MNEMOSYNE
23
SPARSITY
24
AGGREGATIONS
25
WAS IT WORTH IT?
26
CAN YOU DO IT?
27
SHOULD YOU DO IT?
28
THANK YOU!
Description:
Explore the development of Mnemosyne, a distributed indexing layer for big data, in this 52-minute Devoxx conference talk. Dive into the challenges faced by Brandwatch Audiences product and learn how they built a system capable of handling hundreds of millions of social network profiles, billions of posts, and tens of billions of follower graph edges in real-time. Discover the fusion of succinct data structures, free text search, in-memory computing with JVM, CUDA, and Kafka to create a high-performance solution. Gain insights into CAP theorem trade-offs, brute force approaches versus indexes, data structures for sorting billions of records in milliseconds, GPU problem-solving, and JVM implementation. Understand key design principles, data ingestion and storage techniques, in-memory computing concepts, and the use of bitmaps and sparse data structures. Evaluate the project's success and consider the feasibility and advisability of undertaking similar endeavors in your own work.

Mnemosyne - A Distributed Bitmapped Indexing Layer for Big Data

Devoxx
Add to list