Главная
Study mode:
on
1
Intro
2
Approach to data scaling problems
3
2 total engineers
4
First bottleneck: disk IO on old Amazon EBS
5
Django DB Routers
6
PG Replication to bootstrap nodes
7
Scaling up Redis
8
fork() and COW
9
Vertical partitioning by data type
10
No easy migration story; mostly double-writing
11
Replicating + deleting often leaves fragmentation
12
Why not Redis for kv caching?
13
Slab allocator
14
Focus on client
15
Testing & monitoring kept concurrent fires to a minimum
16
Scaling Out
17
Database Scale Out
18
Double write, shadow reads
19
Stressing about Primary Key
20
Data loss, segfaults
21
train + rapidly approaching cliff
22
Logical partitioning, done at application level
23
note to self: pick a power of 2 next time
24
Postgres "schemas"
25
9.2 upgrade: bucardo to move schema by schema
26
ID generation
27
Snowflake, other options
28
41 bits: time in millis (41 years of IDs) 13 bits: logical shard ID 10 bits: auto-incrementing sequence, modulo 1024.
29
Lesson learned
30
minimize moving parts
31
Ending the year
32
Launched Android
33
Stability, FB
34
Scaling cut-overs, ramp- ups, and development
35
Dynamic ramp-ups and config
36
Python Knobs
37
Decouple deploy from feature rollout
38
In memory requirement
39
Simplest thing was breaking
40
Trimming
41
C* cluster is 35% of the size of Redis one
42
Handling deletes
43
Redis way: LREM
44
Not so hot for an AP system
45
2014 project
46
Spam fighting
47
Generic features + machine learning
48
Hadoop + Hive + Presto
49
2010 vintage infra
50
#1 impact: recruiting
51
Wrap up
Description:
Explore the journey of scaling storage systems at Instagram from 30 million to 150 million active users with a small engineering team. Learn about the evolving architecture, challenges faced, and solutions implemented over three years. Discover insights on database scaling, Redis optimization, logical partitioning, ID generation, and lessons learned in minimizing moving parts. Gain knowledge about dynamic ramp-ups, feature rollouts, and spam fighting techniques. Understand the impact of infrastructure choices on recruitment and the importance of adapting to rapid growth in a startup environment.

Storage Systems at a Rapidly Scaling Startup - Instagram's Infrastructure Evolution

Meta
Add to list
#Computer Science #Information Technology #Infrastructure Management #Business #Marketing #Digital Marketing #Social Media #Instagram
00:00
-01:29
1 of 51

1 Intro