Argonauts of Data: Building Scalable and Effective Data Pipelin... Satabrata Paul & Nishchith Shetty
Description:
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Grab it
Watch a 24-minute conference talk exploring how Atlan's data team revolutionized their ETL pipeline performance using Argo workflows. Learn how they overcame challenges of massive data processing by implementing parallelization techniques, retry mechanisms, and effective artifactory loading. Discover the technical journey that led to processing 100 million assets with a 450% reduction in processing time and handling 3 million SQL queries in just 2 hours. Gain insights into scaling data pipelines effectively as speakers detail how they achieved a 50% reduction in overall pipeline runtime through Argo-powered horizontal scaling solutions.
Building Scalable and Effective Data Pipelines with Argo for Large-Scale Metadata Processing