Large Scale Batch Processing with Argo Workflows and Events
Explore large-scale batch processing using Argo Workflows and Events in this 29-minute conference talk by Rakesh Subramanian Suresh and Saravanan Balasubramanian from Intuit. Discover how Intuit manages approximately 40,000 data processing pipelines with Argo Workflows, with 10% running concurrently daily. Learn about the event-driven approach using Argo Events to handle calendar events, upstream pipeline statuses, and data updates. Gain insights into scaling Argo Events to manage around 400,000 different events for pipeline workflows and achieving exactly-once triggering. Delve into the architecture of the Batch Processing Platform, multi-cluster scaling, high availability strategies, and pod distribution budgets. Understand the process of creating data processors and pipelines, as well as scheduling and managing data dependencies. Get a glimpse of future developments in this field.