Bloomberg's Journey to a Multi-Cluster Workflow Orchestration Platform

Bloomberg's Journey to a Multi-Cluster Workflow Orchestration Platform

Explore Bloomberg's journey in developing a multi-cluster workflow orchestration platform for automation, data pipelines, model training, and general utility compute. Learn about the challenges faced with rapid adoption, including increased latency and query timeouts. Discover the evolution from an initial implementation using bespoke in-cluster pull agents and a source-of-truth to a current approach leveraging a highly-available PostgreSQL database and Kine. Gain insights into the investigation of related projects and inspirations drawn from Karmada, OCM, and others. Understand the lightweight Kubernetes API server interface design that enables efficient multi-cluster federation for deploying heterogeneous resources.