Lessons Learned from DAG-Based Workflow Orchestration
Explore the evolution of workflow orchestration in this insightful conference talk from MLOps World: Machine Learning in Production. Delve into the limitations of traditional Directed Acyclic Graph (DAG) based systems and discover how Prefect Orion (Prefect 2.0) addresses these constraints. Learn about the challenges data professionals face when fitting workflows into DAGs, including re-running partial workflows, executing long-running workflows, and dynamically adding tasks during runtime. Gain valuable insights from Kevin Gregory Kho, Senior Open Source Community Engineer at Prefect, as he discusses the transition to a DAG-less workflow orchestration system. Understand how Orion enhances developer experience by offering a more Pythonic interface and improved flexibility. Discover how this new approach allows for greater observability into specific tasks within workflows, ultimately leading to more efficient and adaptable data pipelines.