Explore the intricacies of creating extensible workflows using unconventional Python techniques in this PyCon US talk by Benjy Weinberger. Dive into the challenges of designing a programmatic interface for workflow systems, balancing flexibility and constraints. Learn how the Pants build system leverages Python's type annotations, coroutines, and dataclasses to create an elegant extensibility API. Discover how these features can be combined to support complex functionalities like caching, concurrency, and remote execution without burdening task authors. Gain insights into workflow system design, task API challenges, and practical applications in data processing, ETL, and CI/CD. Follow along with real-world examples and understand how to apply these concepts to build more efficient and maintainable workflow-oriented systems.
Creating Extensible Workflows with Off-Label Use of Python