Building Data Workflows with Luigi and Kubernetes
Explore how to build complex data pipelines in Python using Luigi and Kubernetes in this EuroPython 2019 conference talk. Learn about Luigi's problem-solving capabilities for batch job management, including dependency resolution, workflow management, visualization, and failure handling. Discover techniques for packaging Luigi pipelines as Docker images for easier testing and deployment. Gain insights into deploying pipelines on Kubernetes clusters for scalable Big Data processing and cost-effective infrastructure management. Get tips and tricks for optimizing Luigi Scheduler's performance with Kubernetes batch execution. Benefit from a demo project and practical advice tailored for data scientists, data engineers, BI developers, and software developers working with batch jobs and Big Data.