Scheduling Jupyter Notebooks Using Airflow on Kubernetes
Explore how to schedule Jupyter notebooks using Airflow on Kubernetes in this informative conference talk by Akshay Chitneni and Rus Pandey from Apple. Discover the potential of extending Jupyter notebooks beyond interactive data analysis to build data and machine learning pipelines. Learn how Kubernetes constructs such as operators, autoscalers, and pod identities can be combined with Airflow to schedule notebooks and share execution results. Gain insights into creating a comprehensive notebooks platform that addresses user authentication, authorization, scalable compute, and isolated execution environments with dependencies. Understand how this approach can empower data scientists and engineers to accelerate the productionization of their work.