Building a Batch Processing Platform for Data Pipelines Using Argo and Kubernetes
Explore Intuit's scalable batch processing platform using Kubernetes and Argo for efficient data pipeline management, addressing scheduling, orchestration, and dependency challenges in AI and data engineering.
Building a Batch System for the Cloud with Kueue
Learn about Kueue, a cloud-native job scheduler for Kubernetes that optimizes resource allocation, implements fair sharing, and enhances batch processing efficiency in multi-team environments.
Kubernetes Batch Processing at Scale - A Scheduling Perspective
Explore Kubernetes batch processing challenges and solutions, including custom scheduling for large-scale operations, Gang Scheduling, DRF, and workload colocation for improved resource utilization.
Enabling HPC and ML Workloads with Latest Kubernetes Job Features
Explore new Kubernetes Job API features for distributed Batch/AI/HPC workloads, including Indexed Jobs and Pod Failure Policy, with real-world examples from DeepMind and Lawrence Livermore National Laboratory.
Transforming Data Processing with Kubernetes - Journey Towards a Self-Serve Data Mesh
Explore Intuit's use of Kubernetes with Data Mesh principles to transform data processing, enabling efficient management of 100,000+ pipelines and fostering a collaborative data culture for improved decision-making.
Unified and Pluggable Cluster Management Platform at LinkedIn
Explore LinkedIn's journey in adopting Kubernetes for cluster management, addressing challenges and integrating with existing systems at scale.
Scaling Batch, Big Data, and AI Workloads Beyond the Kubernetes Scheduler
Explore solutions for scaling AI, big data, and batch workloads on Kubernetes. Compare projects like Koordinator, Kueue, MCAD, Volcano, and YuniKorn to optimize cluster utilization and overcome scheduling challenges.
A Decade of High-Volume Data and APIs - The Evolution of SIG-Apps
Explore the evolution of Kubernetes workload support and APIs over a decade, highlighting SIG Apps' achievements, milestones, and future directions in cloud native computing.
Exploring Armada: Multi-Kubernetes Cluster Batch Job Meta-Scheduler
Explore Armada, a multi-Kubernetes-cluster batch job meta-scheduler, with Carlos Santana. Learn its features, benefits, and best practices for optimizing cloud-native computing workflows.
Growing Pain: Scaling to 10K Workflows per Week
Insights on scaling a workflow orchestration platform to handle 10K workflows weekly, addressing challenges like spiky submissions, resource management, and diverse user requirements.
LoRAX - Serving Thousands of Fine-Tuned LLMs on a Single GPU
Explore LoRAX, an innovative LLM inference system enabling efficient serving of multiple fine-tuned models on a single GPU. Learn about dynamic adapter loading, heterogeneous batching, and scheduling techniques.
Batch Processing Tekton Pipelines in OpenShift
Explore Tekton for batch processing pipelines in OpenShift, covering design, implementation, and best practices with real-world examples from Red Hat's insights data processing.
LLMOps: Using Nvidia TensorRT SDK for GPU Inference
Optimize GPU inference using Nvidia TensorRT SDK. Convert models, compare throughput, and explore batch sizes, data precision, and runtime options for enhanced performance.
Enhancing Flexibility and Productivity with Access Patterns and Storage-Agnostic Abstractions
Explore a platform bridging data management gaps, offering storage-agnostic abstractions and ACID guarantees across streaming, batch, and random-access storages for flexible, consistent data handling.
Riding the Stream Processing Wave
Explore LinkedIn's stream processing challenges, innovations in machine learning applications, and advancements in nearline processing. Learn about state management and offline-online convergence in large-scale data systems.
How Delta Lake with Azure Databricks Can Accelerate Your Big Data Workloads by 100x
Explore advanced Delta Lake features on Azure Databricks for reliable, scalable data lakes. Learn optimizations for analytics and ML workloads, including file compaction, Z-Order partitioning, and unified batch/streaming.
Dynamic Management of Docker Containers in Multi-Cloud and Edge Using MiCADO
Explore MiCADO: an open-source framework for dynamic management of Docker containers across multi-cloud and edge environments, built on Kubernetes. Learn about its components, policies, and applications in various scenarios.
Big Data Analytics in Near-Real-Time with Apache Kafka Streams
Explore near-real-time data analytics using Apache Kafka streams. Learn to process, aggregate, and distribute large volumes of data efficiently, moving beyond traditional ETL methods for instant insights.
Stop Writing Dead Programs
Explore the future of programming with a critical look at current practices and innovative ideas for live programming environments that could revolutionize problem-solving in technology.
Improve Your App UX with One Model Processing Insights from Batch and Streams
Learn to enhance app UX using a unified model for batch and stream processing. Explore Dataflow's capabilities through a real-time demo, gaining practical insights on managing and visualizing data streams.
Scalable Stream Processing - A Survey of Storm, Samza, Spark and Flink
Comprehensive overview of popular stream processing systems for real-time Big Data analytics, comparing Storm, Samza, Spark, and Flink. Explores use cases, architectures, and trade-offs for low-latency data processing.