Scaling Machine Learning Workflows to Big Data with Fugue
Learn to scale machine learning workflows from Pandas to distributed computing using Fugue, an open-source abstraction layer. Discover how to port Python code to Spark or Dask with minimal changes, enabling framework-agnostic and scale-agnostic data scie…
Pandas Parallel Transformations with Dask and Pandarallel
Optimize data processing with Dask and Pandarallel for faster Pandas transformations. Learn parallel computing techniques to enhance performance.
Applications of Distributed Task Parallelism
Explore distributed task parallelism in Julia, comparing Dagger with other frameworks and libraries. Learn about unique features for efficient, scalable, and maintainable code.
Terraforming Earth - How Terraform Bridges the Multi-Cloud Gap for Earth Observation
Explore how Terraform bridges multi-cloud gaps in Earth Observation, enabling efficient data processing and resource management across distributed infrastructures for improved accessibility and analysis.
High Performance Scalable Support for Big Data Stacks with MPI - Leveraging MPI4Spark and MPI4Dask
Discover how MPI4Spark and MPI4Dask enhance big data frameworks by leveraging high-speed networks and HPC systems for improved performance and scalability in distributed computing environments.
Reproducible ML Experimentation with Rubicon-ML
Explore reproducible ML experimentation using rubicon-ml, an open-source Python library for capturing and storing model training information in a searchable way, ensuring full reproducibility and audit-ability.
From Built-in Concurrency Primitives to Large Scale Distributed Computing
Explore asynchronous execution in Python, from basic concurrency to distributed systems. Master Future and Executor interfaces, asyncio, and frameworks like Dask for efficient large-scale data processing.
Pandas and Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
Explore Dask DataFrame 2.0's improvements and performance compared to Spark, DuckDB, and Polars. Learn about new features, benchmarks, and future developments in distributed computing with Python.
Two Sigma: Push Computing Research Boundaries with Google Cloud
Explore TwoSigma's AI/ML platform integrating Ray, Spark, and Dask on Google Kubernetes Engine. Learn about architecture, capabilities, and on-demand GPU access for investment strategy testing.
A Cross-Platform, Cross-Tool Ray on Kubernetes Deployment
Deploy Ray, Spark, Dask, and Jupyter on mixed-architecture Kubernetes clusters. Learn cross-building containers, handling Ubuntu versions, and glibc versioning for heterogeneous Ray clusters.
Pandas and Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
Explore Dask DataFrame 2.0's improvements, comparing its performance to Spark, DuckDB, and Polars using TPC-H benchmarks. Learn about new features enhancing speed, efficiency, and usability in big data processing.