DiskArrays.jl for Working with Large Disk-Based ND-Arrays
Explore the capabilities of DiskArrays.jl for handling large disk-based nd-arrays in this conference talk from JuliaCon 2024. Learn how this Julia package addresses the challenges of implementing the full AbstractArray interface for scientific data formats like NetCDF, HDF5, and Zarr. Discover the AbstractDiskArray interface, which simplifies backend implementation by requiring only methods for reading and writing dense hyperrectangular subsets of arrays. Gain insights into the package's features for querying internal chunking structures, enabling efficient mapreduce and broadcast operations. Examine how downstream packages like Rasters.jl, YAXArrays.jl, and DiskArrayEngine.jl leverage DiskArrays.jl to provide efficient and user-friendly ways to perform operations on larger-than-memory compressed arrays.