Hyperspace - An Indexing Subsystem for Apache Spark
Explore the design, implementation, and operationalization of Hyperspace, an indexing subsystem for Apache Spark, in this 32-minute conference talk by Databricks. Learn about the foundations of the indexing infrastructure, including API design and integration with Spark's Catalyst optimizer. Discover how Hyperspace enables users to build, maintain, and leverage indexes on various data formats for query acceleration and resource cost reduction. Gain insights into the multi-user concurrency model and the development roadmap for open-sourcing this technology. Through presentations, benchmarks, code examples, and notebooks, delve into the world of efficient data indexing for large-scale datasets ranging from GBs to PBs, addressing both batch-style queries and explorative analytics.