Generalized Sub-Query Fusion for Eliminating Redundant I-O from Big-Data Queries

Generalized Sub-Query Fusion for Eliminating Redundant I-O from Big-Data Queries

Explore a 19-minute conference talk from USENIX OSDI '20 that introduces RESIN, an optimizer extension designed to eliminate redundant I/O in big-data SQL queries. Learn about Generalized Sub-Query Fusion, a novel technique that identifies and fuses sub-queries computing on overlapping data into the same map-reduce stages. Discover how this approach can optimize query execution by reducing disk and network I/O, sometimes eliminating expensive binary operations like Joins and Unions. Gain insights into the implementation of RESIN in sparkSQL and its performance improvements on the TPCDS benchmark suite, demonstrating speed-ups of 1.1-6x for 40% of queries and a 12% reduction in overall execution time.