Using Apache Spark and Differential Privacy for 2020 Census Data Protection
Explore the innovative use of Apache Spark and differential privacy in protecting respondent confidentiality for the 2020 US Census in this 29-minute talk. Dive into the challenges of balancing data accuracy with privacy protection while distributing $675 billion in federal funds and apportioning the US House of Representatives. Learn about the custom-built Spark application that performs millions of optimizations using mixed integer linear programs on a massive cluster. Discover the design of this differential privacy application and the sophisticated monitoring systems implemented in Amazon's GovCloud to oversee multiple clusters and thousands of application runs. Gain insights into the TopDown Algorithm (TDA) and how it addresses key challenges in monitoring Spark. Understand the importance of the Disclosure Avoidance System in enforcing global confidentiality protections for census data.