Scaling Security Threat Detection with Apache Spark and Databricks

Scaling Security Threat Detection with Apache Spark and Databricks

Explore advanced security threat detection techniques using Apache Spark and Databricks in this 24-minute conference talk. Learn about Apple's innovative solutions for addressing scale complications, including notebook-based testing CI, self-tuning alerts, automated investigations, and DetectionKit. Discover how to reduce testing time, amplify signal from noise, automate incident containment, and formalize job configuration and testing. Gain insights into modular pre/post processor transform functions and stream-compatible exclusion mechanisms using foreach Batch. Understand the challenges of cyclical investigations, pattern finding, and the importance of document recommendations and automated suggestions in security threat detection.