Building Advance Analytics Pipelines with Azure Databricks
Dive deep into Azure Databricks, a fast and collaborative Apache® Spark™ based analytics platform optimized for Azure. Explore Spark's technical overview, Azure Databricks' key collaboration features, cluster management, and tight data integration with Azure data sources. Follow a detailed walkthrough of an advanced analytics pipeline built using Spark and Azure Databricks. Learn about Apache Spark APIs, Spark Application internals, and hidden technical debt in ML systems. Discover Databricks core concepts, anomaly detection techniques using the KDD Cup 1999 Network Intrusion dataset, and how to create custom transformers and estimators. Gain insights into productionizing machine learning workloads, Spark Structured Streaming, and Databricks developer tooling. By the end of this 49-minute conference talk, acquire the knowledge to build and deploy sophisticated analytics pipelines using Azure Databricks.