Building Modern Data Pipelines with Spark on Azure HDInsight

Building Modern Data Pipelines with Spark on Azure HDInsight

Explore common patterns for building end-to-end data analytics pipelines using Apache Spark on Azure HDInsight in this conference talk from PASS Summit 2017. Dive into architecture examples, integration points, and various components of modern data pipelines. Learn about edge computing, notebooks, data visualization, and custom visualizations. Discover how to import data, use Hive context, and manage resources with Yarn. Examine batch and ETL pipelines, data sources, and pipeline tools like Azure Data Factory. Gain insights into real-time pipelines and data lineage considerations for building robust, scalable data solutions.