Master Databricks and Apache Spark - Introduction
Learn the fundamentals of Databricks and Apache Spark, exploring scale-up vs. scale-out concepts and laying the groundwork for mastering these powerful data processing tools.
Master Databricks and Apache Spark Step by Step - PySpark Using RDDs
Explore PySpark and RDDs in Databricks: Learn about lazy evaluation, transformations, and actions while analyzing data using Resilient Distributed Datasets.
Using SQL Window Functions in Databricks and Apache Spark - Lesson 16
Learn to leverage Spark SQL window functions for advanced data engineering on Databricks, including cumulative totals, ranking, and aggregations alongside detail rows, enhancing your data manipulation capabilities.
Using SQL Views in Databricks and Apache Spark - Lesson 12
Learn to create and utilize SQL views in Databricks for efficient data analysis and pipeline reuse. Explore common table expressions, aggregations, and case statements for advanced querying techniques.
Understanding Spark SQL - Lesson 6
Explore Spark SQL's powerful features for data engineering, including query optimizations and support for Python, R, and Scala, enhancing parallel processing performance.
Using the Data Science Process - Lesson 5
Explore the data science process from project inception to completion, covering key steps for successful implementation in Databricks and Apache Spark projects.
Create a Spark Cluster - Master Databricks and Apache Spark Step by Step - Lesson 4
Learn to create and use Apache Spark clusters, understand their differences from Databricks, and explore data querying with Zeppelin notebooks in this step-by-step guide.
Databricks Workspace Tour and Notebook Demo - Lesson 3
Comprehensive tour of Databricks Workspace, demonstrating tools and features with focus on the proprietary notebook IDE. Provides foundation for advanced Databricks and Apache Spark usage.
Create a Databricks Workspace - Lesson 2
Learn to create a Databricks workspace in Azure, understanding its components, pricing, and setup process. Gain practical skills for setting up your cloud-based data analytics environment.
Pandas UDFs and Python Type Hints in Apache Spark 3.0
Explore redesigned pandas UDFs in Apache Spark 3.0, leveraging Python type hints for clearer, more Pythonic data science workflows and improved static analysis capabilities.
How Adobe Processes 2 Million Records Per Second Using Apache Spark
Explore Adobe's high-performance data processing using Apache Spark, covering optimization techniques, join strategies, structured streaming, skew handling, and integration with Redis for enhanced efficiency.
Accelerating Data Processing in Spark SQL with Pandas UDFs - Optimization Techniques
Learn to optimize Spark SQL data processing using Pandas UDFs. Discover techniques to speed up queries, profile jobs efficiently, and leverage custom UDFs for significant performance gains in large-scale data pipelines.
Apache Spark vs Databricks - On-Premise Performance Comparison
Comparison of on-premise Databricks and open-source Spark implementations, showcasing 10x performance gains in cyber workloads and addressing challenges of air-gapped analytics solutions.
Hyperspace - An Indexing Subsystem for Apache Spark
Explore Hyperspace, an indexing subsystem for Apache Spark that enhances query performance and reduces resource costs. Learn about its design, implementation, and benefits for data analytics on large datasets.
Operationalizing Apache Spark MLlib Models with SAS Model Manager
Learn to operationalize Apache Spark MLlib models using SAS Model Manager and Workflow Manager. Discover how to automate and govern the entire model lifecycle, from building to monitoring, accelerating deployment while maintaining control.
Data Microservices in Apache Spark Using Apache Arrow Flight
Explore efficient machine learning pipelines using Apache Arrow Flight in Spark, enabling faster data transport and simplified microservices for improved ML model production.
Building a Data Platform for Mission-Critical Analytics on AWS with Databricks
Explore building enterprise data lakes on AWS with Databricks, complementing AWS strategy. Learn from HP's success in transforming business through unified data analytics and overcoming data lake challenges.
Delta from a Data Engineer's Perspective - Creating Robust Big Data Solutions
Explore a data engineer's challenges in creating robust Big Data solutions, covering Delta, Spark streaming, schema enforcement, and time travel functionality in data processing.
Running Apache Spark on Kubernetes - Best Practices and Pitfalls
Learn best practices and pitfalls for running Apache Spark on Kubernetes, covering setup, performance optimization, resource management, security, and future developments in this comprehensive overview.
Portable and Scalable Data Visualization Techniques for Apache Spark and Python Notebooks
Explore portable and scalable data visualization techniques for Apache Spark and Python notebooks, covering strategies, packages, headless rendering, animations, and big data integration.
Preventing Common Pitfalls in Production Streaming Jobs
Explore key strategies for optimizing streaming jobs in production, covering input parameters, stateful streaming, output configurations, and job modifications to ensure performance and fault tolerance.