Spark Over a Mesos Secured Productive Environment
Explore advanced Spark and Mesos integration for secure, isolated Big Data environments. Learn to enhance protection, manage secrets, and implement network solutions in enterprise-level deployments.
Getting Started with Delta Lake
Learn to create a Lakehouse architecture with Delta Lake, an open-source storage framework. Explore data reliability challenges, hands-on examples, and benefits for big data use cases in modern data engineering.
Socializing the Elephant - Integrating Big Data Using Egeria Open Metadata
Explore Egeria's open metadata framework for seamless integration of Big Data with diverse information systems, enhancing enterprise-wide data management and governance.
Building Modern Data Streaming Apps with Open Source
Discover best practices for building data streaming applications using open-source frameworks. Learn to orchestrate streams, perform ETL, enhance events, and make continuous queries with tools like Apache NiFi, Pulsar, Spark, and Flink SQL.
Lessons Learned from Big Data and AI/ML Collaboration for Giant Hogweed Eradication
Explore big data and AI/ML collaboration for invasive plant eradication, covering challenges, architecture, data flow, and lessons learned in scalable machine learning systems.
Bringing the Power of ONNX to Apache Spark - Integrating Deep Learning Inference
Explore integrating ONNX with Spark for efficient deep learning inference pipelines. Learn about Ascend Hardware Platform's ONNX contributions and Spark community's simplified API for DL inferencing.
Automating Cloud-native Spark Jobs with Argo Workflows
Learn to orchestrate Spark jobs on Kubernetes using Argo Workflows, exploring architecture, resource management, and scaling advantages for efficient large-scale data processing in cloud-native environments.
Democratizing AI with Sparklyr
Learn to leverage Sparklyr for scalable machine learning and AI applications, empowering data scientists to work efficiently with big data in R.
Simplifying Testing of Spark Applications
Learn techniques to simplify testing Spark applications, covering Python UDFs, Pandas UDFs, and functional APIs. Explore practical scenarios and demos for efficient data processing and analysis.
The Pace of Innovation in Delta Lake
Explore the rapid advancements and innovative features of Delta Lake, enhancing data reliability and performance in modern data architectures.
Delta Lake - Diving into Data Lakes Without the Downsides
Explore Delta Lake's innovative approach to data management, offering enhanced reliability and performance for large-scale analytics without traditional data lake drawbacks.
Machine Learning and Apache Spark: A Dynamic Duo
Explore how Apache Spark and machine learning revolutionize industries, enabling data-driven decisions and innovative products. Learn about top use cases, market dynamics, and IBM's commitment to Spark.
Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
Learn how Alluxio enhances Spark and Mesos performance by providing memory-speed data access across diverse storage systems, optimizing big data processing and enabling new workloads.
Building Robust Streaming Data Pipelines with Apache Spark
Learn to build robust streaming data pipelines using Apache Spark, Kafka, and Camel. Explore ETL challenges, data management, and lessons from running these technologies in Docker.
Spotlight on Emerging Data Science Workspaces and Spark as Development Levers
Explore how developers leverage Apache Spark and collaborative workspaces to build machine learning solutions, revolutionizing data applications and digital innovation.
Spark on Kubernetes: Building an Elastic Service - Apple's Approach
Explore Apache Spark on Kubernetes, focusing on autoscaling challenges and solutions for efficient large-scale data processing in multi-tenant environments with advanced features.
Tower of Babel - Making Apache Spark, Kubeflow, and Kubernetes Play Nice
Explore how Apache Spark, Kubeflow, and Kubernetes can work together to process big data matrices, denoise medical images, and create reproducible ML pipelines for COVID research and healthcare applications.
The Future of Reproducible Research - Powered by Kubeflow
Explore the future of reproducible research using Kubeflow, an open-source platform that streamlines data science workflows and enhances scientific collaboration through easily shareable and replicable pipelines.
Spark, Cloud, DBT, Data Warehouses Etc. - Navdeep Kaur
Explore current trends in Data Engineering, including the shift towards new data stacks, tech used by companies, reasons for change, and the transition to data warehouses and delta lakes.
Beyond Experimental - Spark on Kubernetes
Insights on leveraging open-source tech for Apache Spark on Kubernetes, addressing challenges in large-scale production environments and evolving ML data infrastructure in the cloud.
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
Discover best practices for building scalable Spark clusters on Kubernetes using Argo Workflows and Argo Events, enabling efficient resource utilization and seamless scalability for powerful data processing.