The Future of Data Pipelines with Atomic Wasm Transformations - Evolving Role of Data Engineers
Explore atomic Wasm transformations in data pipelines and their impact on data engineering roles, focusing on innovative approaches to data processing and workflow optimization.
Building an Open Source Streaming Analytics Stack with Kafka and Druid
Learn to build a robust streaming analytics stack using Kafka and Druid for real-time data processing, flexible querying, and maintaining data integrity in high-volume environments.
Astronomer's Journey from Cloud Lock-in to Open Source Independence
Discover how Astronomer transitioned from AWS-managed services to open-source solutions, enhancing flexibility and independence in their data engineering platform.
Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
Learn how Alluxio enhances Spark and Mesos performance by providing memory-speed data access across diverse storage systems, optimizing big data processing and enabling new workloads.
Building Robust Streaming Data Pipelines with Apache Spark
Learn to build robust streaming data pipelines using Apache Spark, Kafka, and Camel. Explore ETL challenges, data management, and lessons from running these technologies in Docker.
Deploying Fast Data Pipelines on Mesos and DC/OS
Overview of Fast Data systems, their deployment on Mesos and DC/OS, and their impact on Big Data processing. Learn about the evolving landscape of data pipelines and efficient operational strategies.
What Is a Data Engineer?
Explore data engineering: goals, skills, tools, and its role in the data ecosystem. Learn how data engineers support data scientists and drive business success through effective data management.
Building a Data Science Platform with Argo Workflows
Discover how Bloomberg and Pipekit leverage Argo Workflows and Kubernetes to create scalable data science platforms, enabling data teams to automate pipelines and self-serve their data engineering needs across multiple clusters.
Evolution and Scaling of Feature Store at Uber
Explore Uber's Palette Feature Store evolution, focusing on feature sharing, cost reduction, and scalability. Learn about serving 21000+ features at 25M queries/sec with 99.999% uptime.
Beyond Monitoring - The Rise of Data Observability - Shane Murray
Explore data observability as a solution to data downtime, ensuring data trust at scale. Learn its importance in modern DataOps and gain insights for implementing a robust data quality strategy.
End to End MLOPS Data Science Project Implementation With Deployment
Comprehensive MLOps project implementation covering data ingestion, validation, transformation, model training, and deployment on AWS EC2 using MLflow and GitHub Actions.
Applied Reinforcement Learning for Online Ads and Recommender Systems
Explore how Reinforcement Learning revolutionizes online advertising, from personalized content delivery to stochastic bandit algorithms, covering business impacts and technical challenges in real-world implementation.
Production Ready RAG - DevWorld 2024
Explore production-ready Retrieval-Augmented Generation (RAG) techniques for enhancing AI applications with Michel de Ru's expert insights from DevWorld 2024.
Scheduling Jupyter Notebooks Using Airflow on Kubernetes
Learn to schedule Jupyter notebooks using Airflow on Kubernetes, enabling scalable data pipelines and machine learning workflows with enhanced security and resource management.
Serverless Data Pipelines in Rust - Rust Vienna Meetup
Explore building serverless data pipelines in Rust using DataFusion and object store, with deployment on AWS Lambda. Learn scalable techniques for complex pipelines in production environments.
Activating Microsoft Fabric with Real-time Data for Analytics and AI
Explore real-time data streaming and integration for Microsoft Fabric, enabling advanced analytics and AI capabilities with Striim's new adapters for data warehouses and lakehouses.
Attempting to Tame the Data Beast: A BI Transition Tale
Explore the journey of modernizing a BI platform, from legacy systems to cutting-edge tools. Learn about data ingestion, dbt transformations, and Airflow orchestration, along with insights on overcoming challenges.
Navigating the Data Engineering Landscape in the Era of AI
Explore data engineering's role in AI, addressing scalability and real-time processing challenges. Learn from case studies and gain insights into future trends shaping the field's evolution.
The Data Pipelines Behind Forest Carbon Credits - Why Pachama Uses Flyte to Orchestrate Workflows
Explore how Pachama uses Flyte to manage complex data and ML pipelines for transparent forest carbon credit computation, including workload management, configuration, and lessons learned.
Empowering Developers with Easy, Scalable Stream Processing on Kubernetes
Explore scalable stream processing on Kubernetes with Numaflow, an open-source framework enabling easy, cost-efficient data analytics for various engineering roles, powering large-scale operations at Intuit.
Streamlining Data Preparation with Pydantic - A 25-Minute Guide
Explore Pydantic's power in data validation and cleaning for Python projects. Learn to ensure consistency, handle missing values, and streamline data preparation in just 25 minutes.