The Future of Data Pipelines with Atomic Wasm Transformations - Evolving Role of Data Engineers
Explore atomic Wasm transformations in data pipelines and their impact on data engineering roles, focusing on innovative approaches to data processing and workflow optimization.
Building an Open Source Streaming Analytics Stack with Kafka and Druid
Learn to build a robust streaming analytics stack using Kafka and Druid for real-time data processing, flexible querying, and maintaining data integrity in high-volume environments.
Building Robust Streaming Data Pipelines with Apache Spark
Learn to build robust streaming data pipelines using Apache Spark, Kafka, and Camel. Explore ETL challenges, data management, and lessons from running these technologies in Docker.
Deploying Fast Data Pipelines on Mesos and DC/OS
Overview of Fast Data systems, their deployment on Mesos and DC/OS, and their impact on Big Data processing. Learn about the evolving landscape of data pipelines and efficient operational strategies.
What Is a Data Engineer?
Explore data engineering: goals, skills, tools, and its role in the data ecosystem. Learn how data engineers support data scientists and drive business success through effective data management.
Building a Data Science Platform with Argo Workflows
Discover how Bloomberg and Pipekit leverage Argo Workflows and Kubernetes to create scalable data science platforms, enabling data teams to automate pipelines and self-serve their data engineering needs across multiple clusters.
Applied Reinforcement Learning for Online Ads and Recommender Systems
Explore how Reinforcement Learning revolutionizes online advertising, from personalized content delivery to stochastic bandit algorithms, covering business impacts and technical challenges in real-world implementation.
Scheduling Jupyter Notebooks Using Airflow on Kubernetes
Learn to schedule Jupyter notebooks using Airflow on Kubernetes, enabling scalable data pipelines and machine learning workflows with enhanced security and resource management.
Serverless Data Pipelines in Rust - Rust Vienna Meetup
Explore building serverless data pipelines in Rust using DataFusion and object store, with deployment on AWS Lambda. Learn scalable techniques for complex pipelines in production environments.
The Data Pipelines Behind Forest Carbon Credits - Why Pachama Uses Flyte to Orchestrate Workflows
Explore how Pachama uses Flyte to manage complex data and ML pipelines for transparent forest carbon credit computation, including workload management, configuration, and lessons learned.
Database Native Support for CDC Streaming Pipeline
Explore a database-native solution for Cassandra's CDC logs, addressing common challenges and proposing an improved architecture based on industry insights and managed CDC solutions.
New Features of Apache NiFi - Tips and Techniques
Explore Apache NiFi's latest features, processors, and best practices. Build efficient data flows using cutting-edge techniques, with tips and guides for optimal implementation.
Inference at Scale with Apache Beam
Learn to deploy and scale machine learning models efficiently using Apache Beam for distributed inference on CPUs and GPUs, with practical insights on parallelizing workloads.
Apache Arrow and Go: A Match Made in Data
Explore Apache Arrow's Go implementation, covering Arrow and Parquet libraries, Flight server/client, C Data API integration, and performance optimization techniques.
Machine Learning with Apache Beam
Explore Apache Beam for distributed data pipelines in machine learning, covering inference, data processing, and training with hands-on demos for parallelizing ML workloads.
Ensuring Data Quality in Databricks with Great Expectations
Integrate Great Expectations with Databricks to set, test, and enforce data pipeline expectations, ensuring accuracy and reliability in your data workflows.
Delta Format and Live Tables - Pioneering Data Innovation
Explore Delta format and Live tables, understanding their role in scalable big data pipelines and modern data engineering innovations.
ETL Observability: Azure to Snowflake - Monitoring with Splunk and Power BI
Explore ETL monitoring from Azure to Snowflake using ADF, Splunk, and Power BI. Learn data lineage and observability techniques for efficient pipeline management.
Data Engineering with Databricks - Leveraging the Lakehouse Platform for ETL Pipelines
Leverage Databricks Lakehouse Platform to build ETL pipelines, use Delta Live Tables, and orchestrate tasks with Workflows for efficient data processing.
Introduction to Data Build Tool (DBT)
Master data build tool (DBT) for efficient data product development, including CICD, pipelines, validation, quality testing, deployment, and monitoring.
How to Build Data Pipelines for ML Projects with Python Code
Learn to build data pipelines for machine learning projects using Python. Explore ETL vs ELT, extraction, transformation, loading, and orchestration. Includes practical example of processing YouTube video transcripts.