Unlocking Financial Data with Real-Time Pipelines Using Apache Technologies
Discover how to build powerful real-time data pipelines using Apache Flink, NiFi, Kafka, and Iceberg for processing financial data efficiently, enabling swift decision-making and regulatory compliance.
QuestDB - Building Blocks of a Fast Open-Source Time-Series Database
Dive into the architecture of QuestDB and discover how this open-source time-series database achieves high-speed performance through specialized timestamp handling, efficient data ingestion, and SQL extensions.
Unlocking Advanced Log Analytics With ClickHouse and Kafka
Discover how to build a scalable, cost-effective log analytics solution using ClickHouse and Kafka. Learn JSON handling, data compression, and materialized views for processing billions of log events efficiently.
Apache Airflow 2.x - Modern Workflow Orchestration Features and Improvements
Discover the modern evolution of Apache Airflow 2.x, exploring new UI features, orchestration methods, and testing capabilities while maintaining backward compatibility for workflow management.
Introduction to Orchestrating ETL Processes with Apache Airflow
Master Apache Airflow's workflow management capabilities for ETL processes through practical examples. Learn core concepts like DAGs, Tasks, Variables, and XComs while building efficient data engineering pipelines.
High-Performance Entity Matching Solution Using Vector Embeddings and AWS
Discover how to build a high-performance hotel matching system using vector embeddings, AWS infrastructure, and Qdrant for efficient data processing and improved accuracy in the travel industry.
How LinkedIn Navigates Streams Infrastructure Using Cruise Control - Adem Efe Gencer, PhD
Explore LinkedIn's Cruise Control for managing large-scale Kafka clusters, including architecture, operational challenges, and real-world demonstrations of dynamic load balancing and anomaly detection.
Managing a Multi-Cloud Data and Analytics Platform - Challenges and Solutions
Learn to manage a multi-cloud data and analytics platform, addressing challenges and leveraging benefits across major cloud providers for enterprise-level data, analytics, and AI solutions.
Elixir for Data Engineering - Batch and Stream Processing
Discover how Elixir can revolutionize data engineering with distributed processing, Airflow integration, and Kafka event handling. Learn practical examples for efficient batch and stream processing.
Sound Data Engineering in Rust - From Bits to DataFrames
Explore Rust for efficient data engineering, from low-level bit manipulation to high-performance DataFrames using Apache Arrow. Learn optimization techniques and benchmarks for improved data processing.
Master Azure Automation - The Azure Automation Service
Learn to automate tasks in Azure using Automation Accounts. Explore job scheduling, runbooks, configuration management, and PowerShell modules for efficient cloud operations.
Data Processing Fundamentals - Create GUI for Data Application with Gradio
Learn to create a GUI for data applications using Gradio, covering combined interfaces, tabbed layouts, validation, and integrating filters for enhanced data processing and visualization.
Data Processing Fundamentals - Smarter Python Code With Classes
Learn to write smarter Python code using classes for efficient data processing, including DatasetSelector and DatasetManager implementation with Scikit-learn integration.
Azure Synapse Analytics: Building End-to-End Analytics Solutions
Unified analytics service integrating data warehousing, big data, and BI. Query data flexibly using serverless or dedicated resources at scale. Ingest, explore, prepare, manage, and serve data for immediate insights.
Simplifying Streaming Data Transforms With WASM
Explore WASM-based data transformation for simplified, high-performance real-time applications and data pipelines. Learn architecture, use cases, and see a live demo with Redpanda.
Automating Airflow Backfills with Marquez
Learn to automate Airflow backfills using Marquez, enhancing data pipeline efficiency and reliability in complex workflows.
Lessons Learned from the Migration to Apache Airflow
Explore Apache Airflow for orchestrating complex workflows, with insights on ETL pipelines, Spark integration, and best practices for managing big data infrastructure.
Design Cloud to Edge Architecture Patterns with Azure Arc
Explore Azure Arc for deploying and managing applications across cloud and edge environments, with demos on Kubernetes deployment, AI-enhanced operations, and cross-platform infrastructure management.
Log Processing with Apache NiFi and Apache Kafka
Learn real-time log processing using Apache NiFi and Kafka. Set up the environment, transform data, create flows, and load data into OpenSearch for effective log analytics.
High-Performance Market Data in a Cloud-Native World
Explore challenges and solutions for integrating high-performance electronic trading data with cloud computing, focusing on encoding, protocols, and efficient data analysis techniques.
Data Pipeline - Python Web Scraping for Beginners
Learn to create custom data pipelines for efficient web scraping in Python, streamlining data collection and processing for beginners.