Streaming CDC Data from PostgreSQL to Snowflake - Challenges and Solutions
Uncover streaming ETL secrets: Postgres to Snowflake data ingestion, change data capture, dynamic tables, and Snowpipe streaming. Learn solutions for lower latency, cost savings, and enhanced scalability.
Building a Flexible Data Platform for LLM Training Data
Explore LLM training data platforms: architecture, ingestion, preprocessing, and scaling challenges. Learn from Cohere's experience in managing petabyte-scale datasets for high-quality model training.
Evolving Data Pipelines at Scale with SQLMesh
Discover SQLMesh: revolutionize data pipeline development with virtual environments, automated testing, and accurate change previews inspired by industry giants.
Design Patterns for Data Pipelines Beyond ETL
Discover essential design patterns for managing complex data pipelines in Python, focusing on advanced transformation techniques and Django integration for robust ETL workflows.
Exploring Cribl Stream and Cribl Edge - Data Collection and Observability Pipeline Demo
Discover how to manage observability pipelines with Cribl Stream and Edge, enabling flexible data collection, reduction, enrichment, and routing across your infrastructure for optimized business operations.
Solving Security's Data Problem with Cribl Stream - Building Sustainable Security Architecture
Master data pipeline optimization techniques to enhance security operations, reduce noise, and build sustainable architectures for effective threat detection and response.
Optimizing Telemetry Data Analysis and Log Management with Mezmo
Master telemetry data optimization techniques for DevOps and SREs - from data profiling and cost management to real-time monitoring and sensitive data handling in observability platforms.
Qlik Talend Cloud - AI-Assisted Data Engineering Overview and Demo
Discover how Qlik Talend Cloud revolutionizes data engineering with AI-assisted automation, no-code options, and advanced data integration capabilities for building robust, trustworthy data pipelines.
Google Cloud Storage Solutions for AI and ML Workloads - Performance Optimization and Management
Discover how Google Cloud Storage optimizes AI/ML workloads through GCS FUSE, Anywhere Cache, and Parallel Store features, enhancing performance and reducing training costs for machine learning development.
Data Infrastructure's Impact on AI Value Creation
Explore how data infrastructure shapes AI development, from training data quality to edge computing deployment, emphasizing storage solutions and performance optimization across the complete AI pipeline.
Data Governance and Data Lake Implementation with Delta Lake
Discover how to effectively implement Data Lake with Data Governance through Microsoft's key services, featuring real SK case studies and insights on ML Studio and Power BI integration.
DocETL: AI Agents for Complex Document Processing and Data Transformation
Discover how DocETL framework leverages AI agents and LLMs to transform complex document processing, featuring innovative operators and optimization techniques for enhanced data extraction and analysis.
Machine Learning System Endeavors - From Design to Production
Explore the journey of implementing a video processing AI system, from initial design through production challenges, focusing on ML pipeline management, bug tracking, and real-world impact in smart buildings.
Airflow в кровавом энтерпрайзе
Discover how to effectively deploy and implement Airflow for enterprise automation, with step-by-step guidance on setup, configuration, and best practices for streamlining routine processes.
Airflow Distributed Workloads - ETL Architecture with AWS Lambda and Multi-Threaded Python Scripts
Master distributed workload processing with Airflow, AWS Lambda, and multithreaded Python scripts while exploring ETL architecture for third-party API integration at scale.
Full-link Data Monitoring in Xiaomi Data Integration
Discover comprehensive data monitoring strategies and integration practices used at Xiaomi, focusing on full-link monitoring systems and DataOps implementation techniques.
Practical Pipelines: Houseplant Soil Alerting System with ksqlDB
Develop a houseplant soil alerting system using ksqlDB, exploring practical pipeline implementation for real-time data processing and monitoring.
Using Frameworks to Test Apache Airflow Data Pipelines
Explore frameworks for testing Apache Airflow data pipelines, covering local development, CI/CD, validation, unit tests, and integration tests. Includes a practical demo and key takeaways.
Using Quix, Hugging Face, and InfluxDB for Anomaly Detection
Explore anomaly detection using Quix, Hugging Face, and InfluxDB. Learn about data pipelines, time series databases, and MLOps for effective machine learning implementation.
Working with BigQuery and DataForm - Streamlining Data Management and Transformation
Streamline data management with BigQuery and DataForm. Learn setup, optimization, and automation for efficient, scalable data pipelines and transformations.
Introduction to Data Engineering - Best Practices and Production-Level Infrastructure
Learn best practices for scaling data science projects from local to production, covering infrastructure, reliability, and real-world applications in data engineering.