The Daft Distributed Python Data Engine: Multimodal Data Curation at Any Scale
Explore the Daft distributed Python data engine for multimodal data curation at any scale in this 27-minute talk by Jay Chia. Discover how Daft addresses the fundamental needs of ML/AI data platforms, including terabyte-scale ETL with complex model batch…
Anomaly Detection on 5 Pillars of Data Observability with Dynatrace Davis AI
Explore Dynatrace Davis AI's anomaly detection for data observability pillars: freshness, distribution, volume, schema, and lineage. Learn automation techniques for alerting and monitoring data integrity.
Parallelizing Your ETL with Dask on Kubeflow
Leverage Dask's parallelism capabilities on Kubeflow for advanced ETL processing. Learn to use the Dask Operator to optimize resource utilization in interactive Jupyter sessions and pipeline workflows.
Big Data Analytics for Beginners
Comprehensive introduction to big data analytics, covering key concepts like Hadoop, Apache Spark, and ETL. Ideal for beginners seeking to understand the landscape and value chain of big data.
Smart Data Pipelines to Azure - Ingesting and Migrating Data the DataOps Way
Explore StreamSets' DataOps platform for optimizing data ingestion and migration to Azure. Learn about smart data pipelines, integration with Azure services, and how to speed up cloud adoption using modern data engineering practices.
Strategy and Latest Updates for Data Integration at SAP
Explore SAP's data integration strategy, focusing on SAP Data Intelligence's role in orchestrating complex data landscapes and powering the Business Technology Platform for effective hybrid data management.
Building a Lakehouse with Apache Hudi - Data Architecture at Kuaishou
Discover how Kuaishou Inc implements Apache Hudi to build a robust lakehouse architecture, exploring real-world applications and technical insights for large-scale data management.
Running Remote Shuffle Service to Solve Apache Spark's Dynamic Resource Allocation Challenge on Kubernetes
探讨利用远程洗牌服务解决Apache Spark在Kubernetes上的动态资源分配挑战,实现存储与计算解耦,提高大数据处理的可靠性和可扩展性。
SQL está en todas partes: Explorando alternativas con Ibis
Descubre Ibis: una biblioteca Python para consultas de datos eficientes en múltiples backends. Simplifica el trabajo con bases de datos SQL y ofrece una alternativa a las complejas consultas tradicionales.
Reinventing ETL for Detection and Response Teams
Explore innovative strategies for efficient, cost-effective data collection and log enrichment tailored to Detection and Response teams' unique requirements in cybersecurity.
Supercharging ETL and Analytics with Ray and Daft
Discover how Daft library enhances Ray clusters with distributed ETL capabilities, enabling seamless data processing, analytics, and ML/AI integration for scalable, high-performance workflows.
ETL with Meltano and Singer in the LLM Era
Explore how traditional ETL tools like Singer and Meltano can enhance data pipeline management for LLM applications, addressing production-level challenges beyond basic LangChain capabilities.
Navigating the Landscape of a Fully Open Source Data Stack in 2023
Explore the architecture of modern data stacks, from warehousing to streaming solutions, while understanding the potential and limitations of open-source tools in today's data infrastructure.
Unveiling the Power of dbt and DuckDB - Hype vs Reality
Explore the practical applications and limitations of combining dbt and DuckDB for data transformations, examining real-world use cases and performance considerations.
The Real Modern Data Stack - Building ETL Pipelines with Open Source Tools
Discover how to build powerful, cost-effective ETL data stacks using open-source tools like sling, dbt, duckdb, and dagster - transforming months of implementation into rapid deployment.
Data Engineering and Storage Concepts - With ETL, ELT, Warehouse, Lake, and Lake House
Comprehensive overview of data engineering concepts, including storage solutions, ETL/ELT processes, and modern data architectures like data mesh, with practical examples and best practices for effective data management.
Empowering Developers with Self-Service ETL - Zillow's Approach
Zillow's self-service ETL solutions empower teams to build, maintain, and monitor data pipelines, abstracting complex processes and catering to diverse user needs while leveraging internal services for efficient implementation.
Code Once, Use Often - Declarative Data Pipelines
Learn to build efficient, reusable data pipelines using declarative approaches. Explore techniques for reducing food waste through data-driven solutions and automation in this developer-focused session.
Make Reliable ETL Easy on Delta Lake
Learn to simplify ETL development on Delta Lake, improving data quality and scaling operations. Discover how Databricks streamlines the ETL lifecycle for efficient data engineering and management.
Designing the Next Generation of Data Pipelines with Apache Spark - Zillow's Approach
Zillow's data engineering team shares their journey in redesigning data pipelines using Apache Spark, focusing on scalability, maintainability, and robustness while reducing code complexity.
Data Quality Tools Comparison for Continuous Data Imports
Comparison of open-source data quality tools for continuous imports, covering maturity, documentation, extensibility, and features like data profiling and anomaly detection.