Fixing Small Files Performance Issues in Apache Spark Using DataFlint
Master Apache Spark's file handling optimization techniques and learn to solve small files performance issues using DataFlint. Explore best practices for storage formats like Delta Lake and Iceberg.
Apache NiFi, Kafka, RisingWave, and Iceberg for Stock Data Processing with LLM - Conf42 Python 2024
Explore stream processing with Apache NiFi, Kafka, RisingWave, and Iceberg. Learn real-time analytics, event-driven applications, and integrating stock data with LLMs in this comprehensive overview.
Bytedance Deep Learning: Batch Flow Integrated Training Practice
Explore ByteDance's batch-stream integrated AI training engine, supporting large-scale cloud-native models for various businesses. Learn about flexible scheduling, data orchestration, and performance optimization techniques.
Gatekeeping Data Quality with Apache Iceberg, Toree, and Airflow
Leverage Apache Iceberg, Toree, and Airflow for efficient data quality validation. Customize data audits with Jupyter notebooks and orchestrate workflows at scale using open-source Apache projects.
What is a Headless Data Architecture? - Understanding the Modular Approach to Data Management
Explore headless data architecture: decouple computation from storage, enable modular ecosystems, and streamline data processing for real-time and analytical use cases.
Shift Left: Rethinking Data Management with Streams and Tables
Rethink data circulation using DataStreams and Change Data Capture. Learn to shift data preparation to creation point, building fresh datasets as streams for operational use or Apache Iceberg tables for analytics.
Delivering Portability to Open Data Lakes with Delta Lake UniForm
Explore Delta Lake UniForm's solution for enhancing data lake reliability, performance, and governance across multiple environments. Learn best practices for scaling and optimizing cloud-based data lakes.
Open Data Foundations Across Hudi, Iceberg and Delta - OneTable Integration
Explore OneTable's innovative approach to optimizing data across Hudi, Iceberg, and Delta Lake. Learn how it enhances workloads for various query engines and streamlines data analysis.
Building a Management Layer for Structured and Unstructured Data Lakes
Discover how to effectively manage data lakes using modern architectures, open table formats, and version control systems through practical examples with Databricks, Apache Iceberg, and AWS technologies.
BigQuery and BigLake: Real-World Data Products for AI/ML at Scale
Discover how BigQuery and BigLake power innovative data analytics and AI/ML solutions at scale. Learn from Google Cloud experts and real-world customer experiences at Trendyol and Snap.
How to Migrate from Snowflake to an Open Data Lakehouse Using Delta Lake UniForm
Explore Delta Lake UniForm's ability to create a single data set accessible across multiple open table formats, enhancing data lakehouse architecture efficiency and flexibility.
Capital One's Data Innovation Strategy - You Build, Your Data (YBYD)
Capital One's cloud-based data ecosystem enables self-service innovation for 13,000+ technologists. Learn about their "you build, your data" strategy, enterprise data platforms, and best practices for leveraging data at scale.
Secure and Streamline Data Engineering Workloads with Azure Data Factory and Snowflake
Discover how to optimize data engineering workflows using Azure Data Factory and Microsoft Fabric with Snowflake, featuring bi-directional data access through Apache Iceberg for enhanced security and performance.
Open Standards for Open Lakehouses - Understanding Apache Iceberg, Parquet, Arrow, and Nessie
Discover how open-source standards like Apache Iceberg and Nessie power data lakehouses, enabling flexible and cost-effective data platforms while minimizing vendor lock-in and data movement challenges.
Object Storage Is All You Need - From Cloud Persistence to Modern Applications
Explore how object storage is evolving from simple web storage to a complete persistence solution, examining its benefits, performance aspects, and practical applications in modern cloud systems.