FLiPN Stack for Edge AI - Flink, NiFi, Pulsar
Explore FLiPN Stack for Edge AI: Learn about Flink, NiFi, and Pulsar integration for efficient data processing and streaming in IoT environments.
4 Tricks to Leverage Your Airflow Pipelines
Discover four powerful techniques to enhance Airflow pipelines: CSV-based DAG creation, sharable config utilization, entity-based autogeneration, and event-driven implementation.
Apache NiFi, Kafka, RisingWave, and Iceberg for Stock Data Processing with LLM - Conf42 Python 2024
Explore stream processing with Apache NiFi, Kafka, RisingWave, and Iceberg. Learn real-time analytics, event-driven applications, and integrating stock data with LLMs in this comprehensive overview.
Building Apache NiFi 2.0 Python Processors - Conf42 Python 2024
Explore Apache NiFi 2.0 Python processors for data engineering, covering features, use cases, and real-time generative AI integration with a live demo.
Streaming DataFrames: A New Way to Process Streaming Data - Conf42 Python 2024
Explore streaming DataFrames as a novel approach to process real-time data, enhancing efficiency and simplifying stream processing workflows for Python developers.
Best Practices to Manage AWS Streaming Data Pipeline at Scale
Learn best practices for managing AWS streaming data pipelines at scale, covering data distribution, large record processing, schema registry, and validations.
Building a Modern Enterprise Data Ecosystem on Cloud - Conf42 Cloud Native 2024
Explore modern enterprise data ecosystems on cloud platforms, covering storage, organization, consumption, pipelines, and machine learning. Learn key components and AWS challenges.
Adding Generative AI to Real-Time Streaming Pipelines - Conf42 LLMs 2024
Explore integrating generative AI into real-time streaming pipelines, covering enterprise use cases, model selection, and implementation using tools like NiFi, Flink SQL, and cloud services.
Leveraging the Apache Flight Go Client and InfluxDB for Time Series Data Management
Explore Apache Flight Go Client and InfluxDB integration for efficient time series data management in modern pipelines. Learn writing and querying data using the InfluxDB v3 Go client library.
Go Concurrency Powering Gigabyte-Scale Real-World Data Pipeline
Explore Go concurrency techniques for building high-performance data pipelines, focusing on MongoDB to RDBMS migration and optimizing worker pools for gigabyte-scale data processing.
Why Most Data Projects Fail and How to Avoid It
Discover strategies to prevent data project failures and ensure success in machine learning initiatives. Learn from industry insights and best practices.
No JVM, No Kafka: The Future of Streaming Data Pipelines
Explore the future of streaming data pipelines without JVM and Kafka, focusing on innovative approaches for efficient and scalable data processing in cloud-native environments.
Integrating Computer Vision and NLP for Smart Property Tagging
Discover how Computer Vision and NLP create 'smart tags' for efficient property searches, enhancing user experience and streamlining data processing in real estate.
The Rise of the YAML Engineer - Declarative Paradigms in Data Systems
Explore the declarative paradigm in data frameworks, its impact on data practitioners, and the role of YAML. Dive into state management, GitOps, and the future of data engineering.
Data Pipelines with Celery: Modular, Signal-Driven and Manageable
Explore efficient data pipeline design using Celery, Redis, and signal-based triggering. Learn to segment workflows, manage API limits, and optimize data processing for improved throughput and maintainability.
Four Tricks to Leverage Your Apache Airflow Pipelines
Leverage CSV files, Google Sheets, Airflow Connections, and event-driven techniques to optimize Apache Airflow pipelines and streamline data engineering workflows.
Modern Data Orchestration: Best Practices and Real-World Use Cases
Elevate your data pipeline game with real-world tricks and best practices. Explore patterns, avoid dependencies, and integrate fundamental concepts for efficient data orchestration.
Gatekeeping Data Quality with Apache Iceberg, Toree, and Airflow
Leverage Apache Iceberg, Toree, and Airflow for efficient data quality validation. Customize data audits with Jupyter notebooks and orchestrate workflows at scale using open-source Apache projects.
Automating Azure Data Factory for Rapid Data Ingestion - 20-Minute Solution
Automate Azure Data Factory ingestion into Data Lake using BimlFlex accelerators in just 20 minutes. Learn efficient data loading techniques and DataOps automation.
Azure IoT from Edge to Azure Data Explorer (ADX)
Explore Azure IoT's power with Digital Twin and Data Explorer, gaining insights from real-world applications and field experiences.
Changes to Azure Synapse Analytics and Their Impact on Data Architecture Designs
Explore Azure Synapse Analytics' evolution and its impact on data transformation, storage, and querying methods to evaluate its suitability for your data environment.