A Deep Dive into the dbt Manifest
Explore dbt's manifest.json file, its production, and applications in data pipelines. Learn about deployment configurations, quality enforcement, and streamlined development workflows using real-world examples.
Data Warehouses are Gilded Cages - What Comes Next
Explore DuckDB's potential to revolutionize data analytics, shifting from centralized warehouses to powerful local processing. Learn how this approach can enhance efficiency and user control in data management.
How to Build a Streaming Database in Three Challenging Steps
Explore the foundational choices and trade-offs in building a streaming database with Frank McSherry, Chief Scientist at Materialize. Learn manageable approaches to this complex task.
Making Moves with Arrow Data: Introducing Arrow Database Connectivity (ADBC)
Explore Arrow Database Connectivity (ADBC), a new Apache Arrow subproject for Arrow-based database access. Learn its goals, use cases, and communication with data APIs using Arrow-native in-memory data.
How Investors Think About Data - Trends in Data, ML, and AI Investing
Explore investor perspectives on data, AI, and ML from top venture firms. Gain insights into investment trends, criteria for early-stage teams, and future opportunities in the data infrastructure landscape.
When to Move from Batch to Streaming and How to Do It Without Hiring an Entirely New Team
Demystify the transition from batch to real-time processing. Learn arguments, architectures, pitfalls, and open-source tools for stream processing without overhauling your team.
Declarative Orchestration: Why You've Been Thinking About the Wrong DAG the Entire Time
Explore declarative, asset-oriented orchestration in data pipelines. Learn how this approach simplifies tasks, ensures data integrity, and implements Data Mesh as a living asset graph.
What Makes for an Effective Data Practitioner in 2024
Gain insights on essential skills and career development for data practitioners in 2024. Learn from an industry expert about navigating complexities and advancing in the field.
Ten Years of Building Open Source Standards in Data Engineering
Explore a decade of open-source standards development, from Parquet to Arrow. Gain insights into community-driven innovation and the evolving landscape of data engineering.
The Future of Data Engineering in a Post-AI World
Explore AI's impact on data engineering, including ETL automation, data warehouses, and emerging technologies. Learn to leverage AI for significant business impact in the evolving data landscape.
Flappybird Game and Real-Time Data Pipeline Workshop - Tinybird
Build a real-time data pipeline for Flappybird's leaderboard. Capture in-game events, stream data, and create REST APIs while enhancing SQL skills in this hands-on workshop.
Building a Reliable, Secure and Efficient Event Ingestion Pipeline
Explore challenges in data pipelines, from client interactions to backend complexities. Gain insights into building reliable, secure, and efficient event ingestion systems.
Rising Tides with Radical Transparency - Open Sourcing Your Data Platform
Explore the journey of open-sourcing a data platform, including challenges, cultural shifts, and innovative implementations. Learn to leverage open source for enhancing projects and contributing to the data community.
Upgrading to Drupal 10 Using the Migrate API
Explore Drupal 10 migration using the Migrate API, covering ETL, UI and CLI upgrades, configuration, plugins, and content model changes.
Apache Spark Structured APIs Using Databricks
Master Apache Spark Structured APIs for data manipulation using Databricks. Learn RDD, DataFrame API, data aggregation, and ETL operations in this comprehensive overview.
Orchestration of Data Pipelines Using Airflow
Learn to orchestrate data pipelines with Airflow, integrating various operators like Bash, Python, Spark, Email, and Slack for efficient workflow management and automation.
How to Create and Manage Azure Data Factory Using Terraform
Provision and manage Azure Data Factory with Terraform for scalable, serverless data integration and transformation in the cloud.
Getting Started with Apache Spark
Learn Apache Spark's architecture, data lineage, and DAG concepts. Explore this multi-language engine for data engineering, data science, and machine learning on single-node machines or clusters.
MLOps on Azure Machine Learning Studio
Automate and streamline machine learning lifecycle with MLOps on Azure ML, integrating ML, DevOps, and Data Engineering for consistent, high-quality model delivery.
GenAI for Observability in the Serverless World
Harness GenAI to enhance serverless observability, leveraging vector databases and ML for improved data ingestion, log analysis, and problem resolution.
Generative AI-Powered Use Cases for Data Engineers
Explore AI-driven innovations for data engineers, accelerating code creation and data insights with Duet AI in Google Cloud.