Designing NRT - Near Real Time Stream Processing Systems - Using Python with Storm and Kafka
Learn to build near-real-time stream processing systems using Python, Storm, and Kafka for handling and computing large volumes of data as it arrives, with a focus on creating efficient pipelines.
Feeding Data to AWS Redshift with Airflow
Explore Airflow for efficient data pipelines and learn to integrate it with AWS Redshift for effective ETL processes, schema management, and data transformation techniques.
Using Bonobo, Airflow and Grafana to Visualize Your Business
Hands-on introduction to building business dashboards using Bonobo ETL, Airflow, and Grafana. Learn to create data pipelines, visualize metrics, and improve business insights through practical examples and real-world data.
Building Data Workflows with Luigi and Kubernetes
Learn to build complex data pipelines using Luigi, package them as Docker images, and deploy on Kubernetes for scalable Big Data workflows with on-demand scaling and cost reduction.
Mastering a Data Pipeline with Python - 6 Years of Learned Lessons from Mistakes
Explore key insights and best practices for building robust data pipelines with Python, covering data acquisition, transformation, storage, and workflow management based on 6 years of experience.
Orchestrating Python Workflows in Apache Airflow
Explore Apache Airflow for efficient Python workflow orchestration. Learn strengths, weaknesses, key features, and how to create custom operators based on 5 years of production experience.
Rust for Python Data Engineers
Explore how Rust can enhance data engineering workflows, offering improved performance and safety. Learn about Rust's ecosystem for data processing, databases, and scientific computing to build scalable systems.
Apache Spark vs Cloud-Native SQL Engines
Opinionated comparison of Apache Spark and cloud-native SQL engines, exploring their strengths, use cases, and complementary nature for data processing and business intelligence tasks.
Introduction to Azure Data Factory
Learn Azure Data Factory fundamentals, build new data pipelines, and migrate SSIS packages. Discover how to streamline your data integration processes in the cloud.
Modern Data Science in a Cloud Native World
Explore cloud-native data science pipelines using Kubernetes and Pachyderm. Learn to develop, test, deploy, and scale data workflows while implementing data versioning for enhanced tracking and GDPR compliance.
Processing Data at Scale with Rails
Strategies for handling large datasets using Rails, covering data ingestion, transformation, and storage with tools like ActiveRecord, Sidekiq, and Postgres.
Dat - An Open Source Tool for Sharing and Collaborating on Data
Explore Dat, an open-source tool for data sharing and collaboration. Learn about its features, applications in reproducible science, and how it aims to revolutionize data management like Git did for source control.
Versioning, Syncing & Streaming Large Datasets Using DAT + Node
Explore versioning, syncing, and streaming large datasets using DAT and Node.js. Learn about open-source tools for efficient data collaboration, reproducibility, and building cross-platform data pipelines.
Not a Magic - What to Expect from Machine Learning Projects
Practical insights on implementing machine learning projects, covering data handling, algorithm selection, training, evaluation, and deployment challenges in organizational settings.
Automate Your City Data With Python Using Datasette and GitHub Actions
Discover how to build automated data pipelines using Python, Datasette, and GitHub Actions to extract and analyze city planning decisions and public data for better civic engagement.
Data Pipeline Scorecard: Evaluating Python Processing Approaches
Explore data processing approaches in Python, comparing different methods and their practical applications for building efficient data pipelines and optimizing workflow performance.
Intent Realization in Self-Serve Platforms - PlatformCon 2024
Explore intent realization's impact on reducing friction and improving developer experience in self-serve platforms, focusing on Netflix's data infrastructure and user-friendly tooling.
Introduction to Apache Beam with Examples
Learn Apache Beam's unified model for batch and streaming data processing, with real-world examples from eBay and Mozilla's data pipelines.
Kafka Connectors: Introduction to Kafka Connect Framework
Explore Kafka Connect framework for integrating Kafka with external systems, focusing on cluster setup and connector functionality.
Bridging the Gap Between Batch and Real-Time with Mixed-Latency Pipelines
Explore mixed-latency data platforms, manage data freshness, and tackle pipeline bottlenecks. Learn to merge batch and low-latency processing while maintaining explicit lineage for reliable data products.
Solving Security's Data Problem - Building Sustainable Security Architecture
Discover how data pipelines can revolutionize security architecture by reducing volume, improving signal-to-noise ratios, and creating more efficient, cost-effective security operations.