Elixir for Data Engineering - Batch and Stream Processing
Discover how to leverage Elixir for innovative data engineering solutions in this 24-minute conference talk from Databricks. Explore the power of Erlang's lightweight distributed process coordination for running worker clusters across Docker containers and performing data ingestion. Learn about a framework that integrates Elixir functions as steps in Airflow graphs. Dive into techniques for consuming and processing Kafka events directly within Elixir microservices. Examine real system examples with step-by-step walkthroughs of key elements, covering topics such as services architecture, scheduled job frameworks, and case studies on notification analytics. Gain insights into setting up Kafka consumers and implementing GenServer modules, all without requiring prior Erlang or Elixir knowledge.