Apache Beam - Making Big Data Portable With gRPC
Explore the unified programming model of Apache Beam for creating efficient and portable data processing pipelines in this keynote presentation. Discover the dual meaning of portability in Apache Beam, allowing users to run programs across different execution systems like Apache Spark and Apache Flink, while also enabling the use of supported languages regardless of native execution system support. Delve into Beam's architecture, focusing on translators (runners) and the portability framework built on gRPC services. Learn how these services coordinate the execution of functions isolated in language-specific environments. Understand the benefits of using gRPC in Beam, including clear and easily evolving contracts for function execution coordination and enhanced isolation properties when combined with containers.