Building Observability Pipeline with Fluent-Bit

Building Observability Pipeline with Fluent-Bit

Explore LinkedIn's journey in constructing observability pipelines using open-source standards and tools, with a focus on OpenTelemetry (OTEL) and Fluent-bit. Learn about the adoption of OTEL for tracing, including its schema and SDKs, and the introduction of a transport agent to simplify data backend access. Discover why Fluent-bit was chosen over the OTEL agent, considering factors such as Kafka integration, log streaming capabilities, and resource efficiency. Gain insights into LinkedIn's enhancements to Fluent-bit, including OTEL transport agent with protobuf support, metrics emission for monitoring, and tag management. Understand the potential for these improvements to be contributed back to the open-source community in the future.