High Throughput with Low Resource Usage - A Logging Journey
Explore the challenges and solutions in high-throughput logging with low resource usage in this 35-minute conference talk from KubeCon + CloudNativeCon North America 2021. Dive into the journey of performance optimization in the Fluent Bit project, focusing on Network I/O + TLS, filesystem buffers, routing, and multiplexing. Learn how to scale logging services in distributed environments while managing increased data volumes efficiently. Discover techniques to improve processing speed from 5k/sec to over 30k/sec using a single-core CPU through design improvements and optimal use of Linux OS interfaces. Gain insights into balancing performance and cost, understanding logging workflows, data analysis, and various aspects of the logging pipeline including input/output, processing, routing, and network setup. Compare JSON vs Message Pack for data serialization and explore memory serialization and buffering techniques. Conclude with a practical demonstration of the concepts discussed.