Explore the evolution of Kafka clusters and producers/consumers for large-scale data analysis in this 41-minute conference talk from SREcon23 Americas. Dive into Honeycomb.io's platform engineering team's experiences as they scaled their telemetry systems to handle a 10x increase in incoming data volume over four years. Learn about crucial decision-making processes for broker/server configurations, including instance type selection, sizing considerations, and the trade-offs between local SSD speed and durability. Discover strategies for optimizing network traffic, implementing chaos engineering on brokers for improved resilience, and distributing data effectively across partitions. Gain insights into scheduling consumer updates and observing the impacts of cluster configuration changes to enhance your streaming telemetry systems.