Use Python to Process 12 Million Events per Minute and Still Keep It Simple

Use Python to Process 12 Million Events per Minute and Still Keep It Simple

Discover how to build a high-performance event processing system using Python 3 and open-source tools in this EuroPython Conference talk. Learn to handle a throughput of 1 million events per minute on a 4-core AWS instance while maintaining simplicity and scalability. Explore techniques for data aggregation, failsafe mechanisms, and profiling for Linux services in production. Address unexpected behaviors in Python's Standard Library, such as reading from a file while it's being written, and learn strategies for tackling sudden cloud instance failures. Gain insights into reducing hardware utilization by over 85% and creating client-specific solutions without relying on complex, general-purpose technology stacks.