Ingesting 35 Million Hotel Images with Python in the Cloud
Learn how Skyscanner built a distributed architecture using Python to process and manage 35 million hotel images in the cloud. Explore the challenges and solutions involved in creating an incremental image processing pipeline that discards poor quality and duplicate images while optimizing for mobile devices. Discover the tools and techniques used, including Pillow, ImageHash, Kombu, and Boto, to handle tasks such as ingesting partner-provided images, detecting and removing bad quality and duplicates, resizing images, and ensuring scalability within time constraints. Gain insights into the technical stack, triggering mechanisms, downloading processes, fingerprinting techniques, and methods for choosing the best images in this informative conference talk from EuroPython 2016.