Description:
Explore efficient strategies for downloading a billion small files using Python in this EuroPython 2019 conference talk. Dive into three concurrent downloading mechanisms: multithreading, multiprocessing, and asyncio. Learn design best practices, debugging techniques, error handling, and performance comparisons for each approach. Gain insights into network latency, file size considerations, and API interactions. Examine code examples and performance metrics to understand the trade-offs between different methods. Discover how to optimize your workflow, handle pagination, and improve download speeds. Apply lessons learned to choose the most suitable library for large-scale file downloading tasks.
Downloading a Billion Files in Python
Add to list
#Conference Talks
#EuroPython
#Programming
#Programming Languages
#Python
#Computer Science
#Concurrent Programming
#High Performance Computing
#Parallel Computing
#Multiprocessing
#Multithreading
#Asyncio