Best Practices When Accessing Big Data or Any Other Data
Explore best practices for accessing data on local or shared file systems in this ACCU 2019 conference talk. Learn from real customer examples how simple guidelines and proper tool usage can significantly improve code performance, reliability, scalability, and portability. Discover why data access methods are crucial, especially as datasets grow larger. Gain insights from Dr. Rosemary Francis, an expert in I/O profiling and high-performance computing, as she shares her experience working with scientific and high-performance computing customers. Understand the common challenges in I/O management and learn practical solutions to optimize applications and manage IT infrastructure effectively. Delve into topics such as the noisy neighbor problem, housekeeping and deletes, failed I/O, file system traversals, opens and closes, and storage comparisons. Benefit from a case study on tuning cancer pipelines at the Sanger Institute to see these principles applied in a real-world scenario.