Data Formats for Data Science
Explore various data formats for data science beyond CSV and HDFS in this 43-minute EuroPython Conference talk. Delve into the strengths and limitations of different data storage solutions, including plain text, structured formats, and specialized scientific data formats like HDF5, ROOT, and NetCDF. Compare Pythonic implementations such as xarray, pyROOT, rootpy, h5py, PyTables, bcolz, and blaze for handling diverse data structures and sizes. Gain practical guidelines for choosing appropriate formats based on data characteristics and computational requirements. Discover emerging trends in columnar databases like MonetDB for high-speed in-memory analytics, equipping data scientists with a comprehensive understanding of data format options and their applications in scientific computing.