Versioning, Syncing & Streaming Large Datasets Using DAT + Node
Explore versioning, syncing, and streaming large datasets using DAT and Node.js in this JSConf.Asia 2014 conference talk. Dive into the open-source DAT tool, funded by the Sloan Foundation, designed to facilitate collaboration on datasets of any size. Learn how DAT aims to streamline work with large scientific datasets, enhancing automation and reproducibility. Discover the tool's streaming dataset versioning and replication system, built with a Unix philosophy for modularity and third-party application support. Gain insights into using Node.js and Docker for creating cross-platform data pipelines. Understand how to leverage Node and LevelDB for managing extensive datasets. The speaker, Max Ogden, an open-source software developer working on the DAT project at the United States Open Data Institute, shares his expertise and demonstrates practical examples throughout the 43-minute presentation.