The demand for reproducibility of research results is growing, meaning that it will become increasingly important for a researcher to be able to cite the exact extract of the data set that was used to underpin their research publication. The capacity of computational hardware infrastructures have grown it is now common to have online petabyte data stores, This has encouraged the development of concatenated seamless data sets where users can use web services to select subsets based on spatial and time queries. Further, the growth in computer power has meant that higher level pre-processed data products can be generated in really short time frames.
This means that data sets and data products are needing some form of systematized way of being able to reference the exact version of the data that was used to underpin the research findings, and/or was used to generate higher level products. This was recognised by the RDA Working Group on Data Citation, whose final report recognises the need for Data Versioing. However, there were no specifics on best practice for data versioning, particularly for large volume multi-terabyte and even petabyte scale data sets.
There are two use case for dynamic data. Firstly nothing is done to the existing data sets, and new data are simply being appended at identifiable occurrences. For this case, versioning is more straight forward.
The second use case is more complex and involves existing data sets, models and derivative products being revised with new data, or the data itself revised as processing methods are improved there does not appear to be agreed principles on how data should be versioned.
Versioning procedures and best practices are well established for scientific software and can be used enable reproducibility of scientific results. Are these suitable for data sets or do we need a separate suite of practices for data versioning?
Ultimately versioning will need to be attached to persistent identifiers.
The BoF initially emerged at Plenary 8 in Denver through the discussion available here: https://www.rd-alliance.org/data-versioning-rda-8th-plenary-bof-meeting.