What are you looking for?
Group Session July 3, 2024

Reproducibility at RDA and beyond: What do we know and what do we need

Plenary: RDA 23rd Plenary Meeting – San José, Costa Rica

Submitted by

Meeting objectives

This is a working meeting. We plan to report on ongoing engagement with groups who do work that touches on reproducibility both at RDA and externally; to solicit further community input on priorities and activities of the IG; to explore interest in potential working and task groups; and to proceed with the work of the IG per the new charter.

We have already contacted and gathered information from a range of RDA and external groups which will feature in this plenary session. These groups will be invited to join the session wherever possible to increase engagement and discussion. The list of groups includes:

RDA groups: FAIRsharing WG, Metadata IG, FAIR Principles for Research Hardware IG, Engaging Researchers with Data IG, Sharing Rewards and Credit (SHARC) IG, Evaluation of Research IG.

External groups: iRise, ACM Reproducibility and Replicability Emerging Interest Group (ACM REP EIG), Cornell Social Science Center, cascad, Institute for Replication, UK Reproducibility Network (UKRN), TIER2, ScreenIT, Project TIER, Trace – Transparency Certified, Social Science Data Editors (a working group), CODECHECK, SciScore.

Meeting presenters

Lauren Cadwallader|Mandy Gooch|Limor Peer

Meeting agenda

Welcome, history and objectives of the Reproducibility IG (10 minutes)
Overview: RDA groups and external organizations doing work that touches on reproducibility (based on May 2024 asynchronous discussion) (30 minutes)
Activity: Mapping RDA internal and external groups (30 minutes)
Discussion and next steps (20 minutes)

Target audience

Groups and individuals that have an interest in or focus on reproducibility. The IG seeks to build bridges between the RDA Working and Interest Groups whose work touches on aspects of reproducibility and the many initiatives around reproducibility outside of the RDA. The activities of the IG are of potential interest to the RDA community as a whole as evidenced by attendance and interest in the P20 BoF and the P21 and VP22 sessions on reproducibility.

Group Activities and Scope

The Reproducibility IG seeks to advance and enable reproducibility in research that is based on or producing datasets that require code. The Reproducibility IG follows the broad definition of reproducibility stated by Turing Way (https://the-turing-way.netlify.app/reproducible-research/overview/overview-definitions.html#table-of-definitions-for-reproducibility) in order to provide an inclusive framework for discussions. The goal of the IG is to build bridges between the many efforts external to RDA that address reproducibility, doing so in coordination with the other RDA Working and Interest Groups where appropriate. The primary areas of focus for the IG are to provide a forum for a holistic discussion of RDA developments around data and code relating to reproducibility, to identify and discuss the development and adoption of relevant tools, workflows, and computational infrastructure, and to develop a research agenda aimed to increase the reliability of the scientific record and reduce irreproducibility (see the Reproducibility IG charter, https://www.rd-alliance.org/rationale/reproducibility-ig/rev-002/). IG efforts may produce Working Groups that develop recommendations and solutions. The IG currently has four co-chairs and holds monthly meetings via Zoom (check the IG homepage for information).

Additional links to informative material

May 2024 asynchronous discussion materials: https://join.slack.com/t/rdareproducibilityig/shared_invite/zt-2i00atvxd-IA34XZIE6pUYDB1p43YIBQ|Historical Reproducibility IG https://www.rd-alliance.org/groups/reproducibility-ig.html|10 Things for Curating Reproducible and FAIR Research https://doi.org/10.15497/RDA00074

Short Group Status

RDA has a historical Interest Group on Reproducibility that was active primarily between 2014-2016 and led by Bernard Schutz and Victoria Stodden. The charge of the original group is “to advance and enable reproducibility in research based on or producing datasets. Our goals are to provide community based recommendations and infrastructure solutions, doing so in coordination with the other RDA Working and Interest Groups where appropriate.” The group primarily focused on efforts around the computational infrastructure necessary for reproducible research. In recent years, several efforts at RDA addressed various aspects of reproducibility, including curation (CURE-FAIR WG), software (FAIR4RS WG, Software Source Code IG), infrastructure (Preservation e-Infrastructure IG), discipline-specific requirements (Reproducible Health Data Services WG), and more. These groups have produced recommendations and other outputs and helped shape the conversation about, and the practice of, reproducibility. At the same time, a good number of efforts to advance reproducibility – in terms of technology, infrastructure, policies, guidelines, and education – are taking place outside RDA.

Recent activity:
-At the RDA 20th Plenary in March 2023 in Gothenburg, a discussion was held about reestablishing the Reproducibility IG in a modern incarnation at a session on “Computational Reproducibility: What’s Next for RDA?”. At the meeting, attendees heard about efforts inside and outside RDA and agreed that the historical Reproducibility IG should be reestablished. See community notes https://docs.google.com/document/d/13dKJss65qJjZ6x4Piv3N2ToyaB0CbJzytH1mJ-oRcv4/edit.
-Per RDA process, the charter was revised, and four interim co-chairs led the revision which was posted for community review (24 May, 2023 to 24 June, 2023). See https://www.rd-alliance.org/rationale/reproducibility-ig/rev-002/.
-The Reproducibility IG was endorsed by the RDA Council in July 2023.
-At the RDA 21st Plenary in October 2023 in Salzburg, the IG held a session, “Introducing the reestablished Reproducibility IG,” which introduced the IG charter and shared input from four RDA groups who have an interest in reproducibility and engaged in community discussion to highlight potential areas for prioritization and collaboration. See recording https://www.youtube.com/watch?v=0R_x9r8odf0.
-The IG did not hold a session at the VP22, but conducted an asynchronous conversation on Slack during May 2024 to facilitate participation by groups and organizations that are not part of the RDA. The goal of the asynchronous conversations was to highlight on-going work around different aspects of reproducibility that the IG members may not necessarily be aware of and to foster discussions, new ideas, and collaborations. See Slack channel https://docs.google.com/document/d/1MopSyY5MQiZGZ_mMJp0UY62h5FjcofJvzLceaKKfzCs/edit#bookmark=id.oudtwzr8dvyr.
-During RDA P23, the Reproducibility IG plans to include a mapping activity around the groups and organizations that participated in the discussion.

Estimate of the required venue room capacity

Under 50 seats

Applicable Pathways

Data Lifecycles - Versioning, Provenance, Citation, and Reward
Other (Reproducible research, research integrity, open science, beyond FAIR)

Please indicate at least (3) three breakout slots that would suit your meeting.

Breakout 2. Tuesday, 12 November, 20:30-22:00 UTC
Breakout 3. Wednesday, 13 November, 14:00-15:30 UTC
Breakout 4. Wednesday, 13 November, 16:00-17:30 UTC
Breakout 5. Wednesday, 13 November, 18:30-20:00 UTC